Emergency services were called to the scene at Rafters Rd, off the Gibbston Highway, about 4pm.
Queenstown harbourmaster Marty Black said the men were with a group of about five people having a picnic on the river bank.
While they were throwing sticks to the dog the animal fell into the river and the two men jumped in after it.
"They were swept away about a 1km downstream."
The men were able to get on to a bank, but were stuck there, as it was too steep to climb.
The other members of their party informed emergency services.
About an hour later the men were rescued by private rafters.
Mr Black said it was a lesson to be careful around waterways.
"It just shows, you jump in the river and you think it will be fine, but the river's quite high at the moment. These guys are pretty lucky, really."
Although he was not sure of the specifics, it was his understanding the dog was safe.
