The sea lion came up under the boat, hitting it twice and breaking through the hull in two places.
"I saw this dark figure looming under the boat. I felt it hit and, seconds after, water came gushing up ... it was panic stations," rower Matt Smail, who was sitting right above the break, said.
The eight-man crew was about 200m off shore near St Leonards and decided to head to the closest boat ramp.
"It was the best 1km we rowed all morning. We went flat tack," Adam Garden said.
Then when they got there and pulled the boat out of the water, the sea lion "came up for another inspection", he said.
"It walked up the ramp and gave us a bit of a snarl."
Otago University rowing coach Sonya Walker said rowers often saw sea lions in the harbour and normally managed to avoid them.
"We try and share the water with them."
The damage to the $40,000 boat was repairable, she said.
New Zealand Sea Lion Trust chairman Shaun McConkey said it was not unusual for sea lions to be inquisitive and even bump at boats playfully, but it was the first time he had heard of one doing damage to a boat.
"I'd be very surprised if it was any sort of aggressive attack. It's more likely a combination of playfulness with a light boat."
It was likely the perpetrator was the male sea lion that had been in Otago Harbour for some time, he said.
Department of Conservation biodiversity assets programme manager David Agnew said the incident reinforced the need for people to be aware it was illegal to feed sea lions from boats.
Doing so encouraged them to approach boats, thinking they would get food.