Members of the public saw the body near Queens Wharf and alerted emergency services about 10.40am.
The police maritime unit helped recover the body.
Eyewitness Emma Pigou said she saw police earlier attempting to retrieve the body from the harbour.
"I saw the cops pulling him out, or trying to," she said.
Ms Pigou said the man was wearing blue jeans and a red fleece jersey with some blue colouring.
"Last time I saw, he was lying on the concrete," she said.
"There were a lot of people on the wharf having a look."
Part of the wharf beside Wagamama restaurant was cordoned off.
It appeared the man's body was brought onto the wharf before being taken to the nearby police maritime unit and national dive squad building.
Police were still at the scene after midday and the cordon remained in place.
Ms Pigou said ambulance staff were also at the scene earlier.
The cordon was immediately beside part of today's marathon route.
Police said the body was that of "an older male".
The man's identity and the reason or reasons for his death were not yet known.
In March, 20 year-old man Jamie Gibbon died after jumping some 45m off the nearby Hikitia waterfront crane after a night with friends in the city.
Mr Gibbon's death has sparked calls for tighter security in the area.
