The structure remained standing after the crash at 12.40pm.
The two passengers were shaken and received treatment at the scene for minor bumps, while the driver was uninjured.
One passenger said the southbound bus was travelling from George St into the Octagon when the crash occurred.
She had boarded the bus at the last stop and was looking inside her bag when the bus hit the pole, but said there was no noticeable swerving.
A woman having lunch nearby said "it was incredibly loud".
"It gave me a good old fright," she said.
Senior Sergeant Craig Brown said it appeared the driver lost control of the bus, but why would not be known until a police commercial vehicle investigation unit (CVIU) investigation was complete.
Dunedin Passenger Transport company director Kayne Baas said he was unable to comment until then.
The damage to the bus was worse than it appeared as the exterior was made of fibreglass.
The bus was driven from the Octagon.
Dunedin City Council roading and maintenance engineer Peter Standring said a temporary supporting pole had been put in place.
