Queenstown harbour master Marty Black said an increasing number of people were getting in the way of the ship as it approached and left its mooring spot at Steamer Wharf.
"With the warm temperatures over the past week, more people want to get in the water.
"It's all very well, but they have to be more careful," he said.
"It is becoming a serious issue and, while I often warn people, I cannot be here all the time.
"I do not want to see an accident."
Swimmers in the Queenstown Bay part of the lake usually waded out from the beach at Marine Pde, or jumped into the water from the new wooden boardwalk near Real Journey's ticket office.
TSS Earnslaw senior skipper Graham Carter agreed the issue with swimmers had been on the rise.
He said people jumping off the boardwalk could at least be seen and waved out of the way when the ship left the wharf.
However, he was concerned about swimmers in the bay between the beach and the Queenstown Gardens.
"When we are up on a large, moving boat it is really hard to spot a little head sticking out of the water until we are too close.
"Over the last few days, I have even seen examples of swimmers waving back to us and then continuing to swim straight towards the ship," he said.
"A ship of this type and size simply cannot stop or turn on the spot.
"Our incoming route is to sail close to Queenstown Gardens, arch around the two yellow markers and then approach the Steamer Wharf.
"It would be great if people could keep that in mind and not swim too far into the middle of Queenstown Bay," he said.
Tracey Maclaren, general manager of Real Journeys, which operates TSS Earnslaw, said the company took safety seriously.
"We totally understand people want to take a cooling dip or a swim in the lake.
"We just ask them to be careful.
"The last thing we want is to see anyone getting hurt," Ms Maclaren said.
Mr Black asked people who wished to jump off the boardwalk, to jump on the inside towards the newly paved area, and he urged swimmers in the bay to stay close to the shore and keep an eye out for Earnslaw.
