The concerns come as the Ministry of Fisheries implements a two-month observer programme to assess whether regulations announced last year to protect endangered Hector's and Maui dolphins have been effective and what impact commercial fishing is having on the mammals.
Up to 50 observers would be working around the South Island and west coast of the North Island during January and February.
Port Chalmers Fishermen's Co-operative president Steve Little said fishermen had their "backs up" after what they saw as a "threatening position" in a letter from the ministry, which stated they had a legal obligation to take the observers or face a $250,000 fine.
"It's a big stick, heavy-handed approach. There is a lot of bad feeling out there," Mr Little said.
Many of the 26 members of the society had signed a document saying they would not take the observers out, but when faced with a $250,000 fine "some of these guys backed down".
Fishermen at Taieri Mouth and most in Bluff were still refusing, he said.
Ministry of Fisheries national environment manager Steve Halley said there had been very good co-operation from "by far the majority", but if fishermen deliberately ignored the legal requirement to take an observer, the ministry had little choice but to consider action.
"We are investigating a small number of fishers."
Mr Little said the programme had also brought to the surface unresolved issues surrounding by-catch which meant it was not in the fishermen's "best interests" to have observers on board, he said. Any video footage taken could lead to prosecution down the track.
There was also the "grey area" over whose liability it was if an observer was injured, he said.
Taieri Mouth fisherman Gary Homan said he had never seen a dolphin in the Otago fishing area so he was happy to take out an observer, but was concerned doing so could put him in danger of prosecution for throwing back quota species such as paddlecrabs, for which there was no market.
"We can't avoid catching it. We're doomed to be prosecuted. We do not know what to do. It's an impossible situation."
New Zealand Commercial Fishing Federation chief executive Peter Dawson said there were concerns from fishermen around the country at the impact of requiring observers on in-shore vessels.
The industry was being levied $6 million over three years to fund the observer programme and fishermen also faced extra costs feeding and housing them.
Skippers of small fishing boats could find it difficult to find space or the required safety equipment for an extra person on boat, he said.
Mr Halley said the observer programme was solely aimed at looking at the interaction between fishing gear and marine mammals and that was all the observers were trained to do.
"They are not fisheries officers."
Fishermen's safety concerns had been "taken seriously". The ministry had consulted Maritime Safety about putting observers on small vessels and individual skippers, he said. Observers were issued with their own safety equipment and were trained in vessel operations and work at sea.
"We understand safety is an issue and we do not want to put any vessel at risk."
At the end of the programme, it would be assessed to see if there was a way to continue assessing the overlap between fishing and dolphins a different way, Mr Halley said.
