Five people died after the boat, with 11 on board, capsized off Goose Bay.
Kevin Anderson said while the rescue was still unfolding he saw a whale heading north, diving under the water within two or three hundred yards from the vessel.
The retiree said there was nothing else near the vessel at the time and the sea was dead calm.
Anderson said he had never seen anything like it in the 20 years he had lived in the area.
"We were watching the television and because we had the glare, I had the blinds down and when we'd finished watching we put the blinds up and there's this boat upside down just out here. And there were five or six people clinging to the top of it."
Anderson said he then saw a white boat heading towards the upturned vessel and picking up all but one of the people.
The coastguard then arrived and the remaining man on the boat indicated to them with his hands that there were four or five people still under the boat.
"We could see that clearly from here we had the telescope clearly on it."
He said he spotted the whale shortly after the Coastguard arrived.
"We were just in shock, we just sat here and just watched the whole thing. Nothing we could do, we had all the emergency services down there."
Anderson said they only saw whales near their home "once or twice" a year, and they normally do not go close to boats.
"Terrible, terrible, terrible tragedy that's just out of the blue, there's nothing you can do."
Police have not confirmed it was a whale that caused the boat to flip, saying the vessel had collided with "something" but they were speaking to a number of people to find out what had happened.
However, Kaikoura's mayor Craig Mackle said yesterday that he believed the boat had hit a whale.
Earlier today, the photography society whose members were on the boat spoke of their "tragic loss".
"The Nature Photography Society of New Zealand is saddened by the tragic loss at sea in Kaikoura during a trip by society members that took the lives of five of our members," the group said on its Facebook page this morning.
"We do not yet know what caused the vessel to capsize and that is something that will be addressed by Maritime NZ in due course.
"In the meantime, our thoughts are with the families of those members of the society who have lost their lives."
Shortly after 10am on Saturday, an 8.5m boat, with 11 people on board, capsized in "perfect, flat" conditions.
They were taken to hospital before returning to their accommodation, police told media at a press conference shortly after 5pm yesterday.
The police dive squad recovered the bodies of the five people who had been found dead in the vessel, Sergeant Matt Boyce said.
"We are in the process of identifying those people and our next job is to notify their next of kin."
He said it was an "unprecedented event" which involved a significant response from emergency services and members of the public.
"This is a tragic event and police are providing support to those involved at this very difficult time," Boyce said.
The passengers were believed to be part of a bird enthusiast group who hired a charter boat to visit areas of interest.
"I believe they had come from all parts of the country," Boyce said.
A local Kaikoura resident who assisted at the scene said they knew one of the victims - who was connected to their son's school.
She went to the shore to help with the search.
"It wasn't much, but more than I was adding being at home.
It was difficult to see clearly along the coast due to the large number of seals sprawled along the rocks, she said.
"This will be a horrific time for the families and will be felt throughout our close community tonight and years to come," she said.
"We didn't even lose five people in our 2016 earthquake."
Police could not say who the charter was operated by. The skipper went to hospital but police do not believe he was injured.
The pilot of the first rescue helicopter at the scene said he saw no one in the water when he arrived.
"It's a terrible thing - it's something you don't want to see with people in the water," Kaikoura pilot Daniel Stevenson said.
Just before he arrived, a pleasure craft got to the scene and grabbed the first group of people that were outside the boat and took them back to Kaikoura.
According to Stevenson, five people were rescued, and one person was left, sitting on the upturned hull of the boat.
The coastguard arrived shortly afterwards. They jumped into the water but couldn't get under the boat and requested a diver.
"They jumped into the water out of the helicopter to see if there was anyone trapped inside the vessel," Stevenson said.
The mayor said it was a tragic event that affected many lives, not least of all family and loved ones.
"Our heartfelt sorrow and condolences to you all."
Mackle said he wanted to thank everyone involved in the rescue and recovery.
"Being able to bring everyone home is the best result in such a terrible circumstance."
Whales were resident and had been seen in the area, Mackle said.
Sea conditions at the time were "perfect, flat", he said.
Asked if the boat was overloaded, Mackle said he couldn't answer that.
The coastguard echoed that it was a "tragic operation".
"Our thoughts go out to friends and family of the deceased," the coastguard said.
Richard Hill from Cods and Crays fish and chip shop told RNZ he had worked with the skipper on the boat before.
"He's a very safe man, there's no way he'd take any risks.
''So there's just got to be something, sort of a freak of nature, I would have thought. A very sad day for Kaikoura really, just a shock," Hill said.
Tracy Phillips, Maritime NZ Principal Investigator, said the organisation had sent two investigators from Christchurch to Kaikoura.
"Any investigation activity will commence only after rescue/recovery operations have concluded. We'll be conducting a thorough investigation under the Health and Safety at Work Act and the Maritime Transport Act and we'll do everything we can to support NZ police in their investigation."
Kaikoura District councillor Lisa Bond said the incident was "incredibly heartbreaking".
The South Bay slipway was closed to the public while police responded.
A reporter on the shore at Goose Bay said at 1.45pm he could see three boats, two of which looked to be Coastguard vessels, and four helicopters in the vicinity of the overturned vessel.
Three rescue helicopters were sent to the scene, one from Wellington and two Westpac Rescue Helicopters from Christchurch.
An image on social media showed a number of people on top of the upturned boat, awaiting rescue.
