Heavy rains hit the islands early this week.
On Tuesday, 183mm of rain fell in 24 hours.
Muttonbirder Michael Skerrett, of Invercargill, said it was very fortunate no-one was in the houses, as normally his father would be there at this time of year.
"If he had, he'd have had no show but as he's been a bit ill, he's a bit later going down."
While Mr Skerrett had not seen the event, he had heard the landslip had taken his brother's "work" house down a cliff and on to the shore below "smashing it to smithereens", he said.
A "living" house behind it had also moved down the cliff.
Relatives Allan and Karen Skerrett were nearby when the slip happened and had a narrow escape, he said.
"They heard a loud crack, a whoosh and it filled the gully.
They were covered up to their shoulders."
Another house at the other end of the main island had also been damaged by a slip but to what extent, he did not know.
His nephew had travelled down to the islands to assess the damage and he had sought advice from GNS Science about the stability of the land to advise those on the island.
GNS had advised them to keep off the site while it was still raining and if they needed to retrieve anything from it, to do it quickly, with people keeping a watch from a safe distance, Mr Skerrett said.
Despite the damage, he did not think the event would deter any of the muttonbirders, although it might delay the start of the season until the weather dried up.
"We're pretty resilient people.
"It is one of those things.
"We'll all get stuck in and help each other."
