The trust bought the 1870 Club Hotel and land in Gore St in 2014, in the belief it would be able to sell the building and keep the rest of the site to expand into from its present festival site next door.
But it has not been able to find a buyer for the dilapidated and deteriorating building, and was recently declined permission to knock it down.
The Bluff Oyster & Food Festival Trust has since said the decision, made by an Invercargill City Council commissioner, is putting the festival's future at risk, as it will have to reduce its present site size to provide a buffer from the dangerous buildings.
The festival was viable only at its present size.
But Heritage NZ says it wants to keep working with the trust, even though its previous suggestions had not been taken up by the trust.
Southland and Otago Heritage New Zealand Pouhere Taonga area manager Denise Anderson said the organisation had been working with the trust for several years without success, to try to come to a solution to save the building.
It had advertised the sale of the property in its newsletter, with no success.
The decision to decline the resource consent allowed the owners to explore other options, she said.
She noted the trust had bought the property in 2014, "when its condition was arguably similar to what it is today".
Options included appealing the resource consent decision (which the trust has already said it cannot afford to do), continuing to operate from the site as is, maintaining or restoring the site or moving to a new site, Ms Anderson said.
Oyster trust chairman John Edminstin said the trust had received "heaps" of suggestions about what it should do about the buildings, but none was effective or feasible as it did not have the money to improve, fix or maintain the buildings.
"We bought the property because of the land in the back. We never intended to use the building. We always wanted to sell it, but we did not have a single offer."
He said the work to strengthen the buildings could cost the trust almost $3million.
"We don't have the funds and we feel that we have done our best to find someone else to take the property.
"We never thought the festival would become so popular and big. However, now we are focusing only to ensure the success of the festival."
The Bluff Oyster and Food Festival is on this weekend.
The trust will meet after the festival to consider what to do next.
Invercargill City Council chief executive Clare Hadley said it was too soon to comment on the future of the festival, and said the council would "always be open to any discussions with the Trust."
