Southland Community Housing Group member Margaret Cook said the housing situation in the southern city and surrounding region had moved from ''manageable'' to ''quite dark''.
The group hosted a public meeting on Monday to discuss the problem and what action was needed to address it.
Information collected by the group in April showed 500 people were living in tents, camper vans, garages or on the street.
While there was a housing crisis nationally, the problem in Southland was larger than many realised, Mrs Cook said.
''Those numbers included an increasing number of women in the 40 to 60 age group who said they felt safer on the street than in designated emergency houses.''
At least 150 community houses needed to be built, but only about 40 were planned.
The Southland Regional Development Strategy group's goal of 10,000 more people living in the region by 2025 would also add to the challenge.
As only a small number of houses had been built since the announcement in 2017, Invercargill would have to build two houses a day to reach the goal, she said.
A site in the Kew Bowl area had been bought by Habitat for Humanity Invercargill on behalf of the Southland Housing Action Forum.
Between 20 and 30 houses would be built at the site, but the type of housing would be decided after the community was consulted.
Forum chairman Shaun Drylie believed the main problem was a lack of suppliers and ''understanding how we can make the system more efficient''.
It was important to identify what the real need was and provide opportunities to attract developers and suppliers, Mr Drylie said.
An apartment complex to help with the shortage was suggested at the meeting as a possible solution. However, Mr Drylie believed it would be a challenge.
''People here think it is a great idea for everybody else, but they don't want to live in it.''
Apartments would be good for people who wanted to rent or move to the city for a fixed period of time.
''The apartment market is not mature here.''
