The ferry, belonging to the Hong Kong Electric Company, was taking staff and family members to watch a fireworks display to celebrate China's National Day and mid-autumn festival when it hit the other ship and quickly began sinking near Lama island.
Survivors said they had little time to put on life jackets before the ferry flooded, trapping passengers.
"Within 10 minutes, the ship had sunk. We had to wait at least 20 minutes before we were rescued," said one male survivor, wrapped in a blanket on the shore.
Some survivors said people had to break windows to swim to the surface.
"We thought we were going to die. Everyone was trapped inside," said another middle-aged woman.
The other ship, owned by Hong Kong and Kowloon Ferry Holdings, made it safely to the pier on Lamma, an island popular with tourists and expatriates. It had a damaged bow and several of its passengers and crew were taken to hospital with injuries.
The tragedy was the worst to hit Hong Kong since 1996 when more than 40 people died in a fire in a commercial building.
SEARCH FOR SURVIVORS CONTINUES
Hong Kong is one of the world's busiest shipping channels, although serious marine accidents are rare. It is unclear why the two ferries collided.
"Our ferry left Lamma island at 8.15pm (local time) to watch the fireworks display out at sea, but within a few minutes, a tugboat (ferry) smashed into our vessel," Yuen Sui-see, a director for Hong Kong Electric.
A spokeswoman for Hong Kong and Kowloon Ferry said they were assessing what had happened.
"Our captain is not well and we have not been able to talk to him so far," the spokeswoman told local television.
The night-time collision sparked a major rescue operation involving dive teams, helicopters and boats that saw scores of people plucked from the sea.
Television pictures showed the red and blue bow of the Hong Kong Electric Company ferry pointing skywards, surrounded by rescue vessels.
"We will continue our search. We also don't rule out that some may have swam to shore themselves and haven't contacted their families and so may not be accounted for," Ng Kuen-chi, acting deputy director of fire services told local television.
The search was hampered by the vessel being partly sunken, poor visibility and too much clutter inside the vessel, Ng said.
Teams of men in white coats, green rubber gloves and yellow helmets carried corpses off a police launch in body bags.
Local media reported that children were among the dead.
More than 100 people were sent to five hospitals and nine people suffered serious injuries or remain in critical condition, the government said in a statement.
Hong Kong leader Leung Chun-ying visited survivors of the collision and pledged a thorough investigation into the crash.
Thousands of Hong Kong residents live on outlying islands such as Lamma, which lies about 3km southwest of Hong Kong island.
