Mahathir's stunning defeat of the ruling coalition that has ruled the Southeast Asian country since independence from Britain six decades ago means that, at the age of 92, he will become the oldest elected leader in the world.
Official results showed that Mahathir's Pakatan Harapan (Alliance of Hope) had won 112 of parliament's 222 seats, clinching the simple majority required to rule.
Najib's ruling coalition, Barisan Nasional (BN), had 79.
Two more seats remained to be announced.
Mahathir told a news conference he expected to be sworn in as prime minister later on Thursday.
"The time for change has come, and I hope the people in power realise this," said Asifa Hanifah, a young woman who joined thousands of opposition supporters in central Kuala Lumpur who waved flags, cheered and honked car horns.
Few had expected Mahathir to prevail against a coalition that has long relied on the support of the country's ethnic-Malay majority.
However, he joined hands with his one-time protege, the jailed politician Anwar Ibrahim, and together their alliance exploited public disenchantment over the cost of living and a multibillion-dollar scandal that has dogged Najib since 2015.
Mahathir has promised to seek a royal pardon for Anwar if they win the election and, once Anwar is free, to step aside and let him become prime minister.
Several key roads in the heart of the capital, where violence between races has played out in the past, were blocked off by police as evidence grew that Najib's coalition was on the back foot. In a statement, the police appealed for calm and said that for now the situation was under control.
Najib's United Malays National Organisation (UMNO) party postponed an evening news conference and said Najib, who has ruled the country for nearly 10 years, would address the media at 9:45 a.m. (0145 GMT) on Thursday.
Malaysia's currency weakened in offshore trading on the election result, with the ringgit one-month non-deliverable forward falling 2.4 percent to 4.07 against the dollar.
'History in the making'
The reverse for UMNO, the dominant partner in BN, takes Malaysia into uncharted political terrain, said Keith Leong, head of research at the KRA Group consultancy. "We are witnessing history being made in this country," he said.
Ethnic-Malay Muslims have long tended to support BN for affirmative-action policies that give them government contracts, cheap housing and guaranteed university admissions.
Mahathir's alliance, which counts on urban votes and support from the minority ethnic-Chinese and Indian communities, had hoped the veteran Malay leader would win over voters usually loyal to BN. That strategy appeared to have paid off.
"There has been a significant shift in the Malay vote," said Rashaad Ali, an analyst with the S. Rajaratnam School of International Studies in Singapore.
Mahathir is a polarising figure and many voters are suspicious of him because of his iron-fist rule as prime minister from 1981 to 2003.
But 64-year-old Najib's popularity dropped sharply over the past three years, partly due to a scandal over 1Malaysia Development Berhad (1MDB), a state fund from which billions of dollars were allegedly siphoned off.
Mahathir was once Najib's mentor, but he left UMNO over the 1MDB affair and joined the opposition. Najib, who was chairman of 1MDB's advisory board, has denied any wrongdoing and he has been cleared of any offence by Malaysia's attorney general.
In an even more unlikely change of heart, Mahathir last year buried a feud with Anwar, 70, and the two agreed to join forces to topple Najib.
Mahathir sacked Anwar as his deputy prime minister in 1998. Anwar then started a movement known as 'Reformasi’ - reform - to end UMNO's race- and patronage-based governance, but he was stopped in his tracks by charges of sodomy and graft, which he denied, but for which he was jailed.
Anwar was imprisoned again in 2015, when Najib was prime minister, after another sodomy charge, which he described as a politically motivated attempt to end his career.
