A 57-45 caucus vote in favour of the former prime minister was the third time the Labor leadership had been tested since Ms Gillard ousted Mr Rudd to become Australia's first female leader in June 2010.
The decision by Labor powerbroker Bill Shorten, one of the so-called "faceless men" of the 2010 coup against Mr Rudd, to switch allegiance to him was a crucial step in her ousting on Wednesday.
Mr Rudd could face his first parliamentary test on Thursday, if the opposition moves a motion of no confidence in the minority government on the final sitting days of the 43rd parliament.
However, Opposition Leader Tony Abbott talked down its prospects, saying Mr Rudd should instead name an election date as quickly as possible.
"It is up to the prime minister to demonstrate that he has sufficient support to confirm and maintain a government ... and tell us when the election will be," Mr Abbott said.
The new leader is likely to receive the support of crossbenchers Andrew Wilkie, Peter Slipper, Craig Thomson, Bob Katter and Adam Bandt if it came to a confidence vote in the house.
Mr Rudd said he was resuming the prime ministership, which he first secured at the 2007 election, with a sense of "energy and purpose".
"I simply do not have it in my nature to stand idly by and allow an Abbott government to come to power in this country by default," he said, adding the opposition leader was "steeped in the power of negative politics".
If confirmed as prime minister by Governor-General Quentin Bryce, Mr Rudd will have wide scope to put together his government's cabinet, after deputy prime minister Wayne Swan quit the position and was replaced by Anthony Albanese.
Ministers Swan, Stephen Conroy, Joe Ludwig, Craig Emerson, Peter Garrett and Greg Combet also resigned their ministerial roles. Dr Emerson and Mr Garrett also decided not to contest the upcoming election.
Ms Gillard, who brought on the leadership spill after a petition started to circulate seeking a special caucus meeting, said in her resignation statement Labor needed to put the divisions of the past behind it and unite.
She had faced the twin problems of a minority parliament and "internal division" within Labor.
"It has not been an easy environment to work in," Ms Gillard said.
Ms Gillard won't recontest her Melbourne seat of Lalor at the upcoming election.
Mr Swan, who will recontest his Brisbane seat of Lilley, described Ms Gillard as one of Labor's "toughest warriors" who had achieved much in the past three years.
Mr Rudd had previously stated that under no circumstances would he return to the leadership.
But he said he had gone back on his pledge for three reasons.
"The request that I have received from my colleagues, my belief that the Australian people deserve a competitive choice at the next election and my fear that if we don't offer it Mr Abbott will win by the biggest landslide since Federation," he said.
Finance Minister Penny Wong unanimously becomes Labor's Leader in the Senate, taking over from Senator Conroy. Jacinta Collins becomes the senate deputy leader.
The Liberal party immediately launched a new advertisement campaign quoting Mr Rudd's "broken" promise not to contest the leadership.
"I will leave it to you, the Australian people, to judge if I have made the right call," Mr Rudd said earlier.
Mr Abbott said voters should choose the prime minister, not the "faceless men" of Labor.
Asked when he would be sworn in as prime minister, Mr Rudd said he would await advice.
"We will await the Governor-General on that matter," he said.
