Party returning officer Chris Hayes announced to reporters that Ms Gillard won 71 votes from Labor MPs, while Mr Rudd got 31 votes.
It's the biggest win in a Labor leadership ballot in 30 years.
The leadership was decided at a meeting of 102 caucus members in Canberra's Parliament House on Monday.
NSW MP Michelle Rowland, who would have supported Ms Gillard, was absent from the meeting.
Ms Gillard, who ousted Mr Rudd as prime minister in June 2010, had been expected to win with at least two thirds of the vote.
Mr Rudd resigned as foreign minister last Wednesday and announced he would challenge for the leadership after returning to Australia on Friday.
He said he was not happy with the timing of Monday's ballot, but said he stood for the leadership because he believed he was the only person capable of leading Labor to an election victory over Tony Abbott.
Both the prime minister and Mr Rudd on Sunday called for party unity following the ballot after a public campaign over the past week.
Mr Rudd has said he would not challenge again before the next election and would happily sit on the backbench.
Mr Hayes said the voting process, which began after 10.10am AEDT and was announced around 11.20am, took time because names had to be marked off by scrutineers.
There was a round of applause for the prime minister, he said.
Asked how Mr Rudd was, Mr Hayes replied "he seemed fine".
Earlier reports that the vote was 73-29 in Ms Gillard's favour were wrong, as were reports of a recount.
Mr Hayes said once the vote results were read out there was clapping and a certain amount of relief on the faces of members and senators.
"They now see this news as the vehicle to move forward," Mr Hayes said.
He said Mr Rudd told caucus after the vote he would support Ms Gillard.
"One of the major things he indicated at the conclusion of this ballot was he would be working for a unified Labor team," Mr Hayes said.
