Mendis had a spell of six for 16 to help Sri Lanka earn an eight-run win.
Batting first, Sri Lanka scored 157-9 in 20 overs with Mahela Jayawardene hitting a crafty 86 at Pallekele International Stadium. In reply Australia managed just 149-9 in 20 overs.
Australia appeared to be coasting, having scored 71 runs by the sixth over, with Shane Watson hitting five sixes and six boundaries for 57 runs off just 24 balls, but Mendis pulled the hosts back into the game with three quick wickets.
Mendis, who was dropped for this year's tour of England, said he worked hard to win his place back. He said he tried to bowl line and length and do the variations right.
"This is the result of that effort," he said.
Australia's captain Cameron White was disappointed that the team let a likely win slip away.
"It's a shame that we got off to a flyer and let it off," said White.
Watson top-edged a ball for a high catch to Thisara Perera and David Warner (16) was dismissed in a spectacular fashion when Angelo Mathews threw back a ball into the field to stop himself from going over the line with the ball and conceding six runs . But Jayawardene was quick to catch the ball that deflected off Mathews' hands.
Shaun Marsh was out for a duck stumped by Kumar Sangakkara off Mendis. Left-arm spinner Rangana Herath bowled David Hussey to leave the visitors 75-4.
White resisted with 39 runs only to see his partners deserting him at crucial moments.
Mendis also bagged the wickets of Steven Smith (12), Brad Haddin (0) and Mitchell Johnson (7) to record the best bowling figures ever in the shortest form of cricket - surpassing Pakistan seamer Umar Gul's five wickets for six runs recorded against New Zealand in 2009.
Earlier, Jayawardene hit a six and 12 boundaries during a 64-ball innings to propel the hosts to a competitive score. It was his sixth Twenty20 half-century.
Australian seamers John Hastings and Brett Lee took three wickets apiece.
Lee dismissed Sri Lankan skipper Tillakaratne Dilshan, who made 104 in the first match, for four runs. Dilshan was trapped lbw after he missed a straight delivery with the total on 23. Jayawardene and Dinesh Chandimal shared 43 runs before Chandimal (13) was caught and bowled by leg spinner Smith.
Thisara Perera (9) one of the two changes in the team for the match, was caught by Watson at deep third-man off seamer Johnson.
Jayawardene and Sangakkara combined for a useful 46-run stand for the fourth wicket. Sangakkara made 24 before being caught by White off Hastings.
Hastings also bowled Jeevan Mendis (1) and Anjelo Mathews (5) to finish with best bowling figures 3-14. Lee returned 3-39.
Sri Lanka made two changes to its side from the first match dropping seamers Dammika Prasad and Suranga Lakmal and replacing them with allrounder Perera and spinner Ajantha Mendis.
Sri Lanka also won the first match by 35 runs on Saturday.
