In 80 minutes of gripping drama at Tokyo Stadium, the Six Nations champions built up a 26-8 lead soon after halftime after tries from Hadleigh Parkes and Gareth Davies and drop goals from Dan Biggar and his flyhalf replacement Rhys Patchell.
Australia, fired by their replacement flyhalf Matt To'omua, hit back with second-half tries from Dane Haylett-Petty and captain Michael Hooper and came within a point at 26-25 late in the match.
The Welsh held on, though, and Patchell, who contributed 14 points from the boot, steered them home to only a second win in their last 15 matches against Australia and potentially an easier path through the knockout stages.
Wales had got off to a blistering start, turning over the ball from the kickoff and setting up Biggar to put three points on the board with a drop goal after 36 seconds.
It got better for the Welsh after 13 minutes when, with the referee playing advantage, Biggar put in a crosskick and Parkes leapt above Marika Koroibete to grab the ball and touch down.
Biggar added the extras and missed another drop goal but Wales had struggled to contain the 17-stone Samu Kerevi and a charge from the big centre got the Australians within striking distance.
With the referee again playing advantage, veteran flyhalf Bernard Foley put up a higher, shorter crosskick than Biggar's but it found Adam Ashley-Cooper sprinting in from the right wing and the 35-year-old got past two defenders to score.
Foley missed the conversion but added a penalty before Biggar went off after taking a head knock tackling Kerevi and his replacement Patchell quickly replied with a three-pointer from the kicking tee to extend the lead to 13-8.
Three minutes later, in an incident that is sure to provoke plenty of debate, Kerevi was penalised near the halfway line for leading with his forearm as he ran with the ball into Patchell's tackle.
The flyhalf converted the penalty off the post and if that was a blow to Australia, a heavier one was to follow two minutes later when Davies intercepted a pass from his opposite number Will Genia and sprinted away to score.
Trailing 23-8 at the break, the Wallabies were desperate to score first in the second half but the errors continued and Wales edged three points further ahead when Patchell slotted a drop goal of his own.
To'omua came on at flyhalf for Australia in place of Foley, though, and they finally got the passes to stick with a lovely sweeping move that finished with fullback Haylett-Petty touching down.
The Wallabies had their heads up and laid siege to the Wales line for a good 10 minutes until Hooper managed to force the ball through the line of red shirts and touch down for a converted try that cut the deficit to four.
To'omua and Patchell the traded penalties to set up a dramatic finale, with the last eight minutes played out in the gloom after some of the floodlights failed.
Australia launched themselves time and again at the Welsh defensive line but they just could not find a gap and one final knock-on allowed Wales, so resilient in defence, to cling onto the ball until the final gong sounded.