Blair Kinghorn had the chance to win it for the home side in the final minute of Saturday's clash in Edinburgh but the five-eighth's penalty strike sailed wide.
The Australians hadn't won at the famed ground since 2016, with the victory a massive morale boost as they open their tough five-test tour.
While they were able to break their hoodoo, it wasn't a pretty performance by the visitors.
They talked pre-match about their discipline, yet conceded 15 penalties with Taniela Tupou pinged at the breakdown to give Kinghorn his winning opportunity.
Wallabies coach Dave Rennie admitted he was "relieved" that they had hung on for the victory.
"We got ourselves in a bit of a hole after halftime, an opportunist try and we're down 15-6 but the boys showed a lot of character and fought hard for each other and managed to hang on," Rennie said.
"We've worked really hard and it's pleasing to get the tour off with a win.
"We've got a lot more in us - we need to be a hell of a lot more clinical when we're down their end as we had plenty of opportunities."
The Australians struggled with Scotland's ferocity at the breakdown early on despite the best efforts of Michael Hooper, who was back in the line-up for the first time after a three-month mental health break.
Scotland opened the try-scoring when fullback Ollie Smith showed his footwork to dot down in the 11th minute.
But Australia took a 6-5 halftime lead with Bernard Foley landing a 41st-minute penalty.
The Scots raced ahead early in the second half through a brilliant Kinghorn try.
Off a Wallabies loose pass, Kinghorn toed the ball ahead and then kicked again, with the ball bouncing into the rangy playmaker's hands to touch down.
A penalty then opened up their lead to 15-6.
The Wallabies got back in the game when they took advantage of a Scotland being a man down for skipper James Slipper to score.
Glen Young was lucky to avoid a red card for his clean-out on Tate McDermott, with the halfback not returning to field after the head knock. He may also miss their next test against France.
The converted try closed the gap to 13-15 before McDermott's replacement Nic White drew a penalty with Foley putting his team the decisive point ahead in the 71st minute.
