Wales, shorn of the majority of their first team who have been selected to tour Australia with the British and Irish Lions, dominated possession and territory in the opening half but struggled to turn that into points with Japan, who will host the Rugby World Cup in 2019, defending resolutely.
Ayumu Goromaru made the most of Japan's rare forays into the opposing half by slotting two penalties, while Wales flyhalf Dan Biggar kicked one as the 'Brave Blossoms' went into the break 6-3 ahead.
Wales were quick out of the blocks in the second period and eventually found a hole in the disciplined Japanese defence after moving the ball right then left, with Biggar floating a lovely long pass wide for winger Tom Prydie to cross for a 45th minute try.
Japan, who have won just one match in seven World Cups, did not drop their heads and Australian Rugby League convert Craig Wing brushed through a tackle by Welsh prop Rhys Gill to cross for a try five minutes later.
Goromaru converted from out wide to put Japan 13-8 ahead and it got even better for the Asian Five Nations champions in 10 minutes later when they crossed for a second try.
The fleet-footed Japanese continued to find holes in the Wales backline, taking them into the 22 and, after sucking in defenders, created a huge overlap to allow openside flanker Michael Broadhurst to touch down in the corner.
Goromaru converted again and added a 77th minute penalty to secure Japan's biggest win to date.
Wales won the first test 22-18 in Osaka last week.
