Now Zuru co-founder Nick Mowbray is warning that an AI version of himself was used in an attempt to "recreate me" and dupe Zuru’s chief financial officer, Christian Pellone, on a Microsoft Teams video call.
"This is unbelievable. The sophistication is something everyone needs to be aware of. Mind-blowing," Mowbray warned his followers on LinkedIn this morning.
Mowbray — who told the Herald his AI fake was trying to dupe his CFO into transferring cash — posted screenshots of messages leading up to the appearance of the fake video version of himself, plus a screengrab of Pellone’s explanation.
"Hi Christian, it’s Nick. Are you available for a team’s call?" the exchange began.
Pellone appeared to have got a message on his mobile phone from his boss.
He replied, "Sure Nick. Now?"
Mowbray sent a Teams invite and Pellone jumped on the video call.
"When I clicked on it, Nick was there, talking and gesturing to his ear, like ‘can’t hear you’.
Pellone was sufficiently suspicious of the messages to take screenshots of the text message exchange and the Teams invite, "but not of the AI because I thought it was real ... It wasn’t a caricature or animation of Nic on Teams. It WAS Nick!"
A screengrab of the messages shows that Mowbray displays a mobile phone number, but it is not his regular number.
In follow-up comments to the Herald, Mowbray said his AI imitator asked his CFO to transfer money — using text because he said he couldn’t get off mute (presumably a jape because the AI wasn’t sophisticated enough for a real-time voice exchange).
But "in the text exchange, he started using language that didn’t sound like me", Mowbray told the Herald.
Pellone is now building a case study for the Zuru team "to raise awareness and point to red flags".
He messaged Mowbray, "I couldn’t distinguish between real you and Teams you. Like it WAS you."
Agencies such as Netsafe and Cert NZ (the Government’s Computer Emergency Response Team) have already warned that AI has made it faster and easier for fraudsters to create new scams, and eliminate giveaways like bad English or typos. Now, Mowbray — or at least his AI evil twin — has flagged a new, more sinister turn.
The outgoing government passed no AI-related legislation, in contrast to the EU’s AI Act (due to come into force by the end of this year) or US President Joe Biden’s series of Executive Orders on Artificial Intelligence, and neither of the major parties had AI policy going into the election.
A study funded by the Law Foundation noted the rise of deepfake video, but said it could be dealt with under existing laws. Deepfake-specific legislation would risk undermining freedom of expression, it said.
