Mr Swiggs has been shrouded in controversy amid allegations he sent inappropriate messages to young people.
He lost his seat to People's Choice candidate Jake McLellan by 1078 votes.
A spokeswoman for Mr Swiggs said he was disappointed with the results.
Cr Swiggs has vehemently denied the allegations, which came to the attention of media when voting packs arrived on in voters mailboxes last month.
The city council appointed retired associate Justice of the High Court to look into the allegations. He dismissed three as they were outside the term of office, but found two were "material" requiring further investigation. One was referred to "another agency", which Cr Swiggs has previously said was the police.
But Cr Swiggs released some of the messages subject to the complaints after he said Mr Matthews had not made his decision with "fair consideration of all information".
And last week prominent businessmen including developer Richard Peebles and hospitality and tourism leader Peter Morrison threw their support behind Cr Swiggs.
