More than 20 graves, all more than 125 years old, were vandalised with black spraypaint between Thursday and Friday mornings.
Fourteen had large black swastikas sprayed on them. Another two had "f*** Israel" and "don't f*** with us" scrawled on them.
The Israeli Embassy in Wellington condemned "the vile desecration".
"Sixty seven years after the liberation of the Jewish people from the death camps and ghettoes of Europe, expressions of blind hatred for Jews and for the sole Jewish state resurface," it said in a statement.
"This is clearly the action of a handful of vandals, motivated by hatred and ignorance, who certainly do not represent the opinion of New Zealanders.
"We trust in New Zealand's law enforcement agencies to find the vandals and bring them to justice."
The attack came as officials planned tighter security at the cemetery on the corner of Symonds St and Karangahape Rd.
Edward Bennett from the K Rd Business Association said headstones were pushed over regularly.
New Zealand Jewish Council president Stephen Goodman said there were moves to install a $250,000 security fence around the cemetery, which has long been popular with vagrants.
Mr Goodman said it was incredibly disappointing such an attack could occur in New Zealand.
"Normally these sort of attacks come after some international incident that provokes it, but I can think of nothing."
Samuel Altschul, senior rabbi from the Auckland Hebrew Congregation, said the incident was "very alarming".
"It's cowardly to attack the dead. It's a very un-Kiwi thing to do and it's most saddening. Unfortunately it happens quite frequently."
Rabbi Altschul said a new grave was last dug at the cemetery in 1886.
"Those are the people who helped found Auckland. Those were founding fathers."
Police are reviewing footage from a nearby security camera.
Mr Goodman said he had "more hope than expectation" it would yield any clues as to who was behind the attack.
Auckland Council civic contractor John Stevenson said the graffiti could be removed relatively easily from the newer marble headstones, but a specialist stonemason was required to remove it from the older "more sensitive" headstones.
Mayor Len Brown said the attack on a special part of old Auckland was abhorrent.
"This kind of vandalism has no place in our city. My thanks to council staff for their response. The council and the local board are taking steps to improve the environment in this cemetery and prevent further such attacks."
The last time a similar anti-semitic attack occurred in New Zealand was in August 2004, when a Jewish prayer house was burnt down and 92 headstones were smashed in the Jewish section of Wellington's Makara Cemetery.
