A spokesman for judges said it might be the first time the issue had arisen in a New Zealand courtroom.
Catherine Owen was seated near the front of the public gallery on Tuesday with her daughter, supporting her partner who was due to appear in court.
No other members of the public were close to her.
However, she was in the judge's direct line of sight.
The baby began to cry, so Ms Owen started breast-feeding her.
Shortly after, Judge Phillips asked: "Why is there a baby being breast-fed in my courtroom?", at which point a court bailiff approached Ms Owen, who had already started to leave the court.
"I'm glad I heard him say what he said before the bailiff came over to me," Ms Owen said.
It would have been a lot more embarrassing to have been escorted by the bailiff, she said.
She remained in the public waiting area until her partner's name was called, but decided not to speak to court staff about the incident.
"There's not much point making a massive deal about it. It's his courtroom and it doesn't [adhere] to the same rules.
"A few people have said I should lay a complaint."
She had decided not to take the matter further, but said the embarrassment was caused by "the way he went about it".
Senior judicial communications adviser Neil Billington, of Wellington, said yesterday judges had a "legal authority" to run their courtrooms "in a manner that they consider appropriate to the administration of justice".
"It's based on a legal power that each and every judge has ... Each judge is independent in terms of how they exercise that authority.
Mr Billington said while Judge Phillips had the authority to request Ms Owens leave the courtroom, "I'm not aware an issue of this nature has arisen before".
"Although courtrooms are open to the public, they're not 'public places' in the usual sense ...where members of the public are free to do whatever they like."
A Ministry of Health website states that "under the Human Rights Act, it's illegal for someone to stop you breast-feeding in public".
Mr Billington said Judge Phillips would not comment on his decision "and that's the usual situation".
While the Office of the Judicial Conduct Commission dealt with complaints relating to judicial conduct and matters of law, the incident fell outside that jurisdiction, Mr Billington said.
"The mere decision to require someone to leave ... is not a conduct issue; that's the judge exercising his authority."
A breast-feeding support worker in Queenstown, who preferred not to be named, said she was unfamiliar with the rules governing court rooms but felt if a baby was permitted in court then feeding the baby should also be permitted.
She said if the sound of a baby crying was disrupting the court, then breast-feeding was "a wonderful way to minimise disruption".
