Earlier today Tarrant was sentenced to life without parole, the first time such a sentence has ever been passed in New Zealand.
In relation to the March 15 attacks last year, Tarrant admitted 51 charges of murder, 40 counts of attempted murder and one of engaging in a terrorist act laid under the Terrorism Suppression Act 2002.
The sentence has raised questions about whether Tarrant, an Australian national, could be deported.
But Prime Minister Jacinda Ardern said there was no legal basis to do so at the moment, and if there ever was, the wishes of the victims' families would be paramount.
Little told the Herald the only way an Australian could be deported was after the completion of their sentence.
"It's automatic if you're a non-citizen or non-resident. Any other way would have to be by agreement of treaty with the country of origin, and we don't have that with Australia.
"Our Corrections system is now obliged to detain him for life."
NZ First Leader Winston Peters called for Tarrant to be extradited.
"Now is the time for Australia's Minister of Home Affairs, Peter Dutton, to receive and carry out the terrorist's sentence in Australia," Peters said.
"The Islamic community and all of New Zealand has already suffered enough without having to pay astronomical prison costs to keep him safe in our prison system."
But Little said there was no statutory authority for New Zealand to simply ask Australia to take Tarrant.
"We would have to have a separate treaty or agreement. It's pretty clear people want him detained for life, so we'd need to get Australia's agreement to do that."
It would take years to get a treaty in place - which would be possible given Tarrant's life sentence - and would require new legislation, Little said.
"It's just not on the agenda at the moment."
Australia not asked to take terrorist
Australia's Prime Minister Scott Morrison has not received a request for the Christchurch mosque terrorist to serve his life sentence in Australia.
Australian-born gunman Brenton Tarrant has been sentenced to life without parole for the March 2019 mass murders.
It is the first time a full-life term has been imposed in New Zealand.
New Zealand Prime Minister Jacinda Ardern has not raised his incarceration with her Australian counterpart.
"It's normal practice that criminals convicted of these offences serve their sentences in that jurisdiction and that's my understanding of what the arrangements are," Mr Morrison told reporters in Canberra on Thursday.
"No request has been made to Australia for that to be any different."
The prime minister paid tribute to survivors of the New Zealand shootings, saying he and wife Jenny Morrison met with a group who showed incredible grace.
- additional reporting AAP
