Haden's role as ambassador for the 2011 World Cup is now in doubt after comments he made on television.
"Once they've recruited three, that's it. That's their ceiling. Three darkies... no more," said Haden, who played 117 matches for the All Blacks between 1972 and 1985, including 41 tests.
"In the Crusaders manual, there it is, it's enshrined in their articles, and they've stuck by that. And they know damn well that that's the case. And it's worked."
Mr Key said Sports Minister Murray McCully would meet Haden over the weekend.
"I'm sure they will be having a discussion, and we will see where that leads," he told reporters.
But Mr Key was clearly unimpressed.
"My view on that is that the comments are not only factually incorrect. They are also offensive."
Earlier today, Green Party MP Kevin Hague said Haden had been dismissive of non-white players in the past and said he should be sacked from his ambassador role.
"Andy Haden was a poor choice for a rugby 2011 ambassador due to his organisation of the 1986 Cavaliers tour of South Africa - at the height of South Africa's apartheid era," he said.
Mr Hague said Haden had more recently been dismissive of the significance of apologies from rugby officials in both South Africa and New Zealand over the past treatment of Maori players blocked from touring South Africa.
"If you think about who we want as ambassadors, we don't want people who will bring New Zealand into international disrepute, and that's what Andy Haden will do," Mr Hague said.
Speaking on Radio New Zealand today, Haden stood by his comments, but said it was in reference to Pacific Island players as opposed to Maori players.
He admitted he had probably overstepped the mark by claiming the Canterbury union actually had a written policy on Pacific island players, but said the issue of race-based selection policies did exist.
The offending comments were made during a panel discussion on Sky TV's Deaker on Sport programme on Wednesday.
Crusaders management have strongly denied Haden's claims which also sparked a strong response from New Zealand Rugby Union (NZRU) chief executive Steve Tew.
Haden told Radio New Zealand this morning he had embellished what another former All Black skipper Chris Laidlaw had said in a recently published book.
An unrepentant Haden clarified he was not talking about the exclusion of Maori players from the Crusaders but about Pacific Island players and admitted that he had probably overstepped the mark by saying the policy was "in the manual and enshrined in their articles".
"Yes, (the policy) set in stone and ..., that's wrong, but the principle remains and this is an issue for New Zealand rugby."
He said where this policy was most obvious was in the make-up of the Canterbury academy.
Tew yesterday questioned Haden's suitability as one of six World Cup ambassadors.
"The decision to employ those ambassadors was the minister's - those comments might make one reconsider that appropriateness."
Haden said this morning the issue he had raised had nothing to do with his World Cup role.
Crusaders chief executive Hamish Riach said Haden's claim were untrue.
Crusaders coach Todd Blackadder told NewstalkZB he had been involved with the Crusaders a long time, and he had never seen any evidence Haden was right.
It was unfortunate the comments had been made because they brought the game into disrepute and hurt people and reputations, he said.
The Crusaders team, beaten in the Super 14 semifinals by the Bulls last weekend, included Pacific Islanders Kahn Fotuali'i, Robbie Fruean and Ti'i Paulo.
