Guy McCallum (28), of Dunedin, said yesterday he was taking a break from politics but had not relinquished his party membership.
Dr Whyte sparked controversy in a speech last week in which he likened Maori to pre-revolutionary French aristocracy, and vowed to repeal ''all race-based laws''.
Mr McCallum did not see legal consideration for Maori as special privilege but as legal rights designed to prevent historic discrimination.
Comparing Maori with French aristocrats was ''unfortunate and baffling''.
''Jamie told me he was in search of a stunt when he was here [in Dunedin last month]. He was sitting next to me as he said it, and he shrugged.''
Mr McCallum had had both flak and support from other members since quitting as candidate this week.
Some had told him he had wasted the party's time and money.
He also stood in the seat in 2011.
He did not think Dr Whyte intended to stoke racism, but that was the effect of his actions.
Mr McCallum supports Race Relations Commissioner Dame Susan Devoy's criticism of the speech, and said the situation illustrated ''precisely'' why New Zealand needed the race relations role.
His leader has called for Dame Susan's resignation.
Act's race-based campaigning was ''less premeditated'' than National's 2005 race-based campaign, and was probably a consequence of too little policy development.
He acknowledged he signed off the policy as a party board member, but said he had understood it would not be emphasised, because it was not developed enough to take account of unfairness in society, such as the way white-collar criminals were treated.
''If we want to talk about privilege in this country, that's where the issue is [white-collar crime]. It's got nothing to do with Maori legal rights, that finally Maori are beginning to enjoy along with the rest of us.''
Mr McCallum, a former Act on Campus vice-president who describes himself as an anti-authoritarian on the left-leaning side of the party, said he sensed ''very much'' that the party was moving further to the right.
Act president John Thompson told the ODT yesterday donations were ''rolling in'' to the party after it launched race as an issue in the campaign.
''Since we have come out like that, the money is rolling in.''
Apart from Mr McCallum, feedback had been positive from Act members and candidates about the policy.
He questioned why Mr McCallum was complaining, given he was on the board that signed off the policy.
On whether Dr Whyte told Mr McCallum he was planning an election stunt, Mr Thompson suggested that was ''a figment of Guy's imagination''.
''There are no stunts.''
Asked if focusing on race-based issues was reminiscent of National's 2005 campaign, which was perceived as negative, Mr Thompson said he did not remember any negative comments about the 2005 campaign.
''It actually took Don Brash from being very low in the polls, and National [from] very low in the polls, to being nearly in Government.''
Asked if he expected Act's low polling to receive a boost, he said: ''I hope to see us go up in the polls, yes, but that's not just relating to the [race-based laws policy].''
Of Dame Susan's comments, Mr Thompson said she ''should stick to sport and squash'', and had clearly not read the speech.
Dr Whyte could not be reached for comment, but he told the New Zealand Herald Mr McCallum had indicated to him he was not happy with the speech, so his resignation was not a surprise.
However, Dr Whyte was surprised Mr McCallum had gone public with his criticism of the speech and policy.
''I like Guy. He's a very nice guy. I was honestly confused by his response, because this is a long standing Act policy, but there you have it and he's resigned.
Former Act MP Hilary Calvert, a Dunedin city councillor, said Mr McCallum was sincere and well motivated, and involved in the community.
''He is the kind of person that Act needs. He is a caring, smart person.''
