ACT and NZ First had called for her to resign as Minister for the Prevention of Family and Sexual Violence over the comments.
It comes after anti-transgender activist Posie Parker, also known as Kellie-Jay Keen-Minshull, left New Zealand on Sunday without attending her Wellington speaking event, after her Auckland rally was overwhelmed by hundreds of counter-protesters.
With the crowd making too much noise for her to be heard, Parker was escorted away from the event by security before she could speak. When she appeared, she had tomato juice flung at her, and was pushed and shoved.
Davidson, when approached on the street by Counterspin media host Hannah Spierer after the protests on Saturday, was asked about Parker being violently assaulted.
Spierer and her husband Kelvyn Alp are far-right activists, and their network Counterspin has previously been linked to Steve Bannon, Donald Trump's former chief strategist who also ran far-right website Breitbart News and was connected to the Cambridge Analytica scandal.
Davidson said she was there to reject violence and hate and to "raise love for our trans people and community".
"Trans people are tired of being oppressed and discriminated," she said. "I am the (violence prevention) minister - and I know who causes violence in the world - it is white cis men. That is white cis men who cause violence in the world."
Davidson was also knocked to the ground by a motorcyclist during the protests on Saturday.
In a statement on Monday afternoon, she said she had still been in shock from the crash at a pedestrian crossing, and was "not as clear in my comments to the conspiracy theorists Counterspin as I should have been".
"Violence is unacceptable in any community and as the Minister responsible for Aotearoa's first ever plan to eliminate family violence and sexual violence, I am committed to an Aotearoa where all people are safe and can live peaceful lives," she said.
"Women are overwhelmingly more likely to be victims of family violence and sexual violence at the hands of men. It is also important to acknowledge the disproportionate impact violence has on our rainbow whānau and diverse communities.
"I should have made clear in my comments that violence happens in every community. My intention was to affirm that trans people are deserving of support and to keep the focus on the fact that men are the main perpetrators of violence. I will continue to stand with my trans and non-binary whānau and support action to ensure that everyone can live their lives without fear of hate or discrimination."
She said she would not be commenting further at this time.
Hipkins backs Davidson's statement
Prime Minister Chris Hipkins said it was not a form of words he would have chosen but - from her statement - was not what she had meant to say.
"I think we do need to consider the context in which she made the remarks, she'd just been hit by a motorcycle and was basically being harassed by a right-wing conspiracy group as she went down the road," he said.
Hipkins said nobody in the kinds of situation at the events over the weekend should ever resort to violence.
"I don't believe people should throw things at a protest, whether what they're throwing is soup - or a brick. Ultimately the right to free speech does not extend to the right to physical violence, so I would condemn that regardless of who's engaging in that type of activity," he said.
"I think what we did see over the weekend though was a significant number of New Zealanders exercising their right to free speech, doing so in a way that was respectful and that was about supporting other fellow New Zealanders, and I think that is something we should celebrate."
He said questions over the police response would be best put to police.
Calls for Davidson to resign
NZ First leader Winston Peters that morning had called for Davidson's resignation from her ministerial role over the comments, saying her remarks were "offensive, racist and sexist" and "peak madness".
"Marama Davidson should stand down as Minister for the Prevention of Family and Sexual Violence for the clear and outrageous statement she made at the Posie Parker protest that white straight men are the cause of violence," he said.
"The prime minister must have higher standards than shown by Minister Davidson. New Zealanders neither want nor deserve to be looked upon and insulted by our so called 'representatives' in this way. Is it any wonder there was violence shown at the protest when we have a minister outright supporting it."
Peters made a scene-setting speech last week railing against a secret "woke agenda" ahead of his election campaign this year.
ACT leader David Seymour also said Davidson should "front up with evidence for her statement about white men" and if she could not, should resign.
"Davidson's comments are outrageous and extremely inappropriate for a minister that is responsible for reducing family violence and harm," he said. "Family violence and harm does not discriminate and her obvious bias against one race is deeply troubling."
He said no reasonable person could credibly claim Davidson's comments exercised a professional approach and good judgement, as required by the Cabinet Manual, which sets out rules ministers must follow.
National Party leader Christopher Luxon had joined in on criticism in the afternoon, saying her statements were "wrong and offensive".
"Marama Davidson is making harmful generalisations about an entire group of people," Luxon said. "Chris Hipkins needs to decide what sort of standards he accepts from his ministers. If he thinks this is acceptable, he is setting the bar very low."
Parker had been allowed to enter New Zealand after a challenge to Immigration Minister Michael Wood's decision to allow her entry failed in the High Court on Friday.
Wood had said he condemned her views and would prefer it if she never set foot in New Zealand, but the case did not meet the threshold for ministerial intervention.
Despite Parker not making her Wellington appearance, more than 1000 trans activists and supporters still showed up at Civic Square, with other demonstrations held in Christchurch, to show support for trans rights.