Opposition communications spokesman David Coleman also took aim at the federal government's proposed misinformation bill on Sunday.
Under the changes, Labor wants to bolster the communications watchdog's powers to monitor and regulate content on digital platforms.
Social media companies could be fined up to five per cent of their annual turnover under the Commonwealth's proposed laws.
Musk, who owns social media platform X, responded to a post about Australia's measures with the word "fascists".
Prime Minister Anthony Albanese said social media companies had social responsibility.
"If Mr Musk doesn't understand that, that says more about him than it does about my government," he said on Saturday.
The billionaire doubled down on his comments over the weekend in another post, captioned Far left fascists love censorship.
Under the legislation, the media regulator would be allowed to approve an enforceable industry code of conduct or introduce standards for social media companies if self-regulation was deemed to fail.Most Australians want more action to stop the spread of harmful misinformation online, surveying by three universities shows.
Opposition communications spokesperson David Coleman steered clear of the "fascists" label.
"I wouldn't use those words... I would say contemptuous of free speech," Mr Coleman told Sky News on Sunday.
He said he took issue with the way intentionally misleading and harmful material was defined.
