The show of defiance against the tax, which will cost households hundreds of euros a year, is a significant challenge to the government, days before it unveils a new budget.
Ireland is in the European Union's euro zone and the currency bloc's austerity policies have been a burning issue in many of the 18 member states.
A rapidly improving economy has allowed the Irish government to promise its first budget without any new austerity measures in seven years on Tuesday, but opposition groups say working people are not feeling the improvement.
Paul Murphy from the Anti-Austerity Alliance won a parliamentary seat in the Dublin South West constituency, which was vacated by a member of the governing Fine Gael party who was elected to the European Parliament.
Murphy told supporters: "Recovery is for the rich, it's for the one percent... it's not for the working class people."
His supporters chanted "No way, we won't pay," as the result was announced, a reference to a boycott of the water charges, which come into force at the start of next year.
"There is absolute fury against what the government has imposed on the people," said Martin Kelly, 50, a rail worker holding a placard calling for the government to "stop the great water heist".
"They say this is the last bit, but it's the hardest. People can't take any more," he said.
Since completing an international bailout last year, Ireland has been bucking the trend in Europe's stalled economic recovery, with the government forecasting gross domestic product to grow by 4.7 percent this year.
But over one in 10 are unemployed and more than 100,000 mortgage holders in arrears in a country of 4.6 million.
In the election 57 percent voted for candidates opposed to the water charges, while only 17 percent for the coalition parties. Left-wing nationalists Sinn Fein were heavy favourites to take the Dublin seat, but lost by under 600 votes.
The result represents a fall of almost 50 percent for the governing parties, Fine Gael and Labour, from the 2011 elections, said political analyst Adrian Kavanagh.
The constituency is among the most left-wing in the country, however, and centre-right Fine Gael has been the most popular party in most polls this year, with the support of between 20 and 30 percent of voters.
A second by-election in the Roscommon-South Leitrim constituency was closely poised between an independent candidate and one from centre-right opposition Finana Fail.
The Dublin rally organised by trade unions, anti-austerity groups and opposition parties snaked for miles around the centre of Dublin in the biggest show of force by anti-austerity groups in at least four years.
