In last year's Budget, Mr English redirected $2 billion of the lower-quality spending, taking the total of redirected money to $3.8 billion by the time the May 20 Budget is released.
"That's a significant sum of money we're making available for priority areas such as better health care services, better education and keeping New Zealanders safe.
"The Government will continue to weed out low-quality spending," he said in a speech to the Wellington Regional Chamber of Commerce yesterday.
The Government would live within the $1.1 billion annual operating allowance for new spending it had set itself and restrict annual increases in that figure to 2% from 2011-12.
Most Government agencies would receive no budget increases over the next three or four years as the Government moved to get to budget surplus as soon as possible, Mr English said.
The work done so far showed there was considerable scope to provide better public services by improving process, removing duplication and reallocating resources from low-quality spending to improve frontline public services.
Public service chief executives were coming to terms with the new reality, he said.
The Government had given them time to prepare their agencies to provide better public services with little or no new money over the next three or four years.
"I want to get the Government back into budget surplus as quickly as possible, because surpluses give us choices."
The choices including whether to invest more in public services, pay down debt or resume contributions to the New Zealand Superannuation Fund.
Mr English remained coy about what New Zealanders could expect in his second Budget under Prime Minister John Key.
He reiterated the much-repeated aims of lifting the long-term performance of the economy, reforming of the tax system, better delivery of public services and maintaining firm control of the Government's finances.
He touched on tax reform which had been the subject of considerable public and media debate.
"I don't think I've seen such an open and constructive process around such an important policy area before."
Tax changes would be announced in the Budget and while not pre-empting that announcement, Mr English said there was a compelling case to rebalance the tax system and support the goal of tilting the economy towards savings, investment and exports away from borrowing, consumption and investment housing.
Taxation had a pervasive influence on both the economy at large and on decisions made by individuals.
"Remember, we have an economy where we're spending more than we earn. Tax is one way to change people's choices and help turn that around.
"Any tax changes need to contribute to a better-performing economy, more jobs and higher incomes for families."
The starting point for the Government was that lower personal taxes across the board were a good thing because they gave people incentives to work hard, improve their skills and get ahead here in New Zealand.
New Zealand had that opportunity at a time when many other countries would be forced to increase taxes. That was an important competitive advantage for New Zealand, Mr English said.
The tax package would be broadly cost neutral. The Government could not afford a lolly scramble.
Any increase in GST would be accompanied by immediate compensation for low and middle-income earners, beneficiaries, superannuitants and people receiving Working for Families.
Additional regular adjustments for other ongoing inflationary pressures would be provided as usual.
In addition, under the tax package being considered, superannuitants and people on lower wages would also receive income tax cuts.
"As we've said, we are looking at income tax cuts across the board, not just for people on the top marginal rate.
"It will be important on Budget day to look at the tax package as a whole rather than individual components in isolation," Mr English said.
Labour Party finance spokesman David Cunliffe said National seemed set on slashing more vital public services to fund tax cuts for its rich mates.
"In Mr English's own electorate [Clutha-Southland], thousands of elderly are having their home support they rely on cut so that rich earners like Mr English can pocket an extra $300 a week. The elderly in his electorate apparently represent low value in terms of services."
