The basic rate of tax could be cut by at least 2p if individuals paid more of what they owed, Chief Secretary to the Treasury's Danny Alexander has said.
He said the rest of the tax-paying population had to pick up the cost when the wealthy avoided their obligations.
His comments to the BBC come days after comedian Jimmy Carr apologised for taking part in a tax avoidance scheme.
And former PM Tony Blair said the public mood on tax avoidance had changed in a time of austerity.
Prime Minister David Cameron publicly criticised Mr Carr for being involved with the K2 tax avoidance scheme - which the comedian has now left.
But speaking on the BBC's Sunday Politics show, Mr Alexander said that "as a Treasury minister I would not get into the details of any individual's tax affairs. We have taxpayer confidentiality in this country".
Mr Alexander went on to say: "But what is very important, and what I very strongly agree with the prime minister about... is that the kind of aggressive tax avoidance that has been well reported over the course of the week, the government has been dealing with since we came to office, is something that has to be cracked down on."
He said: "These sorts of schemes that save wealthy people potentially tens of millions of pounds in tax, they are paid for by everybody else.
"If we could narrow the tax gap in this country by a quarter we could reduce income tax for every basic rate payer by 2p in the pound."
It was important "everybody, especially the rich and famous, are paying their fair share of tax", he added.
More than 1,000 people, including Mr Carr, are thought to be using the Jersey-based K2 scheme, which is said to be sheltering £168m a year from the Treasury.
On Thursday Mr Carr said he had "made a terrible error of judgement" over using the scheme. It is legal and Mr Carr made clear in his statement on Twitter that it was fully disclosed to HMRC.
Mr Blair told the BBC's Andrew Marr show that the public were only willing to tolerate such avoidance in more prosperous times.
"I think the mood on this has changed. What people maybe would not have cared about a few years back, if you are in a time of austerity people they care about these things."
International Development Secretary Andrew Mitchell, who also appeared on the programme, said the government's message was that "everybody should pay their tax... not least at a time like this when money is incredibly tight".
Meanwhile, viewing figures for Friday's episode of Jimmy Carr's Channel 4 show 8 Out of 10 Cats dramatically increased, the ratings show.
It was Carr's first TV appearance since being criticised for taking part in the K2 scheme.
An average 2.5 million viewers watched the comic being taunted by the show's panelists, peaking at 2.7 million.
The figure is almost double the ratings for the week before and is the show's highest figures since January 2010.