David Willetts has warned that there will be more cuts to higher education if too many universities opt to charge maximum tuition fees.
Fees in England are set to rise to a maximum of £9,000 from next year.
The government wants most universities to pitch their fees lower - because it faces costs from supporting students' loans.
The universities minister said savings would "reluctantly" have to be found.
Undergraduate fees are rising from their present level of £3,290 a year to a new base level of £6,000 a year - with a maximum of £9,000.
The government has said it wants the top fee to be charged only in "exceptional circumstances" but as independent bodies, universities are free to charge fees they want.
Imperial College, London has become the first to opt to charge the top rate and Oxford and Cambridge appear to be moving that way.
The government says if too many universities charge higher fees, the costs to it will be too high.
But universities believe government cuts mean higher fees are needed for them to "stand still".
Speaking at a conference in Nottingham, Mr Willetts said: "We will all face a problem if the sector tries to cluster at the maximum possible level.
"We set the maximum level at £9,000 because we think there are some circumstances where fees of this level could be justified.
"This problem arises partly because the taxpayer is lending the money upfront, on preferential terms, and we expect that one-third of the loans will never be recovered. If graduate contributions end up higher than £7,500, we would reluctantly be forced to find savings from elsewhere in HE [higher education]."
Under the new plans, universities wanting to charge more than £6,000 will have to enter into a new access agreement with the Office for Fair Access (Offa) setting out how they plan to encourage applications from poorer students and support them financially.
The agreement in effect gives them permission to levy fees above £6,000. Ministers have already said universities not meeting targets on poorer students could face losing their right to set higher fees.
The government will loan students the money to pay their fees, and they will pay it back after graduation, once they are earning more than £21,000.
While urging universities to show restraint, Mr Willetts also told the conference that "how each institution sets its prices is a matter for them, not for ministers or even Offa".
He added that the Browne Review of student funding, published at the end of last year, found that institutions needed about £7,000 per student to match their current income levels, or £6,000 once "efficiencies" had been made.
The government worked out its costs and budgets related to student funding on an estimate that on average universities would charge £7,500 for courses.
The Russell Group, which represents some of the UK's top universities, says higher fees are unavoidable.
Director General of the Russell Group, Dr Wendy Piatt, said: "It is crucial that our leading research-intensive universities are allowed to charge more than £6,000 in fees if they are to maintain their world-class status, give their students the first-rate education they deserve and continue to widen participation.
"Given the far reaching cuts to university funding introduced by the current and previous governments, higher graduate contributions are the fairest and only viable way forward."
Other parts of the UK are facing tuition fee rises too.
It was announced earlier this week that in Wales, the basic fee level is to rise to £4,000, with a maximum level of £9,000.
Welsh students travelling to other parts of the UK to study will have their fees subsidised.
In Northern Ireland, a report commissioned by the Department of Employment and Learning (DEL) has recommended that fees should rise to a maximum of £5,750.
In Scotland, students do not pay tuition fees but there growing calls for some kind of "graduate contribution".