Home Secretary Theresa May has said the UK should quit the European Convention on Human Rights while remaining in the European Union.
Mrs May said the convention, which is separate from the EU, "can bind the hands of Parliament".
David Cameron has previously refused to rule out leaving the convention.
Ex-cabinet minister and Leave campaigner Iain Duncan Smith said her "remarkable intervention" had "utterly undermined" the government's position.
In her first major speech of the referendum campaign, Mrs May said the UK had to "stand tall and lead in Europe" rather than leaving the EU, and that membership made the UK "more secure from crime and terrorism".
She also raised concerns about more countries joining the EU.
The ECHR, which is separate from the EU institutions, was incorporated into UK law by the Human Rights Act.
The Conservatives have pledged to replace the Human Rights Act with a "British Bill of Rights" but have not proposed leaving the convention altogether.
Analysis by Laura Kuenssberg
BBC political editor
The home secretary is one of the most senior politicians in the country. Of course, her backing of the Remain campaign is important. And it's certainly better for the prime minister and the Remain campaign to have her In, rather than Out.
But her qualified support suggests there are three, rather than two, positions at the top of the Conservative party - 'Remain', 'Leave', and perhaps now, 'Reluctant Remain'.
It was the Out campaign struggling to contain their internal differences. Tonight, it feels like cracks might appear on the other side too.
A Downing Street spokeswoman urged journalists not to "overdo the differences" between the prime minister and the home secretary on withdrawal from the ECHR.
She said Mr Cameron "rules nothing out" as he seeks reform of the convention, saying the home secretary had been "setting out a clear distinction between the ECHR and the EU".
Regardless of the outcome of the 23 June EU referendum, Mrs May said she still believed the UK should leave the ECHR which she said "adds nothing to our prosperity".
It was the ECHR, not the EU, that was responsible for controversial human rights decision affecting the UK, she said.
Earlier pro-Leave Justice Secretary Michael Gove warned that five potential new members of the EU - Macedonia, Montenegro, Serbia, Albania and Turkey - would result in millions more people having the right to move to the UK.
Mrs May said Albania, Serbia and Turkey had "poor populations and serious problems with organised crime, corruption, and sometimes even terrorism".
She added: "We have to ask ourselves, is it really right that the EU should just continue to expand, conferring upon all new member states all the rights of membership?"
She said the UK had "forgotten how to lead" in Europe and must re-assert itself to force change from within, adding that it could veto Turkey joining the EU.
In her speech - and during a question and answer session - Mrs May admitted that being in the EU made it harder to control the "volume of immigration". But she said the UK was able to control its border by blocking entry to terrorists.
She praised David Cameron's leadership in trying to seek a better deal for the UK, but said the UK's collective posture was too often to "blame Europe" for its problems and the UK "has to have more confidence to get things done" rather than "shouting from the sidelines".
Mrs May said the issue the British public faced on 23 June was how the UK "maximised" its "prosperity, security, influence and sovereignty".
'Sky won't fall'
In what she said was an analysis of "the risks and opportunities" of EU membership rather than an attack on Leave campaigners, she said the post-war order had seen the UK and other countries "cede sovereignty in a controlled way" by co-operating to prevent a greater loss of sovereignty through military conflict or economic decline.
Mrs May said the question was not whether the UK could "survive" outside the EU given that it was the world's fifth largest economy which had "friendships and alliances across the world" - but whether the UK was "better off in or out".
While "the sky would not fall" in the event of Brexit, she said she had concluded it was a matter of "hard-headed national interest" to remain in, based on security, trade and prosperity.
On security, she said EU membership enabled the UK to access EU-wide information, such as criminal records, to allow the UK to turn away serious criminals and terrorists at the border, fast-track the extradition of offenders and simplify the deportation of prisoners.
The UK had extensive intelligence-sharing with the US and other allies outside the EU, she said but being excluded from the EU-wide information sharing would make the UK "less safe".
However, she called for major changes to the way the UK did business in the EU, including reducing the power of the European Parliament and the European Court of Justice.
'Out of control'
Mr Duncan Smith told the BBC the home secretary's comments on the ECHR and EU expansion had "thrown the gauntlet down" to the government.
On EU expansion, he claimed, the prime minister is "in favour of extending our borders to Syria" by allowing Turkey to join the EU.
"She actually basically questions the very campaign she says that she supports," he said.
"This is a remarkable intervention. She has really thrown the gauntlet down and undermines the whole of the 'vote stay in' campaign."
Mr Duncan Smith said immigration was "out of control" and free movement of people rules meant the UK had to accept criminals coming into the country.
"An elected government in the UK elected on a platform to reduce immigration right now cannot deliver that because the EU is an open border," he told Radio 4's Today.
"You cannot reject anyone unless you can demonstrate categorically that they pose an immediate threat to the life and livelihood of the UK."