The free movement of workers within the EU makes it more difficult to curb immigration to the UK, Home Secretary Theresa May has admitted.
But she told the BBC it did not make it "impossible", as she made the case for a vote to stay in the EU on 23 June.
If the UK left the EU it would have to accept free movement rules "without any say" over them in order to access the single market, she warned.
Vote Leave said Mrs May had "given up" on trying to control immigration.
"The only way to take back control of our borders and have a fair immigration policy is to vote Leave on 23 June," its chief executive, Matthew Elliot said.
But speaking on the BBC's Andrew Marr show, Mrs May - who has so far kept a low profile in the referendum debate - said immigration policy and border control were "different" issues and should not be conflated.
In her first major intervention in the campaign since backing Remain, the home secretary said: "Yes, free movement makes it harder to control immigration but it doesn't make it impossible to control immigration."
She said controlling immigration - whether from inside or outside the EU - "is hard", and added: "We have constantly to be working at it, which is exactly what we're doing, so you can't just change one feature and assume that is going to have an impact."
Warning against an EU exit, Mrs May said: "If we were to be outside the European Union and still want the sort of access to the single market that people talk about... where that's been done for other countries, they have to accept the free movement rules without any say over those rules."
Free movement of people gives all EU citizens the right to travel, live and work wherever they wish within the bloc of countries.
An independent analysis recently forecast that three million more people would come to the UK by 2030 if Britain stayed in the EU.
Mrs May said she understood the public's "concerns" over immigration, and conceded the current level - well above the government's "tens of thousands" target - is "too high".
But she insisted the government was taking steps to address it, citing action to tackle "abuse" of free movement rules.
She also said the UK had "an opportunity" to change the rules in Britain's favour from within the EU, and said this is what David Cameron had achieved through his EU renegotiation, by reducing "pull factors" for migrants.
Mrs May contested claims by Leave campaigner and cabinet minister Michael Gove that EU membership means the UK cannot control its borders, saying the UK is not in the border-free Schengen area and "can stop people entering" the country.
"But crucially we can only do that if we've got the information we need to have," she added, as she made the case for continued EU membership.
Commenting on the home secretary's remarks, Vote Leave's Mr Elliott said: "It's clear that even Theresa May has given up on the government's promise to control immigration.
"She admitted that EU membership makes this impossible and that David Cameron has failed to bring any powers back from the EU to change that."
Meanwhile, justice minister Dominic Raab, who supports an EU exit, said negotiations following a potential vote to leave could include discussions on implementing visas for EU travellers heading to Britain.
"At the moment President Obama's administration... is looking at new visa requirements and screening from Germany, Belgium, Greece, France because of the recent terrorist attacks.
"I think we should at least have the power and the control to do that to make sure we keep Britain safe," he told the BBC's Sunday Politics programme.
Asked whether that would mean UK citizens requiring visas to travel to Europe, Mr Raab said: "Or some other kind of check".