Ex-Northern Ireland Secretary Lord Mandelson has accused Theresa Villiers of having "her fingers in her ears" over claims there would be no hardening of the Irish border after a Brexit.
Ms Villiers, the current Northern Ireland secretary, has said the existing border arrangements would continue much as they are if the UK left the EU.
People can move freely between Northern Ireland and the Republic of Ireland due to the Common Travel Area (CTA).
It pre-dates the creation of the EU.
The CTA is an informal arrangement but a report by MPs on the Northern Ireland Affairs Committee said that in the event of a British exit from the EU (Brexit) the future of the CTA "would be put into question."
Legal evidence heard by the committee suggested that a continuation of the CTA would require agreement of other EU members.
Lord Mandelson said: "Theresa Villiers is not right about this.
"The thing about the Leave campaigners is they want us to take this dramatic step but every time some negative or undesirable consequence is pointed out they just wish it away.
"They put their fingers in their ears or their heads in the sand."
Ms Villiers has said there were "some risks" migrants could enter the UK illegally via an open Irish border but added that they could be dealt with by the authorities after they arrive.
The referendum will take place on 23 June.