Travellers who have moved on to a site in Hall Green in Birmingham said they were there to see the Pope.
Councillor Martin Mullaney said the families had moved on to a field next to Sarehole Mill, a childhood haunt of Lord of the Rings author JRR Tolkien.
He said they had told him they had no idea the event was ticket only.
Pope Benedict XVI is holding a special Mass in Cofton Park on Sunday 19 September, at the end of his four-day UK visit.
Mr Mullaney, Birmingham City Councillor for Moseley and Kings Heath, said there were about 16 families who said they had made their way from County Donegal in Ireland.
"I asked them what they were doing in Birmingham and what they said to me was, 'We've come here to see the Pope because we want to get his blessing at the Mass in Birmingham."
He said more travellers may be on their way so the city needed to be proactive and let them know there will be no admission without a pass.
"The pilgrim passes have all already been allocated to parish priests, and there will be no admission to the mass without a pass."
An eviction notice has been served on the families and they are being encouraged to move on to a disused car park in Moseley.
The council is working on proposals for temporary sites where travellers can stay for the duration of the Pope's visit.
Preparations for the visit have begun with access to some areas of Cofton Park being restricted as engineers design walkways and begin to put up the main structures, such as a stage and altar.
The site the travellers have moved on to is thought to have provided inspiration for Tolkien's Lord of the Rings trilogy.