Campaigners opposing plans for a "super prison" housing 1,000 inmates in East Yorkshire said the site would leave them feeling "very vulnerable".
Outline planning permission has been approved by East Riding Council for a £91m Category C jail near an existing high-security prison in Full Sutton.
Residents said the safety of the village was at risk and there would be an increase in traffic.
The government said modern jails would help cut re-offending.
Maddy Ruff, 53, who has lived in the village for 13 years, said: "From a safeguarding perspective we feel very vulnerable.
"People who are in the last two years of their sentence will be allowed to leave the prison and wander around. There's children in the village and our safety will be at risk."
She said the 38,217 sq m building, which is proposed to be built on Moor Lane, would result in a "huge amount of traffic" arising from an increase in "inmates, visitors and staff".
"It's going to absolutely ruin our small village," she said. "The road network is completely unsuitable. The bus only comes twice a day.
"We're not going to sit by and let this happen."
Resident and campaigner Liz Pert said: "As far as I understand they'll be getting visitors up to three times a week. A thousand prisoners means up to three thousand visits.
"The village just can't cope with all that traffic."
Andrew Neilson, from the Howard League for Penal Reform, said building bigger prisons was not the answer and called for "better management of the size of the prison population".
In a statement, the Ministry of Justice said: "We have undergone extensive work to ensure the Full Sutton site is suitable for a new development.
"We fully understand the concerns of residents, and are committed to mitigating any disruption that may arise from the development of the site."