Some inmates are "living in fear" choosing to stay in their cells 24 hours a day at a prison where violence has increased, inspectors said.
Most of these "self-isolators" at HMP Featherstone, near Wolverhampton, felt at risk because of their debts.
Chief Inspector of Prisons, Peter Clarke, said there had been a "shocking worsening in standards", and violence against staff had increased.
The decline is not acceptable but reflects pressures, a spokesman said.
The unannounced October inspection was held two months after inmates started fires during a week-long disturbance.
Mr Clarke said the number of men who chose to self-isolate was a "symptoms of the lack of safety" at the prison.
Inspectors said the abuse these prisoners suffered included other prisoners urinating under their cell doors.
"They could not access showers, exercise, food or work without fear of repercussion," the report said.
Inspectors said while some officers showed concern, others referred to them as "scaredy cats" and their predicament as "self-inflicted".
The Category C prison, in Staffordshire, which holds about 650 men, was last inspected by the HM Inspectorate of Prisons in 2013, when it received a positive report and scored highly in two out of three areas.
The latest report found a "sharp decline" in three out of four areas, with safety assessed as "poor".
Two thirds of prisoners said it was easy to obtain drugs, with a fifth admitting they had developed a drug problem while inside.
"The backdrop to the decline at Featherstone was clear evidence of poor industrial relations, staff shortages and some significant prisoner unrest," Mr Clarke said.
The report noted the quality of teaching inside the prison was good.
Michael Spurr, head of the National Offender Management Service, is convinced the Featherstone governor could turn its fortunes around.
"The deterioration in performance at Featherstone isn't acceptable, but reflects the real pressures which the system has faced over the last few years," he said.
"The government has set out a clear plan for reform in the Prison and Courts Bill laid before Parliament last week, including investing more than £100m to provide 2,500 additional prison staff."