The EU's policy of aiding Libyan authorities to intercept migrants and return them to detention is "inhuman", the United Nations said on Tuesday.
The UN's human rights chief accused European countries of ignoring warnings over the deal struck with Libya.
Authorities say it has caused detainees to rise sharply to almost 20,000.
UN monitors who visited facilities said they found "thousands of emaciated and traumatized men, women and children piled on top of each other".
The UN High Commissioner for Human Rights, Zeid Ra'ad al-Hussein, said migrants had been locked in hangars and stripped of dignity with "no access to the most basic necessities".
He described the situation as fast-deteriorating, and the Libyan detention system as "broken beyond repair".
"What was an already dire situation has now turned catastrophic," Mr Hussein said, urging the international community to stop turning a blind eye.
Libya's Department of Combating Illegal Migration (DCIM) said almost 20,000 people were now in their facilities, dramatically up from just 7,000 in September.
Thousands of migrants have died attempting to cross the Mediterranean from Africa into Europe.
In August, Italy and the EU put in place a deal where they train the Libyan coastguard to intercept boats and return migrants to Libya, which is a large transit hub.
It has seen arrivals drop dramatically, but remains controversial as concerns about abuse and overcrowding in Libyan facilities persist.
The UN monitors who visited Tripoli detention centres said detainees reported beatings, torture and sexual violence both in transit and in the detention facilities themselves.
"We cannot be a silent witness to modern day slavery, rape and other sexual violence, and unlawful killings in the name of managing migration and preventing desperate and traumatized people from reaching Europe's shores," Mr Hussein said.
Just this week European government representatives met with African leaders in the Swiss capital Bern to discuss ways to support vulnerable migrants in Libya, but failed to announce any concrete steps.
Libya has become a major transit hub country for migrants looking to reach Europe from North Africa.
The country remains fraught with violence following the overthrow of its long-term leader Muammar Gaddafi in 2011.