Labour says it would scrap a government scheme that allows offices and industrial buildings to be converted into homes without planning permission.
The party said changes to permitted development rules in England had led to the creation of "slum housing and rabbit hutch flats".
It also said developers had been able to avoid building affordable homes.
The Conservatives said the plans would "cut house building and put a stop to people achieving home ownership".
In 2013, the government changed planning rules to allow developers to turn offices, warehouses and industrial buildings into residential blocks without getting permission from the local council, in a bid to boost house building.
The rules have since been further relaxed, leading to 42,000 new dwellings being created from former offices in the last few years.
However, permitted development schemes are exempt from official space standards and also from any requirement to provide affordable homes.
Labour said the policy had seen the loss of more than 10,000 affordable homes, and meant that flats "just a few feet wide" were now counted in official statistics as new homes.
It said its policy was still to build 250,000 new homes a year in England with 100,000 being "genuinely affordable".
"This Conservative housing free-for-all gives developers a free hand to build what they want but ignore what local communities need," said John Healey, Labour's shadow housing secretary.
"Labour will give local people control over the housing that gets built in their area and ensure developers build the low-cost, high-quality homes that the country needs."
In one permitted development scheme at Newbury House in Ilford, an office block has been turned into 60 flats measuring as little as 13 sq metres each.
According to national space standards, the minimum floor area for a new one-bedroom one-person home is 37 sq metres.
Critics say the schemes can be damaging to residents' mental wellbeing, as well as being miles from amenities and conducive to crime.
At Terminus House - a converted office block in Harlow - crime jumped 45% in the first 10 months after people moved in and by 20% within that part of the town centre.
Tackling the housing crisis?
But some developers warn that without permitted development many office to residential schemes would no longer be viable.
The government says the rules are helping tackle the housing crisis and allowing people to get on the housing ladder.
Of the 13,526 homes delivered under permitted development last year, more than three quarters were built outside of London
Marcus Jones, Conservative vice-chair for Local Government, said: "Labour's plans would cut house building and put a stop to people achieving homeownership.
"We are backing permitted development rights, which are converting dormant offices into places families can call home.
"Whilst Labour put politics before our families, the Conservatives are delivering the houses this country needs so every family has a place to call home."