Outdoor air pollution is contributing to about 40,000 early deaths a year in the UK, say the Royal Colleges of Physicians and of Paediatrics and Child Health.
They say diesel emissions have been poorly controlled.
And indoor air pollution has been overlooked.
Tobacco still poses the biggest indoor threat, but wood-burning stoves, cleaning products and air fresheners can contribute.
Mould and mildew in poorly ventilated rooms can also cause illness.
"Being indoors can offer some protection against outdoor air pollution, but it can also expose us to other air pollution sources," the report says.
"There is now good awareness of the risks from badly maintained gas appliances, radioactive radon gas and second-hand tobacco smoke, but indoors we can also be exposed to NO2 [nitrogen dioxide] from gas cooking and solvents that slowly seep from plastics, paints and furnishings.
"The lemon-and-pine scents that we use to make our homes smell fresh can react chemically to generate air pollutants, and ozone-based air fresheners can also cause indoor air pollution."
Co-author Prof Jonathan Grigg said there was now clear evidence that air pollution - largely from factories and traffic - was linked to heart disease and lung problems, including asthma.
"As NHS costs continue to escalate due to poor public health - asthma alone costs the NHS an estimated £1bn a year - it is essential that policy makers consider the effects of long-term exposure on our children and the public purse," he said.
Prof Grigg said the public could also help by:
- walking, cycling or taking the bus or train instead of driving, when possible
- keeping gas appliances and solid fuel burners in good repair
- making homes more energy efficient
Prof Stephen Holgate, asthma expert at Southampton University and chairman of the reporting group, warned against complacency.
"We all have a part to play to cut environmental pollution. We can't see it, smell it or taste it, which is why people do not necessarily think we have a problem," he said.
Indoor air pollution
- Not as big a problem as outdoor pollution, but experts say it has been overlooked and needs studying
- Heating and cooking appliances can release particulates and nitrogen oxides that can damage the lungs and heart
- Carbon monoxide from faulty boilers and heaters can be fatal
- Mould and mildew in damp and poorly ventilated homes can cause respiratory problems
- Common household products - cleaning products, air fresheners - contain organic chemicals that evaporate into the air
- Carpets and furniture can also release these volatile organic compounds
- At present, not much is known about what health effects occur from the levels of organics typically found in homes
"When you see cars piling up on the way to school taking their children, the fumes directly from the vehicle in front are being vented straight into the car behind, and exposing their child - and yet we are ignoring this," he added.
Prof Holgate called for authorities to monitor pollution levels more closely, build new homes away from busy roads and consider closing particularly polluted roads at certain times.
He also advised people to open and close windows in the home several times a day.
"It's amazing that we are now living in these tight, sealed homes that we are frightened of opening the window and letting a bit of fresh air in," he told BBC Radio Four's Today programme.
A spokesman from the Chemical Industries Association said: "This report should be seen as a significant contribution to an important debate. Chemical and pharmaceutical businesses take their responsibilities extremely seriously in both the production and use of their products, with due regard to the health of their employees and the public at large.
"Improving air quality is central to that responsibility and we will continue to be proactive in meeting our regulatory requirements and delivering innovative solutions that seek to minimise the risks of exposure."