Even moderate drinking during the earliest months of pregnancy may be damaging, say researchers in Leeds.
Their study is the latest in a long debate over whether it is safe to drink at all during pregnancy.
The findings, published in the Journal of Epidemiology and Community Health, suggest the chances of premature birth increased.
The NHS recommends people avoid alcohol during pregnancy or when trying to conceive.
But says if people choose to drink, then they should not have more than two units of alcohol (about one pint) twice a week.
Heavy drinking in pregnancy is known to be damaging as it can affect the baby's development. But there is far more debate about drinking at the upper limit of the NHS guidelines.
Around seven in every 100 births in the UK is premature.
The study on 1,264 women in Leeds showed drinking more than the two units limit doubled the risk of premature birth, but even drinking at the limit increased the risk.
Camilla Nykjaer, one of the researchers at the University of Leeds, told the BBC: "This is a very sensitive issue, we don't want women who are pregnant now to panic, the individual risk is actually low.
"They shouldn't drink, they should stop drinking if they have been drinking during the pregnancy."
However, a study of more than 11,000 five-year-olds, conducted by University College London, showed drinking one or two units of alcohol a week during pregnancy did not raise the risk of developmental problems in the child.
Prof Yvonne Kelly who conducted that research told the BBC: "Heavy drinking is really very, very bad, but at low levels, in the work we've done we haven't found any negative effects in childhood.
"It's a massively charged area, getting the tone of this right is quite difficult.
"The guidelines are there, women are sentient beings and can choose - it's hugely politically charged all of this, I guess people will make their own judgements."
Dr Patrick O'Brien, a spokesperson for the Royal College of Obstetricians and Gynaecologists, said: "While the safest approach would be to choose not to drink at all, small amounts of alcohol, not more than one to two units once or twice a week, have not been shown to be harmful after 12 weeks of pregnancy.
"Pregnant women should always consult their midwives or doctors if they have any concerns about their alcohol intake."