Absent fathers should be "stigmatised" by society in the same way as drink-drivers, the prime minister has said.
A father himself, David Cameron said "runaway dads" should feel the "full force of shame" for their actions.
Writing in the Sunday Telegraph to mark Father's Day, he said it was not acceptable for single mothers to be left to bring children up on their own.
Labour said the PM was being "utterly disingenuous" and his policies were making life harder for single mothers.
The prime minister also said he was determined to keep his election pledge of tax breaks for married couples.
The policy had been dropped by the coalition because of Liberal Democrat opposition.
'Beyond the pale'
The Camerons have three children, Nancy, Arthur and Florence - who was born last August.
Their first child, Ivan, who was born profoundly disabled and needed 24-hour care, died in February 2009.
Mr Cameron said traditional family life was the "cornerstone of our society" and that even when parents were separated, fathers had a duty to support "financially and emotionally" their children, spending time with them at weekends, attending nativity plays and taking an interest in their education.
Where men were unwilling to face up to their family obligations, he said that it was up to the rest of society to make clear that such behaviour was unacceptable.
"It's high time runaway dads were stigmatised and the full force of shame was heaped upon them," he said.
"They should be looked at like drink-drivers, people who are beyond the pale.
"They need the message rammed home to them, from every part of our culture, that what they're doing is wrong; that leaving single mothers, who do a heroic job against all odds, to fend for themselves simply isn't acceptable."
But Erin Pizzey, founder of domestic violence charity Refuge, said Mr Cameron was displaying a lack of understanding about the reality of family break-ups.
"There are a lot of reasons why [fathers are] not with their children... not least that women won't let them," she said.
Ms Pizzey said it was wrong to single out men, adding: "There is a vast mass of women who are equally as feckless as the men and we never talk about them."
The prime minister also talked about his admiration from his own father, Ian Cameron, who died last year aged 77.
"Seeing him get up before the crack of dawn to go and do a hard day's work and not come back until late at night had a profound impact on me," he said.
For Labour, shadow home secretary Yvette Cooper said Mr Cameron's words were "hollow".
"His government is making it much easier for deadbeat dads to run from their responsibilities by charging mums to use the Child Support Agency," she said.
"And now he wants to reward runaway dads who remarry with a marriage tax break, whilst still cutting tax credits for the mums left behind holding the baby."
The much-maligned Child Support Agency (CSA) was effectively axed in 2008 when it was rolled into a new body - the Child Maintenance and Enforcement Commission (CMEC).
Earlier this year, the government announced proposals to encourage parents to reach their own arrangements over child maintenance - rather than relying on the CMEC - by charging those unable to do so.
They would be required to pay an upfront fee of £100 (or £50 if they claim benefits), plus an ongoing charge of between 7% and 12% of the money paid where payments are collected and enforced by the CMEC.
Fiona Weir, from the charity Gingerbread - which campaigns on behalf of single parents - also criticised the proposals.
"If the prime minister really wants to support heroic single parents, he must withdraw these damaging proposals which would limit access to the CSA," she said.
To mark Father's Day, a survey of 1,000 fathers - all with children aged under five - found only 15% of fathers believed their partners were more competent at caring for their baby than they were.
But 61% said they felt the general public did not trust men to care for babies as much as they trust women to.
The survey also showed that nearly half said they felt they had been ignored by maternity services, while 18% said they were not aware of their right to take two weeks' paternity leave and pay.