Failure to address educational inequalities in England means taxpayers have to "pay twice", says an alliance of 25 education organisations.
Unless every child gets a fair chance, the cost of schooling is followed by more expense later, it says.
The Fair Education Alliance sets out five national targets to break what it calls "the class ceiling".
Education Secretary Nicky Morgan said the government wished to ensure all pupils achieved their full potential.
The report, Will We Ever Have A Fair Education?, says educational unfairness starts before children even start school, with poorer five-year-olds less ready for learning than their more affluent peers.
The Fair Education Alliance was launched in June, with 25 organisations including businesses, education charities, trade unions and campaign groups pledged to work to "significantly narrow the achievement gap between young people from our poorest communities and their wealthier peers by 2022".
The alliance promises to monitor progress made every year against its five fair-education targets.
This is the first "report card" and the authors say it "shows a worrying picture".
"Against every single fair-education impact goal there is a significant gap between the most and least deprived," it says, and the gaps are "even bigger" than anticipated.
The five national targets aim to narrow the gaps between rich and poor from "cradle to career".
- At primary school, poorer pupils are half as likely to meet expected numeracy and literacy standards.
- At GCSE, 65% of wealthier pupils get five good grades while 63% of poorer pupils do not.
- At 16, poorer pupils are twice as likely not to be in employment, education or training.
- At 18, richer pupils are four times more likely to attend the most selective universities.
- Throughout education, poorer pupils are more than twice as likely to be excluded.
The report suggests the gap has narrowed in recent years, particularly in London.
"If all primary schools in England performed as well as those in the capital, the gap would be reduced by 65%."
But it says progress is far too slow, with poorer children in affluent areas, where they are a minority, particularly unlikely to get the support they need.
The authors say addressing these inequalities would boost the economy by £6bn a year by 2030 and £56bn a year by 2050.
Alan Milburn, chairman of the Social Mobility and Child Poverty Commission, said this was the first time such a broad range of organisations had come together "to help ensure every child is able to go as far as their talents and aspirations will take them".
Mr Milburn said schools were working to defy the odds and provide children "with the education and life chances that they deserve.
"But breaking the link between demography and destiny requires radical new solutions that bring together the expertise and commitment of our entire society."
Education Secretary Nicky Morgan said: "Our plan for education is designed to ensure every child, regardless of their background, leaves school prepared for life in modern Britain.
"We welcome this report and share the Fair Education Alliance's commitment to continuing to close the gap between disadvantaged pupils and their peers and at the same time raising the bar to ensure all pupils achieve their full potential.
"To do this we have invested in high quality provision for the early years, encouraged a higher calibre of graduates into teaching and introduced a new curriculum which embodies high expectations in every subject to drive up standards and raise aspiration."
Brett Wigdortz, chief executive of the Teach First education charity, said it was time to make educational success a "national priority which will determine the UK's collective future".
He said that the alliance would "lead activity and press for action... to improve the life chances of children from poorer backgrounds".