Teachers criticise Cameron's Olympic school-sport remarks
Head teachers' leaders have called the prime minister's comments on school sport "extremely unfair".
David Cameron called for "a big cultural change" in favour of competitive sports in schools and suggested some teachers were not "playing their part".
The Association of School and College Leaders (ACSL) said Mr Cameron failed to recognise the "huge contribution" many teachers made to school sport.
Teachers say cuts are affecting sport.
The idea of an Olympics legacy is high on the political agenda, with politicians, sportsmen and women and commentators talking of the need to ensure young people benefit from the 2012 London Games through greater involvement in sport.
The Culture Secretary Jeremy Hunt described school sports provision as "patchy" earlier this week.
And some commentators have pointed out that a relatively large number of Team GB's medal winners went to private schools.
However, many of the most high-profile gold medal winners went to state schools, including Jessica Ennis, Mo Farah, Bradley Wiggins and Andy Murray.
Mr Cameron told LBC 97.3 FM radio: "If we want to have a great sporting legacy for our children - and I do - we have got to have an answer that brings the whole of society together to crack this, more competition, more competitiveness, more getting rid of the idea all-must-win prizes and you can't have competitive sports days."
He added that the problem was not simply one of money, but of "some teachers not wanting to join in and play their part".
"We need a big cultural change - a cultural change in favour of competitive sports. That's what I think really matters."
The comments have angered unions representing teachers and head teachers, who say the removal of funding for a sports partnership scheme for schools has damaged provision.
The coalition government is also coming under fire for removing the requirement that schools in England provide pupils with at least two hours of sport a week - the minimum expected in schools in other parts of the UK.
At the time, in autumn 2010, Education Secretary Michael Gove said he wanted to cut bureaucracy for head teachers, while trusting them to give pupils the best physical education and more opportunities to play sport competitively.
He expected that schools would at least provide that much sport and PE.
In a letter at the time, he wrote: "I would expect every school to want to maintain as a minimum the current levels of PE and sport each week for every pupil."
On Wednesday, a spokeswoman for the Department for Education said: "This was not a target - it was an unenforceable aspiration.
"No more than two in five pupils took part in competitive sport when we told schools they no longer had to inform us of how much sport pupils were doing."
But Labour has hit back, saying the two hours rule was a target and that between 2002 and 2010, the number of young people doing at least that rose from 25% to 90%.
ASCL deputy general secretary Malcolm Trobe said: "We all want to build on the immense success of our Olympic teams and we understand that schools have an important part to play in this.
"However the prime minister's criticisms of school sport are ill-informed, unfair and fail to recognise the huge contribution that many teachers make to sports in schools.
"Many teachers, not just PE staff, willingly give of their time to motivate and coach young people in a wide range of sports."
Mr Trobe said the government had removed the funding for what he called "the most successful school sports scheme ever" - the School Sport Partnership.
This was a scheme linking sports colleges and clubs with schools, which brought more specialist PE teachers in to schools.
National Union of Teachers general secretary Christine Blower said: "These are foolhardy remarks by a prime minister who seems not to realise he is the architect of a worsening situation.
"Many Team GB medallists attended state school, which makes ludicrous his suggestion that teachers are letting the side down."
Mr Cameron said the government was investing £1bn in school sports over the next four years.
Chief executive of the Youth Sport Trust, John Steele, said he welcomed Mr Cameron's focus on school sport, but teachers needed more time and resources.
"There is some fantastic work going on in schools to deliver sport from some very dedicated staff," he said.
Stephen Twigg MP, Labour's shadow education secretary, said: "If we are get to get more school pupils doing competitive sport, the prime minister needs to take some responsibility.
"In 2010, with Labour, nearly eight in every 10 pupils were taking part in competitive sport, an increase of one fifth from 2007.
"But in the last year, government cuts have meant that 60% less staff time has been dedicated to organising sport in our schools."
The Conservative MP Damian Hinds has accused teaching unions of being hypocritical about school sports.
He says a work-to-rule being staged by the NASUWT means teachers are being advised not to run after-school clubs.
He said: "It is utterly hypocritical for the unions to blame the government for a lack of sport in school at the same time as ordering their members not to help with activities outside of school hours."
The NASUWT rejected Mr Hind's remarks, saying its was telling its members to carry on with out-of-hours clubs if they wanted to, but that they could stop doing so if the activities had been "imposed" on them.
It is staging a work to rule in its dispute with the government over pay and pensions.