The number of people participating in sport at least once a week has fallen by 200,000, according to Sport England.
Some 15.3 million people played sport between April 2012 and April 2013, down from 15.5 million in October's figures.
However, there are still 1.4 million more people playing sport than when London won the Olympic bid in 2005.
Sports Minister Hugh Robertson said: "We remain committed to delivering a lasting sports participation legacy from London 2012."
The cold winter was identified as a factor in the drop.
Sport England's figures - which are compiled twice a year, for the 12 months to April and the 12 months to October - are for the number of people aged 16 and over playing at least 30 minutes of sport at moderate intensity at least once a week.
They were compiled from telephone interviews conducted with a cross-section of 161,000 people, with all sporting activity counted - organised or not.
Swimming has maintained its position as England's most popular participation sport, with just under 2.9 million people taking part at least once a week.
But football has slipped from second to third place in the table, behind athletics.
More than 1.9 million people played football once a week during 2012-13, a drop from the figure of almost 2.2 million for 2011-12.
A Football Association spokesperson said that poor weather had contributed to the drop in the number of people playing the game.
Local authorities also faced difficulties finding the money to maintain pitches damaged by the weather, he added, which recreational football had suffered in the difficult economic climate.
The spokesperson said: "Today's results in Sport England's Active People Survey are both surprising and concerning to the FA as they do not reflect the long-term trends in football participation.
"We will therefore take the time to work with Sport England to fully understand these results in much greater detail."
The Sport England figures show that of 29 sports that recorded a change in once-a-week participation figures, nine showed a year-on-year increase, while 20 suffered a decline.
"These figures show we're holding on to the growth achieved over the past 18 months, despite some poor recent weather," said Sport England chairman Nick Bitel.
"There's a long way to go but it's particularly encouraging to see the numbers for young people are now moving in the right direction."
Shadow sports minister Clive Efford suggested the weather could not be used as an excuse.
"These disappointing figures show the government is failing to deliver the Olympic legacy," he said.
"There should have been a significant increase in participation following the Olympics. It is not acceptable to explain away these figures by saying we have had a bad winter.
"The government has failed to plan ahead and is playing catch-up to get more people participating in physical activity. They are showing a complete lack of planning and any coherent strategy to deliver a legacy."