The number of adults swimming at least once a week in England has fallen by more than three quarters of a million in the past decade - a 23.7% drop.
The drop in participation comes despite more than 500 new swimming pools opening across the country since 2006.
Sport England data shows 2.5 million people swam weekly in 2015-16, down 776,000 from 3.27 million in 2005-06.
The Amateur Swimming Association said more people were choosing activities such as aquacise and wild swimming.
Swimming, however, remains England's most popular mass participation sport, with about one in every 20 people aged over 14 swimming at least once a week.
But data gathered by Sport England's Active People's Survey shows there has been a 24% decline in the number of people regularly participating in the sport.
The fall in the number of swimmers comes as Sport England's data also shows 564 new publicly accessible swimming pools have opened in England.
Of these new pools, 73% have been built in London or the south of England.
"We have a real challenge with swimming because it's not a sport that is continually at the forefront of people's minds," said Nick Caplin, director of participation at the Amateur Swimming Association (ASA).
"I don't worry too much about these figures because what we're finding is that more people are doing different water based activities like wild swimming or aquacise. But what we do need to focus on is ensuring that when people do go to a swimming pool they have a great experience and one they want to repeat.
"Overall I think our sport still has a positive story to tell."
Olympic silver medallist Keri-anne Payne said she thinks people just want different things from swimming these days.
"In terms of the actual sport it is just flourishing at the moment with people wanting to take on new challenges," she said.
"There has certainly not been a drop-off in people taking up swimming, but they are more likely to be taking a different route.
"There are triathletes who do wild swimming and people who just like a bit more adventure; people who like to go to Lake Windermere and swim there, or Loch Lomond, or down to Dover to swim the Channel. It's a new element people are looking for."
David Wilkie, who was Olympic and Commonwealth Games champion in the 1970s, believes the popularity of swimming has declined for a number of factors.
"The choice now in sport is much more varied than it used to be. There is so much more choice and there are more gyms where you have activities like pilates, zumba classes, weights.
"I think the most impact though is swimming with children. Schools don't really offer any swimming because it is expensive, it takes a lot of time to get children to pools, get them changed and get them on and off the bus.
"Also I think obesity is a big problem, we are getting much bigger and I think people are embarrassed about how they look."
The ASA reported a "surge" in the number of people searching for details about local swimming pools after the success of the Great Britain swimming team at the Rio Olympics in August.
Cuts to council budgets have seen some authorities across England either close swimming pools or allow businesses or charities to take them over.
Faye Howard led a campaign in Stocksbridge, South Yorkshire, to keep her local pool open.
"When it comes to participation we have to accept that people are time poor, they want facilities to be convenient," she said.
"We rely on volunteers to keep our costs down but such is the importance of this swimming pool to the local community people are willing to give up their free time to make sure it stays open."
The data also shows every region in England has a seen a fall in the number of people swimming regularly.
The biggest recorded falls are found across the North of England, where participation levels have sunk by about a third.
The decline in participation is attributed to factors such as changes in individuals' preferences or circumstances, or difficulty in accessing local facilities.
Professor Jim McKenna from Leeds Beckett University said the fact people were spending more time at work was also having an impact on the number of regular swimmers.
"As a society we're now working longer and more unpredictable hours. Where once people would nip out in their lunch break for a swim they're now answering emails at their desk and in general leading a more sedentary life."
As the popularity of swimming has fallen over the past decade sports including athletics and activities such as fitness and conditioning have seen participation levels increase by more than 70%.