Culture Secretary Jeremy Hunt has urged Wales, Scotland and Northern Ireland not to boycott a Great Britain football team at the London 2012 Olympics.
The Football Association of Wales chief executive Jonathan Ford says he is protecting the FAW's independence by refusing to let players compete.
That is despite assurances by Fifa president Sepp Blatter that there would be no threat to their independence.
Hunt says it would be "fantastic" to have players like Gareth Bale playing.
"It would be very disappointing if all the players were English and could only be English," he said.
"I'm encouraged by what Sepp Blatter has said, by what Jim Boyce, the incoming Fifa vice-president has said that there is no risk to the independent status of the FAs of Wales, Scotland and Northen Ireland if they do take part in Team GB.
"So I really think this is a time when we need to put football politics aside and think about the athletes."
The associations of Wales, Scotland and Northern Ireland share a concern that merging for an Olympics could be used by other nations as a justification to get rid of their individual national sides.
Blatter gave an assurance over nations' independence during the 125th International Football Association Board (IFAB) conference in early March, at which delegates voted in favour of testing goal-line technology.
Ford used that reversal of policy over goal-line technology to stress why he remain unconvinced by a Fifa promise.
Asked if he was happy with the guarantee earlier this month, Ford told the BBC's Sport Wales programme: "Are you happy with the guarantee at the 124th meeting that technology wasn't going to be introduced? Of course, that changed, too.
"I absolutely took on board Mr Blatter's comments. They were very well received. But Mr Blatter is one person within many in Fifa.
"Ultimately, we want to ensure that we are protected, both now and in the future, and to ensure that we have a Welsh team competing at the highest level possible, representing our country going forth in the European Championships and World Cup.
"I don't want that to change and I'm sure English fans don't want it to change, either.
"Let's be honest [if we combined for the Olympics] there wouldn't be an England team any more, there wouldn't be a Scotland team or a Northern Ireland team.
"So, I think the majority of people in Wales want to see a Welsh team carrying on in football and competing at the highest possible level."