Fans of West Ham, Tottenham and Leyton Orient are united in their opposition of a move to the Olympic Stadium.
They want to remain in their own stadiums, claiming an athletics track will affect match-day atmosphere.
West Ham looked to have secured a switch to Stratford following the London 2012 Games but the stadium will now remain in public ownership.
Hammers fan Nigel Kahn said: "What I'm against is the aesthetics of the Olympic stadium."
The board of the Olympic Park Legacy Company (OPLC) decided to end negotiations with West Ham because of delays caused by ongoing legal disputes.
West Ham season ticket holder Mr Kahn set up a petition against the club's proposed move, which has been signed by thousands of Hammers fans.
Mr Kahn, 41, said: "I'd be happy to leave the Boleyn Ground and go to Stratford - but only if it was to go to a real football stadium.
"At the Boleyn Ground you're up close to the action but the Olympic Stadium is oval-shaped and the distance from the track to the nearest seat is 35m."
But he's not against the prospect of Tottenham moving from the north of London to the east - an area traditionally associated with the Hammers.
Mr Kahn said: "I've never gone with the argument that Spurs will take our supporters. Most West Ham fans don't live in Newham any more.
"But most Tottenham fans don't want to go to Stratford and Leyton Orient will only get gates of 7,000 at best."
Spurs fan Paul Smith, of the spursodyssey.com website, said: "Hopefully Spurs are definitely out of the running now because most Spurs fans don't want to leave White Hart Lane.
"My from-the-heart reaction is I want Spurs to stay in Tottenham and White Hart Lane.
"I look forward to us filling a new stadium there without a running track and maintaining a proper atmosphere."
Mr Kahn said he would not be completely opposed to a ground-share as a "last resort" as long as it was "neutral colours, with no crests or writing".
Rory Sheen, assistant editor of West Ham fanzine 'Over Land and Sea' urged the club's vice-chairman Karren Brady to reconsider the move to the Olympic Stadium.
The 24-year-old, from Buckhurst Hill, said: "I never wanted us to move from Upton Park. It's a proper old-fashioned ground in the streets and a lot of West Ham fans would agree that the ground has acted as our 12th man."