Former Conservative minister David Mellor says asking football to mark the death of Baroness Thatcher with a one-minute silence is a "hopeless request".
Former Prime Minister Thatcher died in London on Monday, aged 87, after suffering a stroke.
"I think it would make a dismal spectacle and we should not try and do it," Mellor told BBC Radio 5 live.
But Premiership rugby clubs Saracens and Exeter will mark Lady Thatcher's death before home matches this weekend.
Saracens will observe a minute's silence before they play Worcester Warriors at Allianz Park on Sunday.
And Exeter have told BBC Sport that they will do likewise ahead of their game against London Irish on the same day.
When contacted London Irish said that discussions between their team manager and the club's chief executive had taken place and that: "if Exeter have made that decision, then London Irish will respect that decision."
One-minute silences were not held in the Premier League when former Prime Ministers James Callaghan and Edward Heath died. However, respects were paid after the death of the Queen Mother.
Wigan chairman Dave Whelan and his Reading counterpart Sir John Madejski want football to mark the death of Baroness Thatcher, who was Conservative prime minister from 1979 to 1990, with a one-minute silence.
But the Premier League and Football League will not ask clubs to mark Lady Thatcher's death at forthcoming games.
It will be left to individual clubs to decide whether they want to pay tribute. It is understood the Football Association has no plans at present to mark her death.
Mellor, the first person to fulfil the role of Secretary of State for Culture, Media and Sport, said: "I think it's a hopeless request.
"I see both sides of Margaret Thatcher, but it doesn't matter if she was the greatest peacetime prime minister, there is no point in asking football to have a minute's silence that lots of people will not observe.
"There is no respect for Margaret Thatcher's memory in a situation where you effectively invite a significant proportion, I suspect, of most football crowds to opt out or behave badly."
Following Madejski's comments, Reading said they will hold a minute's silence before Saturday's home fixture against Liverpool - to mark the 24th anniversary of the Hillsborough disaster.
"I think it is the only minute's silence there should be," said Liverpool manager Brendan Rodgers.
"We are entering into a stage of remembrance here for 96 people that died going to a football game and for the families who have, for many years, suffered and have just about got some sort of justification in the last year.
"For me that is our only remembrance at the game on Saturday."
Elsewhere, the England and Wales Cricket Board is advising counties to fly flags at half-mast on Wednesday, 17 April - the day of Lady Thatcher's funeral.