New Southend United manager Phil Brown will manage the club in the Johnstone's Paint Trophy final on 7 April.
The club had said former boss Paul Sturrock, who was sacked last Sunday, would "lead out" the team at Wembley.
But Sturrock told BBC Essex: "I feel unrest caused to the players and fans on this fantastic day might spoil their enjoyment.
"It was obviously a difficult decision, but I do feel it's the right one as a professional man."
The Scot plans to travel with the supporters, ''have a few beers" with them before the game and "enjoy watching my team as a true fan".
Sturrock was after a run of only one win in their last six League Two games left the side six points adrift of the play-off places.
However, the 56-year-old was told he could take charge of the side to face Crewe at Wembley, even though Brown would have presided over two league fixtures by then.
Sturrock said: "After several days deliberation, and after discussions with the chairman today (Thursday) in which he re-iterated his backing for me to manage the build-up and game, I decided not to accept the chairman's sincere offer.
"I thank the fans for their massive support since my appointment and also their amazing backing for me to manage the final."
He admitted he was disappointed not to be able to lead the side out at Wembley, but was looking forward to sitting with the fans.
"It will be the first time I've ever sat with the crowd at a football game," he said.
"I'll be rooting for the players. It will be a good way to draw a line under my time at the football club, and I'm looking forward to the whole day."
Southend were Sturrock's eighth club as a manager and he has already turned his attention to seeking number nine.
"I feel there's another job in me and I'm very hopeful I will stay in the game. Starting today it's all guns blazing to get another job," he added.