Promoter Frank Warren believes David Haye should retire after failing in his attempt to unify the heavyweight division with a points defeat by Wladimir Klitschko on Saturday.
All three judges scored the Hamburg fight in Klitschko's favour.
"I think he should retire," Warren, 59, told BBC Radio 5 live's Sportsweek.
"He's got money in the bank so why does he need to fight on? He was a great cruiserweight but hasn't got it at this [heavyweight] level."
If Haye does step away from boxing then he will leave with a record of 25 wins, 23 by KO, and two losses.
The London boxer moved up to heavyweight in 2008 and opened with a win over Monte Barrett before beating Russian Nikolai Valuev a year later to take the WBA crown and become the first British boxer since Lennox Lewis to hold a world heavyweight title.
Warren added: "Valuev was one of the worst heavyweights and Harrison was a non-fight. The contest against Klitschko was his first defining fight in this division and I don't think he won a round.
"He got well beaten. He wasn't in it and didn't show up."
British former WBC champion Frank Bruno said Haye will decide for himself whether it is the right time to retire.
"I'm in no position to say that Haye should retire," the 49-year-old told Sportsweek.
"Haye will do what Haye wants to do. He'll be remembered as a very, very good cruiserweight. But heavyweight and cruiserweight are two different cups of tea. Harrods and Primark."
Haye said he would retire from the sport by his 31st birthday in October but former opponent Harrison believes it is unlikely he will make good on his promise.
"I'm living with the same thing now: it's very hard for me to think about retiring coming off a defeat and the way I lost," said the 39-year-old 2000 Olympic super-heavyweight champion.
"I know David's a proud fighter and I know retiring on that performance will live with him and he'll want to come back in a couple of years' time to rectify that.
"So I expect you're going to see David Haye back in the ring again just like you're going to see Audley Harrison back in the ring again."