The chairman of the Pompey Supporters' Trust Ashley Brown says Richie Barker was the wrong man for the club and did not fit in around Fratton Park.
Barker was sacked after only 20 games in charge of the League Two club.
Brown, who is a director at Pompey, said urgent action was needed to preserve their Football League status.
"We felt Richie was the wrong manager, at the wrong place at the wrong time and did not fit in at Portsmouth," Brown told BBC Radio Solent.
Pompey won four of their matches under Barker, 38, and have dropped to only two points above the relegation zone.
Academy boss Andy Awford has been placed in temporary charge of the first team for Saturday's game against Newport County at Rodney Parade.
Brown revealed the decision was only taken on Thursday morning but the board were unanimous in replacing the former Crawley boss - who took over from Guy Whittingham in December.
"It was very difficult, it was not something we wanted to do twice in a year since we have taken over," he said.
"The situation dictated that we had to take action. The most important thing is keeping Portsmouth Football Club in the Football League.
"It was not happening here, I think he will go on and be a great manager somewhere else but we did not see him as the man to get us out of trouble.
"The league position and results speak for themselves but more so we didn't see fight or spirit from the players."
Brown says they considered bringing in an external candidate to take over but their attempts were scuppered.
"We need an impact manager and somebody who is going to get results now as it's important we stay in this league," he said.
"Unfortunately the external options we had from outside were not available so we looked back inside. We have capable staff here who have fight and we thought that would be the best way to get everybody back behind this football team."
Brown, who helped lead a supporter consortium to save the club almost a year ago, also says they are now preparing for life outside of the Football League, should the worse happen.
"We'd be crazy if we did not prepare for that potential outcome," he said.
"So of course the management team here are doing work on what we would do in the Conference. The club can survive in the Conference, it is not where we want to be and the board and caretaker manager believes we can get out of it."