The Italian's failure to pay enough attention to developing young players was also behind his exit, reports BBC sports editor David Bond.
City's preferred candidate to take over is Malaga coach Manuel Pellegrini, although no deal has been done.
Assistant manager Brian Kidd will take temporary charge of the first team.
Roberto Mancini record
He will be at the helm for the final two league games of the season - against Reading on Tuesday and Norwich on Sunday - as well as a post-season tour to the United States at the end of May, which gives City some breathing space to bring in Mancini's successor.
City's statement said he had "failed to achieve any of its stated targets this year, with the exception of qualification for next season's Uefa Champions League".
But Mancini's problems with players, fellow executives and the board were also major factors in his departure from a post he took over from Mark Hughes in December 2009.
The former Inter Milan manager, who also won an FA Cup with City in 2011, blamed the club's transfer policy last summer for his team's struggles in defending their top-flight crown.
Mancini deserved more time - Pearce
City appointed former Barcelona technical director
as director of football last October to oversee player recruitment, while Ferran Soriano also left the Nou Camp to join City as chief executive.
The relationships between manager and boardroom do not appear to have progressed the way City wanted, as they stated the "need to develop a holistic approach to all aspects of football at the club" in the statement explaining Mancini's sacking.
Former City defender
believes Mancini's man-management did not help his cause.
"There wasn't the togetherness between player and manager," Mills told BBC Radio 5 live.
"Basically, Mancini just ignored players from day one. He was the manager, he made decisions, he made no attempt to have any sort of relationship with the players, didn't take them under his wing."
Mancini, who signed a new five-year deal last July, will receive compensation following his dismissal but it will be for much less than the four years left on his contract.
"We've seen it year after year. That's the way football is today - managers get sacked. I don't feel sorry for Mancini, no," said former City striker Rodney Marsh, who scored 36 goals in 118 appearances for the club between 1972 and 1975.
What do City mean by holistic?
Dictionary definition: 'Holistic' means thinking about the whole of something and not just certain parts. In their statement, it is understood City are referring to all aspects of the club - for example, the youth system and first team - pulling in the same direction.
Brian Horton, who managed City from 1993 to 1995, believes City's next manager will be under pressure to deliver trophies.
Horton said: "He is going to have to win something isn't he? Paying the wages that they pay at these top clubs, you have to be winning, not second. Second is not good enough."
England Under-21 coach Stuart Pearce, who was City's boss between 2005 and 2007, is concerned that clubs are too quick to dismiss managers.
"Even the unsuccessful managers need to be given time, let alone the successful ones like Mancini," Pearce said. "We're too ready, maybe, to write off individuals.
"Roberto won't have a problem walking into another job, so maybe he's not a real great case for me to talk about.
"I know there are a lot of fantastic English coaches, and we're looking for them to develop. Don't laud them too much too soon. On the other side of the coin, they need time for that development, because they're still learning the game."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.