For every hit like Moulin Rouge! or Les Misérables there are flops like Rent, Nine and Jersey Boys. Tom Brook explores how the declining genre can renew itself.

In late June, Clint Eastwood’s efforts to turn the hit Broadway musical Jersey Boys into a successful film failed to light up the US box office. But Begin Again, a musical romantic comedy starring Keira Knightley, opened to quite solid business despite being exhibited in only five cinemas. It still faces an uphill battle: looking back over the past year there are several musical films that have failed and only one, Frozen, that was a major triumph. Bringing musical films to the big screen is a business fraught with difficulty.

At one time, during the golden age of the Hollywood musical from the 1930s to the 1960s, the genre was a confident staple yielding entertainment with a worldwide following. After all, who can forget Gene Kelly in Singin' in the Rain or Julie Andrews bursting into song in The Sound of Music?

But since 2000 there have only been a handful of big screen musicals that have really succeeded. Chicago, Dreamgirls, Mamma Mia, Les Misérables and Moulin Rouge are the ones that immediately come to mind. Among the more notable casualties are Nine, The Phantom of the Opera and Rent – which all had been initially presented on stage.

For musical theatre and film historian John Kenrick the right talent has to be in place for a musical to do well – a case in point is the Oscar-winning film adaptation of Chicago in 2002. “[The] people involved in Chicago had solid backgrounds in musical theatre and film. The director Rob Marshall had been a Broadway choreographer long before he attempted to film Chicago,” he says.

Kenrick believes that so many musicals fail because the directors who take them on don’t have the right skills. “What’s happened is [that] far too many people don’t understand that musicals are a very demanding art form – and it’s practically impossible for any filmmaker to suddenly make a successful musical when they have absolutely no background in doing so.”

That is perhaps one reason why Jersey Boys stumbled. Clint Eastwood’s track record includes directing the Bird (1988), a musical biopic of American jazz saxophonist Charlie Parker. But Jersey Boys was a far more elaborate undertaking: an adaptation of the Broadway stage show that tells the story of the 1960s American pop group  The Four Seasons.

“Jersey Boys has been, so far, a resoundingly unsuccessful project,” says Wheeler Winston Dixon, Professor of Film Studies at the University of Nebraska-Lincoln. Dixon doesn’t point the finger necessarily at Eastwood, but he’s not alone in thinking that the director isn’t one you’d associate with this kind of project. “Clint Eastwood directing a musical. It doesn't really seem like it would be the most obvious choice, let’s put it that way,” he says.

Suspending disbelief?

It’s not just the talents of a director or a cast that determine whether a movie musical will work - there’s an ongoing challenge with the audience. Many moviegoers are uncomfortable with the implausibility of performers suddenly bursting into song. Bollywood fans have no problem with this aspect of the genre but audiences in the West often do. “There’s something about musicals that is so inherently unreal,” Dixon says. “Most times you can lose yourself in a film, but if people are talking and all of a sudden they burst into song, it pushes you back, and you’re conscious that you’re watching a film – it’s difficult.”

Some onscreen performers share his view. “I personally am not a musical fan,” says actor Mark Ruffalo, who is one of the stars of Begin Again. “I don’t [dislike]them but I don’t go to see them, and so I’m very sensitive to the idea of some[one] bursting out into song all of a sudden. It’s kind of a little jarring to me.” Ruffalo has a non-singing role in Begin Again, which is a musical film in which the singing is more naturalistic because some of the actors are portraying characters who are musicians.

Ruffalo’s co-star Keira Knightley disagrees – she likes traditional musicals and thinks there is an audience for them. “I think it’s very clear that Mamma Mia’s done very well – and they burst into song – and Les Mis did very well, so clearly I think young audiences do like that,” says Knightley.

Directors are sufficiently mindfulof audience concerns that they may try to disguise what they’re doing. John Carney, the Irish filmmaker behind Begin Again, who also wrote and directed the 2007 hit musical film Once, tries to avoid presenting his films as overly song-laden so as not to alienate audiences.

 “I’ve avoided that by doing what I’m now calling the “stealth musical’,” he says. “You don’t realise it’s a musical until you’ve left and your friend says, ‘We just sat through eight songs… that was a musical.’”

 “I don’t think I’d go necessarily to a musical if somebody told me that I’d have to sit through all these songs,” he adds. “But if they’re well incorporated into the drama then they’re sort of disguised a little bit.”

Carney’s fans are impressed by the way he brings music and song into his films. “It’s the organic way he makes the music a part of telling the story – but doesn’t use music to tell the story,” says musician Adam Levine, lead singer of the band Maroon 5, who also stars in Begin Again in a musical role. He recalls scenes in Once where the two main characters, both musicians, might sing. “They’d be in a piano store and then they’d play the song very naturally and it wouldn’t be like someone launches into song randomly out of nowhere,” he says.

Broadway to the bank

Even though some audiences might be uncomfortable with the implausibility of the traditional musical there’s definitely a significant demand for them – the box office performances of Les Misérables or Mamma Mia! are testament to this. So given the right combination of talent and resources could a new golden era of Hollywood musicals emerge?

Structural change at the studios suggests this is unlikely. “That golden age of film musicals required that each of the studios have an in-house team of people who worked fifty-two weeks a year creating musical films, original musical films,” Kenrick says. “Without that kind of investment it just isn’t viable.”

So what can we expect from the movie-musical form in the near future? It’s long-term fate may be influenced by the success or failure of two big spectacles due for release at the end of 2014.

Into the Woods, an adaptation of Stephen Sondheim’s Broadway musical appears to be loaded with talent – Meryl Streep, Johnny Depp and Emily Blunt lead a cast directed by Rob Marshall, who also helmed Chicago. But Marshall’s track record with movie musicals isn’t flawless – he also directed Nine, which was a major flop.

Then there’s a new adaptation of Annie that stars Jamie Foxx, Cameron Diaz and Quvenzhané Wallis and is directed by the relatively unknown Will Glock. Kenrick doesn’t hold out much hope. “None of the people connected with it have what I would call musical or stage credits. You’d like to see some experience across the table.”

When it comes to reviewing the health of the big screen musical one inescapable fact is the very impressive box office performance of Frozen, last year’s animated musical film. It brought in more than $1.26 billion worldwide to become the fifth highest-grossing film of all time. While audiences may have some reservations about films set in a “real” world where characters burst into song it appears that computer-generated figures singing away in a pure animated fantasyland, as they did in Frozen, is totally intoxicating entertainment.

For this reason the animated musical may now be the way forward. Carney’s Begin Again, a more integrated and naturalistic style of musical, also shows promise, albeit on a small scale.

But despite all the difficulties filmmakers aren’t likely to give up on the traditional musical  because when they truly succeed, as in a Moulin Rouge! or Chicago, those productions can be both magical and unforgettable.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.