As Pete Townshend’s opus turns 45 this month, Greg Kot charts the evolution of the form that was so popular in the 1960s and ‘70s – and where you can find it today.

“Rock is dead, long live rock,” Pete Townshend once declared. The same might be said of the rock opera. Even in an age when portable playlists seem to be on permanent shuffle, there’s still room for a magnum opus or three.

Townshend’s Tommy is the mother of all rock operas – not the first, but certainly the first famous one. It has gone to Broadway and its best-known songs, from Pinball Wizard to The Acid Queen, have become pop standards, thanks to covers by Elton John and Tina Turner. But 45 years ago on 23 May 1969, when it first arrived as The Who’s fourth studio album, Tommy was a radical statement: a story spread across two vinyl LPs that aspired to the heft of Wagner and the spirituality of Meher Baba. It dared to call itself an “opera”, and it turned Townshend’s UK quartet into stadium-filling heavyweights around the world.

Such ambition – or was it pretentiousness? – ruled the 1960s and ’70s. Once the album replaced the single as the coin of the rock-star realm with the arrival of The Beach Boys’ Pet Sounds and The Beatles’ Sgt Pepper’s Lonely Hearts Club Band, the concept album (a cycle of thematically linked songs) and the rock opera (a story told through music) became the great white whales that every ‘serious’ artist chased. 

Where’s that drive today for the big, sprawling Moby Dick of albums? Some might argue that the notion of an ‘opera’ in popular music has gone the way of the eight-track tape. With the transition in the last decade to online peer-to-peer distribution, the rise of iPods and smartphones, and the preference for shuffle-mode listening, singles and individual tracks have taken precedence over albums. Social issues are at play, as well: kids these days just have too much going on. They multi-task, they have countless entertainment options to choose from at any moment (texting, tweeting, snapchatting, clicking a video, sharing a song, trolling Instagram) and their attention spans are allegedly dwindling to nanoseconds. So who has a spare 60 minutes to wade through the maze-like contours of the 2014 equivalent of Tales from Topographic Oceans?

But concept albums are tougher to kill than Keith Richards. We may not be in the midst of a run of Tommy-like works that transform a culture the way The Who once did. But artists are still intrigued and inspired by the long-form format, and are still recording works that aspire to be something more than just a collection of unrelated tunes.

The reason? Storytelling, character development, the possibility of creating an immersive experience between the headphones remain powerful incentives for artists and an attraction for diehard fans who want to take a deeper dive with the bands they love. In the same way that viewers binge-watch multi-part TV series such as True Detective or House of Cards, music lovers don’t want to subsist only on a steady diet of singles. IIt’s far too pat – and inaccurate – to say singles have bounced albums as a listening experience.

High concept

Even in the ‘70s, rock operas co-existed with singles. Here was a time when concept albums seemed to appear every few weeks. In 1972, there was Jethro Tull’s Thick as a Brick, David Bowie’s The Rise and Fall of Ziggy Stardust and the Spiders from Mars and Gentle Giant’s Three Friends. Lou Reed’s Berlin, The Who’s Quadrophenia and Yes’ Tales from Topographic Oceans followed in 1973. Genesis presented The Lamb Lies Down on Broadway in 1974, Willie Nelson introduced the Red Headed Stranger in 1975, Rush hurtled toward 2112 in 1976, and Pink Floyd built The Wall in 1979.

Though the rock-opera rush faded a bit, landmark concept albums/song cycles/operas continued to appear. Hüsker Dü’s Zen Arcade inspired legions of indie rockers when it was released in 1984, not the least of which were the Minutemen, who responded with Double Nickels on the Dime. Queensryche delivered the progressive-metal Operation Mindcrime in 1988, and Marilyn Manson alarmed parents with Antichrist Superstar in 1996. Trent Reznor’s Nine Inch Nails and Billy Corgan’s Smashing Pumpkins spent most of the ’90s labouring over albums that had opera-like story arcs.

The new millennium brought a new surge of concept-heavy releases, led by Green Day, who graduated from pop-punk singles to the Broadway-bound American Idiot in 2004. The Drive-by Truckers built a national reputation on the back of their acclaimed Southern Rock Opera in 2001. My Chemical Romance’s The Black Parade – featuring a cameo by ultimate Broadway diva Liza Minnelli on a song cycle about a dying cancer patient – turned into a huge 2006 hit, with 3 million sales worldwide.

Mastodon cemented their reputation as metal innovators with the 2010 Crack the Skye, which traced the space travels of a quadriplegic who specialises in astral projection. And Arcade Fire’s 2010 release, The Suburbs, won no less than a Grammy Award for best album of the year. Janelle Monáe, Lupe Fiasco, Kendrick Lamar, The Roots and The Decemberists have all bucked the singles trend with concept albums/operas in recent years.

Many of these artists were inspired by similarly ambitious albums when they were first falling in love with music. They appreciated the way words, music, images and packaging worked together to create the sonic equivalent of a great novel or movie. Earlier this year, Rosanne Cash released The River and the Thread, a concept album about journeying through the South and getting reacquainted with her southern roots. The thought of doing a set of thematically linked songs that tell a story is “a crazy thing to do,” she told me. “But I didn’t let that stop me. I’ve always loved concept albums. I grew up with them, so it’s only natural to be making them.”

Greg Kot is the music critic at the Chicago Tribune. His work can be found here.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.