Call it what you will: overdone, overbaked, overproduced. Most of Bruce Springsteen’s albums for the last 20 years have been above-average collections of songs compromised by heavy-handed production. The singer’s latest, High Hopes is no exception, but the problems go back much further than that.

In a string of 1999 concerts in New Jersey, the singer reassembled the E Street Band for a reunion tour, and the first show opened with a galvanizing version of My Love Will Not Let You Down, an outtake from the 1982-4 sessions for the multi-million-selling album, Born in the USA. Danny Federici’s nuclear-strength Hammond organ roared atop Max Weinberg’s surging drums and a battery of crashing guitars.

I couldn’t wait to get home and find the studio original, which had been issued months earlier on Springsteen’s Tracks box set. How did I miss this monster? But when I dug out the recording, the keyboards sounded chirpier and cheerier, sapping the song of the ferocity I heard that night in Jersey. No wonder I had overlooked it.

Similarly, the Born in the USA album tracks always sounded more potent in concert, once stripped of the synthesizers that tried to dance with pop music trends at the time. The album was a massive commercial success and turned Springsteen into a superstar, but the recording draped unnecessary glossiness over visceral songs. Springsteen fans had been spoiled by the immediacy of Darkness on the Edge of Town (1978) and The River (1980), and the stripped-to-the-bone Nebraska (1982). For those of us raised on The Clash and The Ramones, the Springsteen of 1978-82 sounded like a kindred spirit. His version of ‘classic rock’ sounded like it was made by a garage band fighting to be heard.

Soft power

The shift signalled by Born in the USA continued through the next three decades, with diminishing returns. In a recent interview with Rolling Stone, the singer acknowledged that his records had started to sound dated in the ‘90s.

“Through the ‘90s, we lost our recorded intensity,” Springsteen said. “What happened is, we went from being good producers in the 1980s to not very good in the 1990s at producing our own material. We weren’t trying enough. We needed to interact with other people who were making records all the time and making a lot of modern records, and [have] our fingers on the way modern records were sounding.”

He credits respected producers Brendan O’Brien and, more recently, Ron Aniello with modernising his approach. But have they really helped Springsteen make his best albums? Whereas Springsteen has mined a tougher, leaner sound in concert during this period, many of his recordings have gone in the opposite direction and sound bloated and stiff.

O’Brien came on board for The Rising (2002), Springsteen’s first studio album with the E Street Band since the ‘80s. The band had expanded, and with more instruments, there is an unspoken resolve to use as many of them as possible, sometimes to the detriment of the songs. Even as The Rising embraced Celtic music, gospel and country, and tried to tell some hard truths about the post-9/11 world, it sounded soft.

Devils and Dust (2005), an acoustic album roughly in the lineage of Nebraska and The Ghost of Tom Joad (1995), also couldn’t resist some unnecessary decoration. O’Brien strived to complement rough-hewn character studies about a crooked boxer (The Hitter), a desert soldier (the title song) and a child coping with a parent’s death (Silver Palomino) by making them prettier. The rawness in a song like Reno demands to be served cold. Instead it is delivered on a bed of strings, as if trying to cushion its harshness. An unadorned acoustic version performed by Springsteen on an accompanying DVD is far superior.

Human touch

Springsteen got it right with the self-produced We Shall Overcome: The Seeger Sessions (2006), a clutch of songs associated with folk legend Pete Seeger recorded hootenanny style over two days in a farmhouse with a large string band. The singer shouts key changes and cues solos on the fly, approximating the ramshackle intensity of his live performances. And though O’Brien’s big, polished sound returns on the next E Street album, Magic (2007), it’s refreshingly unfussy. Max Weinberg’s drums and a trio of guitars are at the forefront, in an obvious (but mostly successful) return to the sound of Springsteen’s best E Street album, Darkness on the Edge of Town.

Unfortunately, Working on a Dream (2009) is its polar opposite – grand and plush, but not in a good way. With Springsteen emphasising melody, O’Brien channels his inner Phil Spector and then some; the cheese overflows in some of Springsteen’s silliest songs, including the tumbleweed epic Outlaw Pete and a mash note to the Queen of the Supermarket.

Aniello took over for Wrecking Ball. With the Occupy movement as a backdrop, it brimmed with songs about broken opportunity. It had all the makings of an electric sequel to the stark Nebraska, but the songs rarely sounded like a reckoning. The album is primarily a failure of tone. Perhaps Springsteen and Aniello were aiming for a rousing, union-rally sing-along feel, but instead they conjure a sterile, football-stadium roar.

O’Brien produces a handful of tracks on the recent High Hopes, but Aniello once again does most of the heavy lifting. He and Springsteen temper the ferocity and drama of two excellent songs: an electric reworking of The Ghost of Tom Joad and the harrowing American Skin (41 Shots).  Fortunately, there are several spectacular versions of both songs available online culled from his concerts, testimony to Springsteen’s continuing relevance as not only a songwriter but as a performer.

It’s worth nothing that Springsteen hasn’t turned into an oldies act relying exclusively on decades-old hits to carry him through lucrative tours the way many of his peers have. But it’s a shame his studio recordings over the last two decades haven’t been able to consistently document just how intense and moving his best new songs can be.

Greg Kot is the music critic at the Chicago Tribune. His work can be found here.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.