Broken hearts and righteous anger have always made some of the greatest hits. From Dolly Parton’s I Will Always Love You (later recorded by Whitney Houston) to Adele’s Someone Like You and Sam Smith’s Stay with Me, artists have turned their tears to gold.

But does today’s music express an even deeper unhappiness than the songs of the past? That’s the conclusion of two recent analyses examining thousands of US and UK hits from the last few decades. Since the 1980s, emotions such as sadness and loneliness have become increasingly prevalent in song lyrics. Expressions of pure joy, meanwhile – think the Beatles’ All You Need is Love – are apparently less likely to trouble the charts.

What’s the cause of these changes? Does it simply reflect a change in the way music is consumed? Or is it simply expressing the emotional undercurrents of society today?

Tears or fears

Let’s first consider the evidence. Lior Shamir at Lawrence Technical University gathered the lyrics of 6,150 Billboard Hot 100 singles from 1951 to 2016 and fed them to an algorithm. The software had been previously trained identify linguistic markers of different emotional states and personality traits – including sadness, fear, disgust, joy, and extraversion. And although a computer will clearly miss some of the nuances of complex lyrics, its assessments do tend to agree with human judgements.

It correctly identified that the dominant emotion in Bonnie Tyler’s hit Total Eclipse of the Heart, for instance, was ‘sadness’, with a score of 0.51 (out of 1) for that emotion. The Village People’s YMCA, meanwhile, scored 0.65 for ‘joy’, and Queen’s We Will Rock You scored a whopping 0.85 on the scale of ‘extraversion’ – which sounds about right for the rockers’ stomping bravado.

Shamir then averaged the scores for each year and examined how they changed over time. The results were striking. Expressions of anger and disgust roughly doubled over those 65 years, for instance, while fear increased by more than 50%. Remarkably, today’s songs are even more aggressive and fearful than in punk’s heyday. One probable reason for this is the growing influence of rap music, which, like punk, has reflected social unrest and feelings of disenfranchisement. Sadness, meanwhile, remained stable until the 80s, then steadily increased until the early 2010s, while joy, confidence and openness all steadily declined.

“You see a very consistent, very clear change that lyrics become angrier, more fearful, more sad, and less joyful,” Shamir says. “There are very substantial differences between lyrics in the late 50s compared with lyrics in 2015 and 2016.”

To give some examples of this phenomenon, Shamir pointed to a string of hits in the 1950s with the dominant feelings of ‘joy’ – songs like Elvis Presley’s All Shook Up, which scored 0.702 for that emotion, or Little Richard’s Long Tall Sally, which scored 0.82. In contrast, the angriest songs to hit the charts came from the 2000s, including Ne-Yo’s When You’re and Busta Rhymes Tough It (which scored 0.97 on the anger scale). More recently, Taylor Swift’s Bad Blood scores highly on fear, with very little joy, while Miley Cyrus’s Wrecking Ball and Justin Bieber’s Sorry score high on sadness – all of which were some of the biggest hits of the last six years.

Dancing in the dark

The findings broadly agree with a second, independent study by Natalia Komarova, University of California Irvine – a mathematician who had been shocked by the negativity of her daughter’s own music taste. To find out how song emotions had changed over time, she turned to a research database called AcousticBrainz, in which users could apply an algorithm to extract acoustic features – such as the use of major or minor chords and tempo – which it then used to score a song on emotions like sadness. Looking through half a million songs released in the UK between 1985 and 2015, Komarova and colleagues found that the tone of the music had become less joyful since 1985 – just as Lior Shamir’s analysis of the lyrics had also suggested.

Interestingly, Komarova found that the danceability – as measured by features of the rhythm – had increased alongside the negative feelings. So, despite the negative feelings they expressed, the songs were also more likely to get people moving. Just consider Robyn’s hit Dancing on my Own – the pulsing synths and percussion belying the lyrics of loneliness and isolation. In terms of albums, Komarova also points to Beyoncé’s Lemonade and Charlie XCX’s Pop 2 mix-tape as being full of dark but danceable tracks.

Let’s be clear: these are only very broad trends and there are many exceptions to the rule, such as Pharrell Williams’ Happy, and Rihanna’s Diamonds, which were both huge hits with upbeat lyrics. But overall, there does seem to a shift into a darker, angrier, place.

The reasons are unclear, and Komarova is reluctant to offer any specific hypotheses. “[But] one may speculate that this is related to some changes that take place in the society.”

Shamir agrees, pointing out that in the 50s most popular music was a form of escapism – but since the 60s it has been much more socially engaged. “Music changed its role from fun expression to an expression of political views.” Perhaps this can partly explain the shift, but it has to be said that the vast majority of tracks today are not protest songs.

Mood river

If scientists are not certain about what’s driving the trends, I thought it would be worth getting the view from someone who has crafted hit songs over several decades. So, I contacted Mike Batt, the conductor, record producer, singer and songwriter who has penned many hits including Art Garfunkel’s Bright Eyes and Katie Melua’s Closest Thing to Crazy.

Batt points out that the means of consuming music have changed so dramatically – with the streaming of songs now dictating the tracks that enter the charts. These could have determined what kinds of songs become popular – if, for instance, they have meant that older adults, who may be reluctant to stream music, and who may also be less angsty, are having less of an influence on what becomes a hit.

Batt agrees that a more negative tone in today’s pop music also might reflect social changes, though – and even if the reactions to political events are not overtly expressed as protest songs, they may still influence the overall mood.

“Songs tend, either deliberately or not, to hold a mirror up to society, or are at least be affected by what is going on in the world,” he says. “The social media generation is experiencing strongly articulated stress daily. The aggression present in politics and in religious and racial tensions are no greater today than they have always been, but they are thrust more often and more directly into the faces of people. This is bound to reflect in our songs.”

Batt speculates that this may be compounded by the fact that many songs today are written by larger teams of songwriters, who as a group may end up writing songs that fit the mood zeitgeist rather than examining the nuances of personal experiences. “Maybe multiple writers tend to express a more communal view,” he says. And if that mood is generally more negative than in the past, those songs are going to be sadder and angrier.

For his own part, he admits to “waking up each morning a little depressed” by world events, but he says his own songs tend to express personal melancholy – rather than anger or disgust.

And he emphasises that recognising our own sadness can be healthy. “Melancholy makes the world go round.”

It seems likely, then, that pop songs capture the moods of both the artists and the listener. The composer tunes into the zeitgeist, and the resulting sadness or anger is, in turn, more appealing to us as consumers, who recognise our own feelings within the music and lyrics and propel those songs to the top of the charts. The times, they are a changin’, and every hit has a small testament to those lasting shifts.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Music, Culture, Capital, Future and Travel, delivered to your inbox every Friday.