Art at the turn of the last century was not all sun-kissed Monet gardens. It was a time of angst and decadence, expressed through some truly disturbing paintings, writes Fisun Güner.

When we think about art at the end of the 19th Century, who and what comes to mind? Monet and Impressionism, certainly. Toulouse-Lautrec at the Moulin Rouge, perhaps. Post-Impressionism, of course: Cézanne and his heavy-set cardplayers or Mont Sainte-Victoire shimmering on the horizon, magnificent and majestic; Gauguin in his Tahitian paradise; or the last ravishing landscapes of Van Gogh, who died just as the last decade of the century ­was getting into its stride.

Art at the end of the 19th Century is as far removed from Monet’s sun-dappled garden as you can get

But when we think of the art that’s actually characterised as the art of the fin de siècle, particularly the last decade of that century, the mood changes, and it darkens. We think of the art of anxiety and angst, of drama and febrile tension, of an acute sense of alienation. And it’s all as far removed from Monet’s sun-dappled garden at Giverny as you can get.

We might, for instance, think, most famously, of Munch’s Scream, the first version of which the troubled Norwegian artist painted in 1893, or his various depictions of women as vampires, slaking their sexual thirst on unsuspecting men. Or perhaps our thoughts turn to the young and eccentric English illustrator and printmaker Aubrey Beardsley and his darkly erotic, sinuous vixens – exotic femme fatales who could captivate and destroy any man. And as the embodiment of lust and evil, naturally the femme fatale becomes a mascot of the fin de siècle, just as much as the dandy, that ultra-refined aesthete who rises above ordinary moral concerns, becomes its icon.

Or we might even think of an artist who is currently being shown at the Royal Academy in London: the brilliant Belgian painter James Ensor, whose rich palette glows with Rubenesque colours but whose subject matter is dark and satirical: skulls and skeletons and eerie masks, all representing the corruption at the heart of bourgeois society.

Or his fellow Belgian, the lesser known but equally brilliant Léon Spilliaert, whose thin and haunted features stare out forlornly in his self-portraits, presenting himself as a strange, cadaverous figure trapped in gloomy and oppressive interiors – for Spilliaert, too, is at the Royal Academy, where he complements the work of Ensor, his older contemporary who also lived at Ostend.  

Or, in fact, we might even return to Gauguin, who in some ways became a figurehead for these disquieting forces at the end of the century. Gauguin’s fascination with religious and mystical subject matter, and for the purely imaginary and fantastical, were synthesised in his paintings with the ordinary, everyday world that surrounded him - just as James Ensor’s work also provided a synthesis of the real and imagined.

What lies beneath

Ensor grew up above a curio shop in the cold, coastal town of Ostend, where his mother sold trinkets, costumes and grotesque carnival masks to tourists. At first Ensor painted in a loosely Impressionist style, but he retained his childhood fascination with these masks and was soon incorporating them into his work. His favourite motif became that of the surging crowd, where ossified faces are leering, threatening masks that overwhelm the whole picture. For Ensor these props seemed to provide the perfect metaphor for the hypocrisies of polite society.

Ensor’s faces are leering, threatening masks that overwhelm the whole picture

In one painting, The Intrigue, 1890, which forms the centrepiece of the Royal Academy exhibition, a group of grotesques with barely human faces congregate in a bizarre wedding group, the aged bride and her ghoulish top-hatted groom grotesquely inverting family and Christian values to be found in the embodiment of the marriage ritual.  in one self-portrait, painted a year earlier, we find the young Ensor in an elaborate, plumed hat, making him appear slightly ridiculous. Surrounded by a swarm of terrifying masks, he stares out at us with a stern and frank gaze. Is he accusing us of some unspecified crime against him, or simply imploring us to bear witness to his suffering?

In other canvases, Ensor  paints himself as either a skeleton or as a tormented Christ-like figure. In his monumental Christ’s Entry into Brussels in 1889 (actually painted in 1888), Christ himself is seen entering the city on a donkey, but is barely discernable amid the crushing crowd – the implication being that these self-obsessed and conservative scions of the city would barely notice the messiah arriving amongst them. And it may be a sidelong reference to Ensor’s own overlooked genius. Like Munch, one gets a sense that Ensor suffered from an acute persecution complex, though unlike Munch, Ensor’s work has a darkly comic edge to it. His fraught paintings bristle with humour and are all the more vivid for that.

So why did artists revel in such outward expressions of unease and dislocation? In an era of relative peace and stability and, for the few, economic prosperity (an era named, after the destruction of the Great War, as the Belle Époque or Golden Age and which stretched from the 1870s to the war’s outbreak) the art of fin-de-siècle Europe expressed something contrary to those outward signs of confidence. These were anxieties connected with a sense of society’s spiritual emptiness and its growing materialism. This was a rejection of the idea that progress and reason, ideas which intellectuals had embraced and promoted since the 18th Century as Enlightenment ideals, could sustain the spirit.

It signalled a deeper anxiety: our inability to control our own destinies.

But perhaps, in a way, these were also anxieties exacerbated by the end of any century. It might sound a trivial connection, but in our own age we might think back to the alarm over the Millennium Bug, where people actually imagined planes falling out of the sky due to programs having accommodated only two digits instead of four (computers would think, when the hour struck, they were back in 1900).

It signalled a deeper anxiety that is perennial: our inability to know and control our own destinies. Where Do We Come From? What Are We? Where Are We Going?,the title from an 1897 painting by Gauguin, seemed to capture this quest for that deeper knowledge – and he and many other artists of the time looked for answers not in science but in esoteric spirituality, in mysticism and often the occult.  

Darkness and despair

The fin de siècle encompassed the Symbolist and Decadent movements, which were primarily literary movements flourishing first in France but then all over Europe, and which provided a wellspring of ideas for visual artists. As well as Ensor and Spilliaert, one of the most notable artists in Belgium was Fernand Khnopff, who, like Ensor, was a member of the Belgian avant-group group Les XX and whose most striking painting, L’Art (or The Caresses), 1896, depicted a sphinx with a ‘real’ woman’s head and the body of a leopard, her face the picture of ecstasy as she is shown caressing a male figure (Oedipus).

It’s a compelling and unsettling image, made stranger for the meticulous realism of its technique with the enigmatic fantasy of its subject. Elsewhere were the weird Goya-esque fantasies of Austrian Alfred Kubin, and, in Germany, the beguiling and erotic paintings of Franz Stuck, who co-founded the avant-garde Munich Secession group in 1892. A generation older, we might also think of the sweeter though no less haunting dream-visions of French artist Odilon Redon.

In literature, one of the most influential works of the Symbolist and Decadent movements was Joris-Karl Huysmans’ À Rebours (Against Nature). Published in 1884, its young anti-hero is an eccentric, reclusive aesthete called Jean des Esseintes who loathes bourgeois society and so roundly rejects it. Instead he surrounds himself with nascent Symbolist poetry and art, and lives a life of excessive sensual indulgence.

Huysmans’ themes certainly found an echo among many artists of the period. The French writer Barbey d’Aurvilly’s reponse to his seminal novel of Decadent literature was revealing in its insight: “For a decadent of this calibre to emerge and for a book like Monsieur Huysmans’ to germinate within a human head, we must have become what in fact we are – a race in its last hour.”

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.