“Politicians promise you heaven before an election and give you hell after,” wrote the anarchist Emma Goldman. The experience of the legendary Doctor Faustus, who sells his soul to the demon Mephistopheles in return for worldly knowledge and pleasure, has been treated as a metaphor for unholy political pacts. It may even shed light on our own populist moment, from Brexit to the election of Donald Trump. Why does this 500-year-old folk legend resonate in times of crisis, and why does it continue to haunt the Western imagination?

With the exception of Frankenstein, it is difficult to think of a more enduring modern legend

The legend is loosely based on the life of Johann Georg Faust (c 1480–1540), an alchemist and practitioner of necromancy, a form of ‘black magic’. A chapbook speculating on his infamous exploits circulated in the late 16th Century, inspiring Christopher Marlowe’s play The Tragical History of the Life and Death of Doctor Faustus, first performed in London around 1592. At approximately the same time, the legend of Pan Twardowski, a sorcerer who sold his soul to the devil, began to take root in Polish folklore.

- Princess Diana’s game of thrones

- ‘The greatest film-maker who ever lived’

- The writers who defied Soviet censors

The most influential interpretation of the Faust legend was written by Johann Wolfgang von Goethe (1749-1832). The project dominated his intellectual life: the first part of his dramatic poem, Faust, appeared in 1808; the second part was completed in 1831, the year before his death. With the exception of Frankenstein, published by Mary Shelley in 1818, it is difficult to think of a more enduring modern legend – both stories reflect unease about the dawning of a new world, full of possibility and anxiety.

The Faust legend has penetrated every cultural space, including classical music and opera (Schubert, Wagner, Berlioz), fiction (Bulgakov, Turgenev, Wilde), poetry (Pushkin, Byron, Heine), and drama (Havel, Mamet, Gertrude Stein), as well as ballet, sculpture and painting. The folklore has suffused popular culture, from The Simpsons to Queen’s Bohemian Rhapsody. It has been the subject of dozens of films, musicals, fairy tales, video games, graphic novels, comics and manga.

The temptations of Fascism dominate 20th-Century Faustian parables

The legend seems to have particular resonance at times of moral crisis. Mephisto (1936), a novel by Klaus Mann, offers a thinly-veiled portrait of an actor who ingratiates himself with the Nazi regime in order to advance his career. This ‘devils-bargain’ (teufelspakt) helps him become Germany’s most celebrated actor, but when he plays Mephistopheles in a production of Faust, he realises that he is acting the wrong part – he has become Faust, morally compromised with evil.

Mann’s father Thomas wrote the most notable post-war treatment of the legend, Doktor Faustus (1948), which the author described as “the novel of my epoch”. The protagonist is a composer who renounces love in exchange for heightened creative powers, which he acquires by infecting himself with syphilis: as Mann wrote in a précis, “The poison works as intoxication, stimulant, inspiration; transports of exaltation allow him to create wonderful works of genius”. Based in part on the life story of Nietzsche, the novel explores how nihilism and primitivism usurp bourgeois culture. In the dying embers of the Third Reich, the physiological, mental and spiritual degradation of Mann’s protagonist becomes a metaphor for Germany’s moral corruption.

Fascists and finance

The temptations of fascism dominate 20th-Century Faustian parables, most notably Friedrich Dürrenmatt’s play The Visit (1956), in which townspeople are offered a bribe to murder one of their fellow citizens. The corrupting influence of money is also a theme of Stephen Vincent Benét’s short story The Devil and Daniel Webster (1936), written at the height of the Great Depression, in which a beleaguered farmer sells his soul for seven years of prosperity. His lawyer tries to argue that the buyer is a foreign imposter, but the devil proclaims he was present at America’s birth: “When the first wrong was done to the first Indian, I was there. When the first slaver put out for the Congo, I stood on her deck” – an inspiration for the Rolling Stones’ song Sympathy for the Devil.

The Faust legend has thrived in a culture of instant gratification

Perhaps inevitably, the theme of demonic bribery has been the subject in electoral propaganda. An intriguing example is an unaired broadcast by the Conservative party in the run-up to the 1997 UK general election. The premise of the five-minute film is unsubtle: Tony Blair is Faust, encouraged to deceive the electorate by a spin-doctor of the ‘dark arts’, Peter Mandelson. The broadcast was cancelled at the last moment on the insistence of Prime Minister John Major, as he feared its negativity would damage his own party and that the analogy would offend Blair, a devout Christian.

Despite its theological underpinning, the Faust legend has thrived in secular consumer societies, particularly in a culture of instant gratification. From credit cards to fast food, we opt for immediate pleasure even in the knowledge that it brings long-term pain. Faustus states that the only God he serves is his “own appetite”, and Goethe’s Mephistopheles offers him the opportunity to “sample every possible delight… grasp at what you want!” In David Luke’s lyrical translation:

Your palate also shall be sated,
Your nostrils sweetly stimulated,
Your sense of touch exhilarated.

The Faust legend gained traction at a time when the ‘closed’ medieval world was being cleaved open by a new mercantile culture. Karl Marx identified the influx of gold from the New World as the dawn of capitalism, a system he compares to a sorcerer who is no “longer able to control the powers of the underworld he has called up”. This system necessitates exploitation and colonisation, and Marlowe’s Faustus sounds like the giddy first capitalist:

I’ll have them fly to India for gold,
Ransack the ocean for orient pearl,
And search all corners of the new-found world
For pleasant fruits and princely delicates.

Faustus embarks on a grand tour, meeting the Pope in Rome, the German Emperor Charles, and the spirits of Alexander the Great, Darius of Persia, and Helen of Troy (to whom he waxes lyrical: “Was this the face that launched a thousand ships?”). It is a dazzling if ultimately unfulfilling excursion: “A great show! Yes, but only a show”. The doctor finds that “riches, pleasures, pomps” cannot stave off his spiritual malaise – a sentiment shared in the 1999 poem Mrs Faust, written by Carol Ann Duffy: “I grew to love the lifestyle, / not the life”.

Goethe’s major innovation is the introduction of Margareta (also referred to as Gretchen), whose story provides the most poignant episode of the drama. Faust pursues her, seduces her, and then – unwittingly – destroys her and her family. Mephistopheles guides his hand but Faust’s actions are unbearably his own (the demon goads him: “Who was it who ruined her? I, or you?”). The Gretchen story has become a powerful cultural motif, inspiring elegies such as Byron’s:

Her faults were mine – her virtues were her own –
I loved her, and destroy’d her!...
If I had never lived, that which I love
Had still been living; had I never loved,
That which I love would still be beautiful.

Faust tells Gretchen: “My sweet, believe me, what’s called intellect / Is often shallowness and vanity”, and almost every iteration of the legend underscores this disenchantment: it is Byron’s Manfred who discerns “the fatal truth, / The Tree of Knowledge is not that of Life”. Intellectual pursuits have isolated Faust, and failed to provide him with wisdom: “The very thing one needs one does not know / And what one knows is needless information”. Even when the quest for knowledge is successful, it conjures up dark forces, as in Frankenstein.

Compromising circumstances

Faust casts aside his scholarship in order to become a man of action, redrafting the opening of St John's Gospel: “In the beginning was the Deed”. Part Two of Goethe’s drama charts Faust’s attempt to forge the world in his own image. It is a distinctly Enlightenment project: Faust will create a new civilisation by taming the wild forces of nature, whose unproductivity fills him with anxiety (“This drives me near to desperate distress! / Such elemental power unharnessed, purposeless!”). His project is beyond human scale and insensitive to human needs, but it is all in the name of progress and “the masses’ bold, industrious will”.

As the critic Marshall Berman points out, this ambitious scheme of modern development is intolerant of those who do not acquiesce to the plan. Faust learns that an elderly couple, Baucis and Philemon, wish to remain in their remote cottage and refuse to be bought off. The couple’s quiet contentment is a rebuke to Faust’s will to power: he is infuriated by their resistance (“Their stubbornness, their opposition / Ruins my finest acquisition”), and he orders their land to be seized.

The ecological and human cost of this insatiable appetite for expansion is evident in the 21st Century. Climate change is perhaps the most fitting contemporary analogy for the Faustian bargain – decades of rapid economic growth for an elite, followed by grave global consequences for eternity. Similarly, the temptation of nuclear energy has been described in Faustian terms: those “powers of the underworld” unleashed, with the potential of fuelling – and destroying – the planet.

Technology has highlighted our daily Faustian choices: who reads the “terms and conditions”? Smartphones make our attention spans fleeting, and we are like Faust, who promises to part with his soul if ever he lingers to savour an experience (“If ever I to the moment shall say: / Beautiful moment, do not pass away!”). Goethe’s epic ends with Faust envisioning the completion of a project that would liberate not only his workers, but also himself from “restless activity”.

“Every notable historical era will have its own Faust,” wrote Kierkegaard. Our challenge today is that, to some extent, we are all in a Faustian bind. We are plagued by politicians offering easy answers to complex problems – especially when those easy answers are empty promises. The legend warns us to be wary of the cult of the ego, the seductions of fame and the celebration of power. These are hollow triumphs, and short-lived; indeed, “what good will it be for a man if he gains the whole world, yet forfeits his soul?”

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital and Travel, delivered to your inbox every Friday.