Pirates of the Caribbean have been a source of fascination for more than 300 years – and they express an anxiety about modern society, writes Roger Luckhurst.

Talk about a stash of golden doubloons. The Pirates of the Caribbean films have grossed more than $4 billion (£3.14 billion) worldwide, with the fifth and most recent instalment, Dead Men Tell No Tales, still playing in cinemas. Even if we may not see Johnny Depp as Captain Jack Sparrow again, the global audience remains seriously invested in the mythology of pirates and buccaneers of the 18th Century. There’s even been an ‘International Talk Like a Pirate Day’ every 19 September since 2002. But why? Why are we horrified by acts of modern piracy in the shipping lanes off the East African coast, yet celebrate the ruthless acts of violence and theft from pirates like Blackbeard, ‘Calico’ Jack Rackham and ‘Black Bart’ Roberts in the 1720s?

Even without realising it, we echo the sentiment: “Yo ho, yo ho! A pirate's life for me.”

The answer lies partly with the 200 years of popular culture that has romanticised these figures hovering on the edge of the law. The deeper reason? Our profoundly ambiguous relationship to the rise of the modern bureaucratic state where social behaviour is so regulated that the untrammelled freedom represented by piracy can be alluring – also why Western gunslingers, Prohibition-era gangsters, medieval knights, and even that scruffy space smuggler, Han Solo, hold such appeal. Even without realising it, many may echo the sentiment of Disney’s original Pirates of the Caribbean theme: “Yo ho, yo ho! A pirate's life for me.”

The so-called ‘Golden Age’ of Caribbean and Atlantic piracy runs from about 1690 to the executions of Blackbeard in 1718 and Black Bart in 1722. This era was a result of a quirk of maritime law. At the time, nations at war were able to declare captains of ships ‘privateers’. These captains had an official ‘letter of marque’ that allowed them to attack any enemy ship and take their goods, provided they gave a cut of the booty to their own government. The term ‘buccaneer’ was applied particularly to the English, Dutch and French ships that preyed on rival Spanish ships at the time. Alongside official navies, then, these were sanctioned mercenaries. Imperial governments have always relied on such figures at the furthest stretch of their power, where dubious legality and official deniability can be extremely useful.

The problem came when alliances shifted or peace was declared. Captain Kidd had been in and out of the Royal Navy and worked as a legal privateer in the 1690s, but he became an illegal pirate, pursued across the Atlantic for what had become crimes, and was executed by the British in 1701. After the Treaty of Utrecht was signed in 1713, thousands of privateer captains and crews found themselves without a job unless they returned to the Navy or merchant shipping. Those who carried on in the old way were pirates. Captains like Edward Teach were now on the other side of the law, and so he transformed himself into the legendary Blackbeard. This trade thrived off the coast of Africa and in the Caribbean, where the islands were full of vast riches from slave plantations, but were held as colonies by a patchwork of competing European powers, so that legal authority in local waters was always weak.

The last of the Golden Age crews were chased down by official navies and executed by the 1720s. However, periods of war often leave a wake in which piracy briefly flourishes again. When the Napoleonic Wars came to end, there was a flurry of piracy, particularly off the coasts of Latin America, where pirates such as Benito de Soto gained international notoriety after his trial and execution in 1830. Piratical activities off Somalia peaked in 2003 in the absence of a strong central government, a vacuum that was filled by local warlords. Piracy there has declined since international co-operation has underpinned the stronger policing of shipping lanes.

Printing the legend

How pirates slipped into popular legend is easy to trace. Just as the Golden Age was coming to an end, a certain Captain Charles Johnson (a pseudonym that no one has cracked) published a famous compendium of the biographies of these desperados, called A General History of the Robberies and Murders of the Most Notorious Pyrates (1724). It was full of bloodthirsty detail, breathlessly narrated, and secured the legends of Blackbeard, Black Bart and a host of others. Pamphlets and broadsides detailing the terrible crimes and suitable punishments of legendary criminals have been bestsellers since printing began, and pirates were popular subjects in the early 18th Century. The author Daniel Defoe wrote several pamphlets about pirates around the time he completed his famous story of shipwreck, Robinson Crusoe (1719). Johnson’s book was a publishing sensation, and a second volume of somewhat more fanciful accounts appeared in 1728.

Blackbeard adorned his beard with lit matches to look ‘like a fury from hell’

Johnson’s book has been the source of nearly every cultural representation of the pirate ever since. It is easy to see why, from passages like this:

“He assumed the cognomen of Blackbeard from that large quantity of hair which, like a frightful meteor, covered his whole face and frightened America more than any comet that has appeared there a long time. The beard was black, which he had suffered to grow of an extravagant length; as to breadth it came up to his eyes. He was accustomed to twist it with ribbons, in small tails.”

Going into battle, Johnson says, Blackbeard adorned his beard with lit matches to make himself appear “like a fury, from hell.” Brutal punishment and death is of course the outcome of his rogue adventures, but even Johnson ambiguously declares the murderous Blackbeard a “courageous Brute”. Ian McShane has memorably played Blackbeard in the Pirates films, but these evocative details already suggest sources beyond the shambolic style of Keith Richards for Johnny Depp’s Jack Sparrow.

Johnson’s General History directly inspired Robert Louis Stevenson’s Long John Silver in Treasure Island (1883) and Captain Hook, allegedly Blackbeard’s former bo’sun, in JM Barrie’s play, Peter Pan (1904). These characters secured the pirate as a liminal figure of fear and fun in children’s fiction, but adults had long been rendered breathless by one of the first mass publishing sensations of the modern age, Lord Byron’s epic poem The Corsair in 1814. Corsair was the French term for a privateer, and Byron was particularly focused on their ambiguous status at the edge of the law: “He knew himself a villain – but he deemed/The rest no better than the thing he seemed.This book, from the pen of a scandalous libertine poet, sold 10,000 copies on the first morning of publication. Superstar Byron, too, fused his image with the legal renegade of the pirate and that persona has suffused our conception of the Romantic poet ever since. It’s a direct line of descent down from this figure to the strutting and preening of the ‘70s rock star.

Plunder and profit

Robert Louis Stevenson, another scandalous bohemian writer, wrote Treasure Island a chapter a day on the hoof to entertain his children. It was a key text in reviving the form of the romance in the late 19th Century, a form that Stevenson avowedly proclaimed to be “sensual and illogical” and actively designed to resist the stifling moral and intellectual restrictions of the Realist novel that was dominant at the time. Stevenson’s romance, stuffed with peg legs, treasure maps and talkative parrots, was another huge seller. By this point, the thrilling throwback of the Golden Age pirate seems inextricably linked to mass culture in Europe and the US.

This association was only reinforced by early Hollywood. The novelist Rafael Sabatini was responsible for introducing Captain Blood in 1922, remembered now as being portrayed in the movies first in the silent era by Douglas Fairbanks in the ‘20s and then in the talkies by Errol Flynn in the ‘30s. We owe the affected West Country accent and rolling ‘Arrrghs!’ of the pirate to Robert Newton’s portrayal of Long John Silver in Treasure Island (1950). These swashbucklers and subversive dandies were less earnest than many of the heroes in Hollywood films of the time – and thus more appealing.

The allure of the pirate is more than just as a romantic figure on the margins of the law. The Golden Age pirates appear just as the legal forms of modern nationhood and international trade agreements were developing, with increasing statutory, social and moral restrictions on the expression of the self. Many pirates – including numerous women – refused to return to the unrelenting and brutal hierarchy of the Royal Navy, merchant shipping or even civilian life on land, and lived instead by a wholly other code of honour. These pirate codes, some of which have survived, read like statements of anarchistic community, with collective decision-making, no hierarchy, elected leaders and an equal distribution of labours and profits. Also: lots of rum.

This is why the violence, murder and rape associated with them is so often airbrushed out of history. The fictional pirates since Johnson and Byron have offered a fugitive vision of acts of radical freedom that refuse dreary administrative modernity. We cheer on Jack Sparrow’s latest escapade, and keep no count of his killings, because his sensuous, madcap roguery is our fantasy projection from inside what the sociologist Max Weber called our “iron cage of rationality”. In an era where voters have rejected career politicians and bureaucrats for the wild adventures of our own brand of reckless buccaneers, I suspect that the idea of the pirate will continue to speak to our times.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital and Travel, delivered to your inbox every Friday.