In a binge-reading, box set-watching culture, the humble serial is having a revival. Hephzibah Anderson looks at some of the best storytellers, from Scheherazade to JK Rowling.

It’s a loaded pistol in a crowded room, the lights failing seconds before gunshots ring out. It’s a car pulling into the driveway of a house while inside a trail of clothes leads to the bedroom and a cheating spouse lies entwined with her lover. It’s a question or a revelation or something suddenly remembered just as the train leaves the station. Above all, it’s climax interrupted. You may not see a cliffhanger coming but you certainly know when you’ve been left dangling.

As a literary device, cliffhangers tend to be shunned by highbrow authors. They smack of genre fiction and perform best, you might suppose, in novels that belong in a series, whose writers need to keep readers eager during the months it takes to crank out the next instalment. Little wonder that it came of age during the 19th Century, when fiction was for the most past consumed serially, in magazines. As the now-forgotten novelist Charles Reade quipped of his craft, “Make ‘em laugh, make ‘em cry, make ‘em wait – exactly in that order”.

But for all that they’ve grown to be associated with pulp fiction – or more damningly still, television soap operas – cliffhangers are a serious business. Just ask Scheherazade, whose life depended on them as she unfolded her never-ending tale for her bloodthirsty listener, night after night. They’re also big business, catapulting the likes of Charles Dickens to enduring international stardom, and helping make fortunes for latter-day novelists such as JK Rowling.

Chapter and curse

One man who’s fully appreciative of the power of the cliffhanger is Julian Fellowes. After five years of coaxing viewers to obsess over the upstairs-downstairs shenanigans at Downton Abbey, the master manipulator is looking to pull off a similar feat on a still smaller screen: the one on your phone or tablet. Harnessing 21st-Century technology to revive that Victorian staple, the serial, Fellowes’ new novel, Belgravia, will be released in instalments as an app. Technophobes can wait until June, when it will become available as a hardback, but the rest of us will be able to read – or listen to – a chapter a week over the course of three months.

The story opens at the Duchess of Richmond’s famed ball on the eve of the Battle of Quatre Bras in 1815, and according to the author’s website, each chapter promises “twists and turns and cliffhanger endings”. It’s these endings that will prove crucial to Belgravia’s success, because though there’s a small financial incentive to subscribe to all 11 episodes at once, a week-by-week option means he’ll have to keep fans on tenterhooks in between instalments.

Literature’s ‘Who shot JR?’ moment came three decades later

The current glut of good scripted television has made us less snobby about the cliffhanger. Surrendering to shows like Breaking Bad and Homeland, we’re reminded that it can be about something more haunting, not just egregious audience manipulation. But with TV embracing the device so tightly, it’s easy to forget just how literary the cliffhanger’s roots are. The term that we use to describe Scheherazade’s trick was in fact coined in response to a novel by Thomas Hardy. A Pair of Blue Eyes was published serially in Tinsley’s Magazine in 1873. At the end of a chapter, Henry Knight, one corner of a love triangle, is left literally hanging off a cliff. Readers had to wait an entire month to find out whether he survived. Literature’s ‘Who shot JR?’ moment came three decades later, when Dickens’ fans, eagerly awaiting the next instalment of The Old Curiosity Shop, rioted at the New York docks. Little Nell was perilously ill in the previous chapter, and their anguished anticipation was such that as the boat from England arrived, they began shouting at the sailors, begging for spoilers.

As the boat from England arrived, they began shouting at the sailors, begging for spoilers

Nor is it only serial-spinners who weave cliffhangers into their tales. Shakespeare used them – think of Romeo and Juliet or the opening of The Tempest. So did Evelyn Waugh in Vile Bodies, John Fowles in The French Lieutenant’s Woman, Ian McEwan in Atonement and Margaret Atwood in The Handmaid’s Tale.

Stringing it out

In psychology, the desire that cliffhangers tap is known as the Zeigarnik effect, named after the Soviet psychologist Bluma Zeigarnik. Watching waiters in a Viennese restaurant, she noticed that though they could remember orders that were in the process of being served, they were unable to hold the completed ones in their memory. Back in her lab, she put together an experiment, asking 20 participants to try to solve puzzles or string beads together. Some of the participants were interrupted midway through, and when they were later quizzed, these people were twice as likely to remember what it was that they’d been doing. It’s human instinct to crave resolution.

And yet with cliffhanger-driven stories, we know that we’ll be denied closure again and again. There is something – dare I say it? – ‘meta’ about the cliffhanger. Even as it keeps us wondering about what happens next, it’s reminding us that the story is ultimately a construct, that it’s fiction, and this makes our suspension of disbelief still more thrilling. Meanwhile, left to wait, our imaginations run riot – fuelled now by social media speculation and the recap industry – encouraging us to personalise the narrative and deepen our connection with its characters. We’re desperate to know what happens but we’re not quite ready for it to be over. Is there a more delectably frustrating trio of words than “to be continued”? Sometimes, even the finale denies us, especially on television, where writers will resist closure in desperate, doomed hopes of getting another season green-lit. (What was going on at the end of Twin Peaks?)

Although technology is driving Julian Fellowes’ retreat to serial fiction, that same force has altered our cultural consumption in ways that would seem to weaken the cliffhanger’s hold over us, and lessen its importance to would-be Scheherazades. In Dickens’s day, audiences might have had to wait a full month to find out what happened next. In 2016, when we’re likely to be streaming our television or devouring fan fiction virtually as it’s written, the next instalment is often just a click away. With all these other narratives jostling for our attention, Belgravia’s cliffhangers will have to work extra hard. Then again, perhaps the abundance of such instantly gratifying alternatives will actually whet our appetite for something that’s more of a tease, which draws out the suspense and makes us wonder awhile. The story of our infatuation with cliffhangers is, as they say, to be continued.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.