Books that claim to offer the answers to life’s challenges are big business – though many serious readers look down on them. But, as Lindsay Baker explains, the genre is seeing an intellectual resurgence.

The self-help book has long been ridiculed in literary circles – and not without reason. As a genre it has become known for its relentless psychobabble and earnest, simplistic platitudes. Its detractors regard it as formulaic and full of hollow promises, the domain of the gullible and the undiscerning.

But things are changing. The literary pariah is finally coming in from the cold, it seems, and is undergoing a surprising transformation into one of the most intellectually vibrant areas in the books industry. For the many people who would never dream of reading a cheesy self-improvement book − or anything with a remotely new-age whiff about it for that matter − there is an increasingly wide range of critically-acclaimed alternatives on offer. Indeed, the rise of literary self-help is already underway, and in the UK has been tipped as the publishing success story of 2014, perhaps even to rival the celebrity-memoir boom.

Psychoanalyst Stephen Grosz’s recent book, The Examined Life: How We Lose and Find Ourselves, is a prime example. Subtly lifting the lid on the human condition through true stories from the therapist’s couch, it has been a surprise bestseller. Another recent hit has been the School of Life series – how-to guides to modern living that reshape the genre into something intelligent and thoughtful.  And ‘bibliotherapy’ – books as therapy − has become a much-vaunted phrase. Look no further than The Novel Cure: An A-Z of Literary Remedies, which explores literary-fiction reading as a panacea for life’s ills. Another recent hit is New York Times reporter Charles Duhigg’s The Power of Habit, and just out is My Age of Anxiety, by Scott Stossel, editor of The Atlantic.

But is the how-to-live genre simply returning to its roots? In ancient cultures, it was the philosophers who provided guides to living, with thinkers such as Epicurus pondering the meaning of human life, and texts such as Seneca’s On Anger and Cicero’s On Friendship and On Duties. Some have also cited the Bible as the most important example of early self-help literature. Later, French Renaissance philosopher Montaigne was among the writers asking essential questions: what is it to be human? How do we live?

Winning formula

The phrase ‘self-help’ was coined in 1859 as the title of a book by Scottish author Samuel Smiles but it was not until the 20th Century that the self-help book really took off commercially. Big hits included the 1936 ground-breaker by Dale Carnegie, How to Win Friends and Influence People, the business-orientated Seven Habits of Highly Effective People by Stephen Covey (1989), The Road Less Traveled by M Scott Peck (1978), and the empowering Feel the Fear and Do It Anyway by Susan Jeffers (1988).

These books sold in their millions, and spawned thousands of inferior imitators, all with essentially the same message − take these steps and your life will change forever. Yet as Jessica Lamb-Shapiro points out in her new book Promise Land: My Journey Through Self-Help Culture, although it is a huge, multi-billion industry, 80 percent of self-help consumers are repeat buyers, which strongly suggests that the books may not actually be working. In other words, the possibility of change in the face of life’s stubborn realities is not as easy to realise as these step-by-step guides suggest.

The early signs of a new, more philosophical mood in self-help came in the 1990s, with the emergence of Swiss-British philosopher and author Alain de Botton, arguably the godfather of intellectually credible self-help. His books How Proust Can Change Your Life (1997), The Consolations of Philosophy (2000) and Status Anxiety (2004) were thought-provoking trail-blazers, and he has continued in this vein, most recently with Art as Therapy, published in 2013.

‘Shelf help’

And now, it seems, the publishing industry has finally caught up with de Botton. Vintage is among the publishers to recognise the trend, with a series that is the ‘literary alternative’ to the traditional January glut of self-improvement tomes. The launch of their Shelf Help series has been timed to coincide with so-called ‘Blue Monday’, the third Monday of January, traditionally deemed to be the most depressing day of the year.

Shelf Help aims to offer a ‘different proposal for self-improvement’, says former Man Booker judge and curator of the series, Alex Clark. So why now? “We’re definitely living through an age of emotional openness and a willingness to share,” she says.  “A lot of people may not sign up to the idea that you can turn your life around in ten easy steps. But just because you read posh literary fiction doesn’t mean you’re immune to life’s challenges and crises, and there’s no reason why you shouldn’t look to books as a source of solace. Books are powerful spaces that can help us to expand the way we see and think about the world and ourselves.”

You will find no bullet points or step-by-step instructions here, it seems. The series is, says Clark, “very wide-ranging – and they’re about starting a conversation, they’re not telling you what to think.” For instance, British author Jeanette Winterson’s Why Be Happy When You Could Be Normal? explores the author’s difficult childhood,. As Clark puts it: “The reader may not have a literal identification with the subject matter but it’s about what it’s like to come through a severe, disrupting trauma.”

Like several others in the series, Winterson’s book blends memoir with a self-help sensibility. In a Vintage podcast, Winterson speaks about how her love of reading, and how we can be “chemically altered by the force of language.” She says reading can be better than therapy: “A life with books is a continual drip of therapy, because you’re always seeing things differently,” but reading books, she says, allows “multiple readings of the self and that allows great freedom – emotional freedom and psychic freedom.” There is, says the author, something “talismanic” about certain books, and cites among others Georges Perec’s masterpiece Life A Users Manual. They “will go on changing” as the reader does. 

Teach Us to Sit Still: A Sceptic’s Search for Health and Healing is another in the series that draws on memoir. In the book, ward-winning novelist Tim Parks confronts his own mortality when he becomes seriously ill. It is darkly comic and honest – and helpful too, though Parks describes his book as not self-help but an ‘exploration’. He tries a breathing exercise, he tells BBC Culture, “which eventually led me, much against the grain of my rational convictions, towards meditation. In short, sitting still.”

Parks was surprised by the reaction it got. “It was rather a shock when I started getting letter after letter thanking me for having written something that helped,” he admits. “Because I had really offered no solutions or prescriptions, just described a transformation. In the end, literature is perhaps helpful when it is honest, and when it intersects with our life in a felicitous way. There is certainly no formula for writing such books, and I have no plans to write another. That said, perhaps people are getting a little more willing to confess that they need help of some kind. There are few of us who don’t.”

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.