The writers of Eloise, Goodnight Moon and more pushed boundaries of behaviour. Dark episodes in their pasts may have proved fruitful, writes Hephzibah Anderson.

As new parents and anyone who’s ever gone rummaging through their old childhood libraries quickly realises, much of the best literature for kids is bonkers. It all makes divine sense when you’re four or five or 10, but return as an adult to Alice’s Adventures in Wonderland or The Cat in the Hat, and what’s striking is that so much sublime whimsy and exuberant creativity could have been conjured up by our fellow grownups.

Eloise had been an imaginary friend of author Kay Thompson since childhood

Then again, there’s a theory that children’s authors – the best of them, at any rate – never really grow up. Lewis Carroll famously – notoriously from today’s perspective – preferred playing games with children to adult conversation. Kenneth Grahame amassed a vast collection of toys – in his 20s. And Dodie Smith, a fascinating writer best known today for her children’s novel The Hundred and One Dalmatians, used to say she never felt quite grown up. (At under five feet tall and with a high-pitched, perpetually girlish voice, she perhaps had more excuse than most.)

- Why tyrants love to write poetry

- The Jane Austen novel you don’t know

- The writers who defied Soviet censors

There’s no arguing that the behaviour of certain of these writers makes their whippersnapper protagonists look good as gold. Take night club entertainer and Eloise author, Kay Thompson. She was, by all accounts, a difficult woman to work with. Hilary Knight, Eloise’s illustrator, finally threw in the towel after Thompson seized his hand in an attempt to direct his pencil. A fondness for flouncing off saw her quit everything from a role in The Pink Panther (she hated her costume) to a gig concocting a fragrance for Tiffany’s (she had “the mind of a grasshopper”, one of the company’s executives despaired).

The way publicists spun it, Eloise sprang spontaneously to life when, late for a rehearsal of her act with the Williams Brothers, the ordinarily punctual Thompson apologised in a child’s voice. “Who are you, little girl?” someone asked, to which Thompson replied, “I am Eloise. I am 6.” It became a rehearsal game, then a one-woman show at the Plaza in Manhattan, and finally, in 1955, a book. But according to biographer Sam Irvin, Eloise had been with Thompson since childhood, an imaginary friend whose voice she’d channelled all her life. When a made-for-TV film first started shooting, Thompson would hide under a table on set and voice Eloise’s part herself, making the child actor cover her mouth with a book.

Were she not an author, psychiatrists might have had something to say about all this. That she later became addicted to amphetamines and was shockingly thin suggests her ornery flightiness was symptomatic of something altogether darker. But what was obnoxious and ultimately tedious in a grown woman was sassy in a small girl. The destructiveness, the wilfulness, the caprice – it’s all part of why Eloise is so beloved. No prizes for guessing Thompson’s reply when asked about the inspiration for her heroine: “Eloise is me! All me.”

Kenneth Grahame’s courtship was conducted via letters written in babyspeak

Children’s literature is chock-a-block with contenders for authors most like their creations. Theodor Geisel, aka Dr Seuss, dealt with writer's block by ducking into a secret cupboard, hidden behind a bookcase, in which he kept a vast array of hats. Edward Gorey might easily have slipped from one of the frames of his own illustrations, fond as he was of floor-length furs and long scarves. These he paired with jeans and trainers, reminding us that while surreally macabre (he didn’t like the word), his work was also pungently humorous.

Though children loved his books, Gorey didn’t return their enthusiasm. Another thing that seemed not to register in the otherwise broad beam of his curiosity (he loved everything from ballet to TV soaps) was romantic relationships. Indeed, he once claimed not to know whether he was gay or straight. He may not have been reclusive but he was certainly solitary in his later years, and when he died in 2000, he left most of his estate to a charity for animals, including bats and insects.

Children at heart

Gorey isn’t alone in feeling a greater kinship with animals than with most of his fellow human beings. EB White, the exquisite essayist who also wrote Stuart Little and Charlotte’s Web, was cripplingly shy his whole life. It wasn’t only in fiction that he hid behind animal characters, either. In 1930, when he penned a few lines to congratulate his wife, New Yorker editor Katharine Angell, on her pregnancy with their first child, he did so in the voice of Daisy, their Scotty dog.

Grahame, creator of that other timeless anthropomorphic pastoral, The Wind in the Willows, was also shy. Like Gorey, Grahame showed little to no interest in sex. He was a 38-year-old virgin when he found himself pursued by Elspeth Thomson, the scatty yet controlling 35-year-old daughter of the inventor of the pneumatic tyre. Their courtship evolved over letters written almost exclusively in baby speak.

I don’t particularly like children – Margaret Wise Brown, author of Goodnight Moon

Shyness seems not to have afflicted Margaret Wise Brown. Her name may not be well known but the titles of her two most famous books – The Runaway Bunny and Goodnight Moon – most certainly are. Their American author was born into a tony, establishment East Coast family, and her early education included a stint at a Swiss boarding school. In her 20s, working as a teacher, she started writing children’s books.

Don’t let these biographical details fool you. Brown blew her first literary earnings on a cartful of flowers, wrote with quill pens, painted glow-in-the-dark stars on the ceiling of her apartment, installed orange trees and live birds in a Paris hotel room and founded a society whose members could declare any day of the year Christmas. She also had tumultuous affairs with men and women, and was an avid beagler, a sport that consists of chasing hares on foot. As to her readers, Brown famously confessed to a reporter, “I don't particularly like children.”

Frances Hodgson Burnett longed for daughters and so dressed her sons in frills

Brown died aged just 42 but very much in the manner in which she lived: capriciously, and in defiantly high spirits. After an emergency appendectomy in France, she gave a cancan kick to show a nurse how very fine she was feeling. It sent a blood clot travelling from her leg to her brain.

Eccentric or mad?

Children’s literature is bursting with examples of eccentric authors. Frances Hodgson Burnett longed for daughters and so dressed her sons in frills. Russell Hoban, whose early career encompassed the Frances books and that unsung masterpiece, The Mouse and His Child, worked in an extraordinary room dense with waist-high piles of books and curiosities like windup toys, that he dubbed his ‘exobrain’. You’d think it was home to a kleptomaniac shut-in if you didn’t know better. In her new biography of Edward Lear, Jenny Uglow observes that the heroically talented artist, versifier and traveller was, from boyhood, “three parts crazy”. But does that mean certifiably so? At what point does eccentricity become something more problematic?

Now that we have the vocabulary and sharper diagnostic tools, some authors have spoken out about their mental health battles. JK Rowling has opened up about her depression, and wrote the Dementors into being to illustrate how it feels to be a sufferer. Robert Munsch, author of the perennial bestseller Love You Forever, has been open about his struggles with bipolar disorder and addiction.

Ever since the ancient Greeks decided that creative inspiration must be a kind of divine madness – furor poeticus – the connection between mental illness and creativity has been much probed. It remains contentious, however. While the mad artist may be a centuries-old stereotype, you don’t need to look very hard these days to find mental illness in every walk of life, from sport to tech. Perhaps the most that can be said with any conclusiveness is that a certain kind of sensibility lends itself to truly great children’s writing.

In Lear’s case, he always felt “jarry and out of tune” with the world into which he was born. Diagnosed as an epileptic in early childhood (Lewis Carroll was a fellow sufferer), he also had dreadful eyesight. Having grown up solitary with remote parents, he was further ostracised in adulthood by being a gay man in Victorian England.

The wondrous Maurice Sendak, another ailing child, turned to books when he was confined to his bed. The loss of family members in the Holocaust foisted upon him an early sense of mortality. “Childhood is a tricky business,” he once told an interviewer. “Usually, something goes wrong.”

Grahame lost his mother when he was five, after which his father vanished into the alcoholism that would eventually lead to his death, penniless, in a Le Havre boarding house. Brown was the product of a depressive mother and a barely-there father. Gorey had enough space to teach himself to read aged three, and had already devoured Dracula by the time he turned five.

Contrary to the notion that children’s authors don’t grow up, so many of the best of them seem never to have had the chance to be children in the first place. Childhood was a lonely and not entirely safe place for them. Maybe they made up for it in later life, but it’s the darkness that they glimpsed in their earliest years that makes their work so vital, providing the shadowy contrast that enables their invented happily-ever-afters to shimmer.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.