From the richly-plumed red fox to the big-eared fennec fox, foxes look adorable. Because of this, people are sometimes tempted to keep them as pets.

However, those who have tried have struggled. Unlike dogs and cats, the different species of fox have not been domesticated.

Domestication only happens over a long period of time through selective breeding. Cats and dogs were domesticated by humans thousands of years ago to be pets and companions. Sheep, goats and other animals were domesticated for food.

But there may be more to it than that. People who have tried to simply tame individual foxes often speak of a stubborn wildness that is impossible to get rid of. This suggests that foxes are harder to tame than other animals.

However, one extraordinary experiment has found a way to domesticate foxes. This one study could help us understand how our ancestors domesticated other animals, and indeed what domestication is.

Biologist David Macdonald studied foxes at close quarters for years. For a time, he had foxes living at home, which he recounted in his 1987 book Running with the Fox.

Foxes are wild animals and do not fare well as domestic pets

The foxes did not last long in Macdonald's house. He found that they would tear up the living area and create chaos. Others who have tried living with foxes report the same thing.

Richard Bowler, a wildlife photographer based in Wales, looks after a few foxes in a large outside space at his home. He reports that they are nervous and shy.

The youngest fox, a vixen called Hetty, is extremely shy around people – even though she was captive-bred, and Bowler and his partner fed her through the night from when she was one week old. He describes the temperament of the foxes as "highly wired".

In the UK it is legal to keep a fox as a pet, but that does not mean it is a good idea.

The Royal Society for the Prevention of Cruelty to Animals (RSPCA) does not condone the keeping of foxes as pets. "Because foxes are wild animals and do not fare well as domestic pets, they should not be kept as such. Even the most experienced fox experts have had difficulty in keeping adult foxes successfully in captivity as they have very specific needs," it says.

They will stand and stare at passers-by on the streets

Occasionally people connected to wildlife rescue centres report that they have managed to tame foxes. However, usually these animals are recovering from toxoplasmosis, a parasitic disease that damages the brain, leaving the animals unafraid of human touch.

Meanwhile, Britain's urban foxes are often described as being bold and brazen around humans, compared with their countryside cousins. They will stand and stare at passers-by on the streets and even approach people with food.

It is possible that human behaviour in towns and cities has altered the behaviour of individual foxes: if a fox grows accustomed to being fed by hand by one person, it may be more likely to approach another. However, this does not qualify them as tame.

So pet foxes are not generally a good idea. Unless, that is, the fox is from the only tame population in the world, an extraordinary scientific experiment that started life in Soviet Russia.

In the late 1950s, a Russian geneticist called Dmitry K. Belyaev attempted to create a tame fox population.

Through the work of a breeding programme at the Institute of Cytology and Genetics at Novosibirsk, in Russia, he sought to trace the evolutionary pathway of domesticated animals. His test subjects were silver-black foxes, a melanistic version of the red fox that had been bred in farms for the colour of their fur.

Belyaev died in 1985, but the project is still ongoing. It is now overseen by Lyudmila Trut, now in her 80s, who started out as Belyaev's intern.

The Russian fox farm was the first of its kind.

"There is archaeological data that people made individual attempts to domesticate the fox, but this process was not finished," says Anastasiya Kharlamova, one of Trut's research assistants. "Possibly the reason was that the cat was domesticated at a similar time, and supplanted the fox as a possible candidate to be domesticated."

Belyaev's experiment aimed to replay the process of domestication to see how evolutionary changes came about.

There are many unanswered questions relating to domestication. One is what traits or qualities Stone Age people selected for when they set out to domesticate animals. Belyaev believed that selection for just one trait – tameability – would be enough to create a domesticated population.

It was a risky area of research.

About 10% of the foxes displayed a weak 'wild-response', meaning they were docile around humans

The study of genetics had been essentially banned in the USSR, as the country's dictator Joseph Stalin sought to discredit the genetic principles set out by Gregor Mendel. Stalin's death in 1953 gave scientists more freedom, but in the early years Belyaev nevertheless worked under the cover that he was breeding foxes to make better fur coats.

First, Belyaev and Trut travelled to various fur farms in the Soviet Union, from Siberia to Moscow and Estonia. There, they chose foxes to take to their own farm in Novosibirsk.

They selected the animals based on how they responded when their cage was opened. About 10% of the foxes displayed a weak "wild-response", meaning they were docile around humans.

"The main task at this stage of selection was eliminating defensive reactions to humans," Trut wrote in 1999. Animals that were friendlier and tolerant to human touch, even to a small degree, were picked out. Those that hid in the corner or made aggressive vocalisations were left in the farm.

Of those friendly foxes, 100 vixens and 30 males were chosen as the first generations of parents.

When the cubs were born, the researchers hand-fed them. They also attempted to touch or pet the foxes when they were two to two-and-a-half months old, for strictly measured periods at a time.

The aggressive and fear avoidance responses were eliminated from the experimental population

If the cubs continued to show aggressive or evasive responses, even after significant human contact, they were discarded from the population – meaning they were made into fur coats. In each selection, less than 10% of tame individuals were used as parents of the next generation.

"As a result of such rigorous selection, the offspring exhibiting the aggressive and fear avoidance responses were eliminated from the experimental population in just two to three generations of selection," Trut wrote in a study published in 2009.

The foxes at the fox-farm were never trained to become tame. They lived in cages and had minimal contact with humans. Belyaev's aim was to create a genetically-distinct population, so he simply selected for particular behavioural traits.

"Belyaev had one main goal at the beginning of experiment: to reproduce the process of historical domestication at the experiment, during a short time," says Trut. "This goal didn't change. But during the experiment the understanding of evolutionary process changed."

By the fourth generation, the scientists started to see dramatic changes.

The cubs were beginning to behave more like dogs. They wagged their tails and "eagerly" sought contact with humans. They whined, whimpered and licked researchers just like puppies would.                

The foxes could 'read' human cues and respond correctly to gestures or glances

The process was surprisingly quick. "By intense selective breeding, we have compressed into a few decades an ancient process that originally unfolded over thousands of years," wrote Trut in 1999.

These foxes were called the "elite of domestication", and as the generations passed the proportion of these elite cubs grew. By 2005-2006, almost all the foxes were playful, friendly and behaving like domestic dogs. The foxes could "read" human cues and respond correctly to gestures or glances. The vocalisations they made were different to wild foxes.

"The proudest moment for us was creating a unique population of genetically tame foxes, the only the one in the world," says Trut.

Brian Hare is associate professor of evolutionary anthropology at Duke University in Durham, North Carolina and author of the 2013 book The Genius of Dogs. He travelled to Russia on the Trans-Siberian railroad to visit the farm, in order to compare fox cubs with dog puppies for a study published in 2005.

"The fox farm experiment was crucial, in that it told us that domestication can happen relatively quickly in the right circumstances," he says. "The fact that in fifty generations, they were wagging their tails and barking, this is really incredible."

It was not just the foxes' personalities that were changing

The key point is that the experiment offers a hint as to the stages by which domestication takes place.

"Before, we knew that dogs and wolves were descended from the same ancestor, but we didn't know how," says Hare. "What came first? The fox experiment showed that just by selecting for friendliness, all these other changes, including an increase in social skills, happened by accident."

In fact, Belyaev and Trut soon found that it was not just the foxes' personalities that were changing. Their bodies were too.

"The main surprise was that, together with changing of behaviour, many new morphological traits in tame foxes start to appear from the first steps of selection," said Trut.

The domesticated foxes had floppier, drooping ears, which are found in other domestic animals such as dogs, cats, pigs, horses and goats. Curlier tails – also found in dogs and pigs – were also recorded.

All these changes were brought on by selecting for one trait: tameability

What's more, "in only a few generations, the friendly foxes were showing changes in coat colour," says Hare.

The process seems to be ongoing. "At the more advanced steps of selection, changes in the parameters of the skeletal system began to arise," Trut wrote. "They included shortened legs, tail, snout, upper jaw, and widened skull."

The foxes started looking more delicate and, put simply, "cute".

Their reproductive habits also changed. The domesticated foxes became sexually mature about a month earlier than non-domesticated foxes. Their mating season was longer and they could breed out of season. On average, their litters had one more cub.

All these changes were brought on by selecting for one trait: tameability. This gives us a big clue to how domestication works.

The physical traits Belyaev and Trut found, like the floppy ears, were those you would expect in a juvenile. But the domestic foxes carried them through into adulthood, suggesting the selection process had slowed down aspects of their development.

This might have something to do with chemicals in their bodies.

Selection has even affected the neurochemistry of our foxes' brains

Belyaev reasoned that selecting for tameability changed the mix of hormones and neurotransmitters the foxes' bodies made. He believed behavioural responses were "regulated by a fine balance between neurotransmitters and hormones at the level of the whole organism".            

For example, the drooping ears of the domesticated foxes might be a result of slowing down the adrenal glands. This could arrest the cells before the ear has time to stand to attention.

"Selection has even affected the neurochemistry of our foxes' brains," wrote Trut. One example she described was a drop in the "hormone-producing activity of the foxes' adrenal glands."

Domestic foxes also had higher levels of serotonin than farm-bred foxes. That is intriguing, because serotonin is "thought to be the leading mediator inhibiting animals' aggressive behaviour." Serotonin, like other neurotransmitters, is critically involved in shaping an animal's development from its earliest stages.

The project continues to this day. As of August 2016, there are 270 tame vixens and 70 tame males on the farm. However, it has run into financial problems.

"The current situation is not catastrophic, but not stable at the same time," writes Kharlamova. "The main reason of instability is of course the expense of this experiment."

In the 1990s, the institute supported itself by selling fox pelts. At the end of the 1990s, they started to sell the foxes as house pets. At present, a Florida-based company called the Lester Kalmanson Agency Inc imports foxes for those who want to keep them as pets. Each fox costs $8,900, because of the delivery costs.

With the foxes now tame, the researchers are trying to identify the genes that change under selection for tameness. "The main current goals are focused on molecular-genetics mechanisms of domestic behaviour," says Trut.

Belyaev and Trut's experiment may even tell us something about our own evolution.

We became friendlier first, and then got smarter by accident

In particular, one under-appreciated point about our species is that we have, essentially, domesticated ourselves. This is borne out in our behaviour. While we have committed our fair share of atrocities, on the whole we are far less aggressive and violent than our closest relatives, the chimpanzees.

This suggests that human evolution selected for cooperation, tolerance and gentleness – and not, necessarily, for intelligence.

"We always assume that intelligence is responsible for our success," says Hare. "That humans became smarter, which… allowed us to invent wheels and agriculture and iPhones. But what if that wasn't what happened?"

Hare suspects that, "like the foxes, and like dogs, we became friendlier first, and then got smarter by accident. This would mean that our prosocial skills, the skills that allow for cooperation and friendliness, were what made us successful."

We do not know if that is true. But it is a rather encouraging thought.

Join over five million BBC Earth fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.