It is like a scene from a horror film: spider webs several metres wide that are home to thousands of spiders. Why did these spiders turn social?

Eight legs, eight eyes and a pair of venomous fangs – the average spider is so well-equipped that it's easy to see why arachnophobes are terrified. But there is one saving grace: at least spiders are solitary, so they are usually encountered in small numbers.

That's certainly what I thought.

But then I found myself trudging along a muddy path in the Peruvian Amazon jungle, face to face with a spider colony several thousand strong. Their funnel-shaped web arched from tree to tree, a structure containing too many of the creepy crawlies to count.

These were spiders but not as I knew them: they appeared to function as a society, just like ants or bees.

Most spiders are indeed lone wolves, but a scant handful have evolved a level of sociality to rival ant colonies, bee hives, or even primate societies.

Communal spiders work together to build, maintain, and clean their webs

Anelosimus eximius, the species I encountered in the rainforest, is not the only kind of social spider in the world, but it does construct the biggest webs. Some can reach more than 25ft (7.6m) feet long and 5ft (1.5m) wide. A web that size could contain as many as 50,000 individual spiders. That is a lot of legs, eyes and fangs.

A. eximius was first discovered more than a century ago by a French arachnologist named Eugène Simon. More social spiders have been discovered since. One was found as recently as 2006, in Ecuador, by entomologist Leticia Avilés.

Sociality shows up in at least seven spider families. In all we know of around 25 social species among the 45,000 described spider species on the planet. Sociality evolved at least a dozen separate times among them.

While the details vary from species to species among the social spiders, many of their features are similar.

For example, an A. eximius colony contains adult males and females as well as youngsters, but the majority of spiders on the web are females. In the early 1980s, researchers found that males comprise only between 5% and 22% of any colony's populace.

Communal spiders work together to build, maintain, and clean their webs. They cooperate in capturing prey, and dine together when they snare a large feast.

The females feed their offspring by vomiting up food for them, just like mother birds. They even regurgitate food for juveniles other than their own. In other words, they work together to care for the youngest in the colony. It is a sort of spider day-care.

Nearby colonies tend to be related to each other, because new colonies form in one of two ways.

Sometimes, a large web is broken in half, perhaps by heavy rainfall or falling branches, or even a falling monkey. In its place, two smaller colonies are left behind.

My students know more about social spiders than your average arachnologist

Other times, individual females will strike out on their own, setting up a new web. Eventually other dispersing females will join the group; perhaps ones who tried to set up their own webs but failed. Survival is more likely in groups, after all.

These two processes together help to explain why A. eximius colonies tend to be found in clusters, sometimes as many as 40 related colonies within just a few kilometres.

A. eximius is perhaps the best known of the social spiders, but that is not saying much.

Gregarious arachnids are not much studied – perhaps because so few of them exist in the first place. "My students know more about social spiders than your average arachnologist," says Linda Rayor, an entomologist at Cornell University in Ithaca, New York, who studies Australia's social huntsman spiders.

Why was the bee subject to such an awful fate?

"Most animals are solitary," she tells me, "whether you're looking at mammals, or birds, or whatever. Social behaviour in otherwise solitary animals is cool."

Cool indeed, if somewhat creepy.

As I watched the social spiders go about their day deep in the jungle, a bee found itself stuck in the giant web. Suddenly, dozens of the eight-legged predators descended en masse onto the struggling black-and-yellow striped insect. It was impossible to see through the writhing mass of tiny spider legs, but one thing was certain: that bee was done for.

If death by one spider seems bad enough, it must be nothing compared to an attack by a whole swarm of them.

Why was the bee subject to such an awful fate? Or, to put it another way, how and why did a few spiders become social when the overwhelming majority of their fellow species are solitary?

The first spider, the proto-spider, was probably solitary. So the spiders that gave rise to today's truly social species must have been too. But then they did something remarkable: they turned sub-social: they learned to tolerate each other, at least for a time – even if they did not exactly enjoy hanging out in groups.

As they mature, they become intolerant of each other

That early tolerance could have come about as a result of parental care. In some spiders alive today, that is just defending an egg sac, but in others – the ones that are poised to evolve sociality – it looks more familiar to our mammal brains.

In those species, females provide food and protection for their developing offspring, and occasionally even to other juveniles. In some species, mothers even make the ultimate sacrifice, allowing themselves in their last moments to become a meal for their brood. Sub-social spiders can still be cannibals, after all.

"They're the spiders that already have cooperation as juveniles," says arachnologist Jonathan Pruitt of the University of California, Santa Barbara. "But as they mature, they become intolerant of each other."

In other words, juvenile tolerance alone is not enough to turn spiders fully social. The environment in which the spiders live also has to be right. Researchers have discovered three ecological elements that often lead to cooperative living among arachnids.

Social spiders tend to feed on bigger prey, for one thing.

Solitary spiders living in places where it is difficult to subdue large or more profitable prey alone may eventually figure out that it is in their interest to work together. It is a smaller step from tolerance to cooperation than from aggression to cooperation.

Social spiders just about always build webs

Another common feature is heavy rain. Rain does not have to be frequent, but when it is really intense, it has the potential to seriously damage spider webs. When a tempest takes out the web and threatens a spider's survival, the ones with just a glimmer of social behaviour might fare just a little better than their isolationist peers.

As a result, some have hypothesised that rough weather favours cooperation, since in those conditions it is the spiders that share the task of maintaining and repairing their webs that fare best. "Many hands make light work of web repair," says Pruitt.

Then there is the web itself. There are only a couple of instances in which spiders that do not build webs have evolved to be sub-social. These web-less social spiders, living in places like the Australian deserts or in African scrub habitat, have to cope with unusual circumstances – like enormous prey or particularly aggressive raiding ants.

But these cases aside, social spiders just about always build webs. And most of these webs have complex, three-dimensional structures, like the cone-shaped A. eximius webs I encountered in the jungle.

Pruitt thinks that could be a simple matter of mechanics. It is easier for two spiders just on the cusp of evolving sociality to join their webs together if they exist in three dimensions than if they weave flatter, two-dimensional ones, like Charlotte's eponymous web.

Some combination of tolerance, prey size, rough weather, and web geometry combine to create the perfect storm for social behaviour to emerge in spiders.

Anyone can do any job, and all are capable of reproduction

Add in the fact that it is easier to defend yourself against predators in a group, and that communal webs offer spiders the relative safety of staying in one place rather than risking travelling away from the web onto which you are born to live somewhere else, and for some species, cooperation is a good deal.

These spiders, like A. eximius with its massive funnel pits of doom, may superficially seem to behave like social insects. But in truth they could hardly be more different.

Bees and ants sort into separate castes. Some are workers, some are soldiers (or drones), and then there is a reproductive class. Those not included in the reproductive class are sterile; they could not reproduce even if they wanted to.

Social spiders, meanwhile, are more egalitarian. Anyone can do any job, and all are capable of reproduction.

It is not that spider societies do not rely on the division of labour. It is just that those roles are not strictly governed as they are for the social insects.

Spider vocations are intimately tied to their efficiency at those vocations

And while some female social spiders do not wind up reproducing, it is not because they are physically or genetically incapable. Instead, it is usually just because they have not lived successful enough lives. It takes a lot of nutrition to make a few hundred spider babies, and a spider that does not have a rich enough diet is not really up to the task.

Instead, roles in spider colonies are usually sorted on the basis of age and sex. Researchers are increasingly coming to realise that social spiders also sort themselves according to their individual personalities. "They're incredibly well organised," says Pruitt.

By paying close attention to individual spiders, he and others have discovered that certain spiders are more likely to spend their days attacking predators, while others are more likely to repair the webs, help keep parasites away, clean the web, rear the young, and so on.

"Spider vocations are intimately tied to their efficiency at those vocations," he says.

In spite of a lack of genetic diversity, they exhibit this exquisite diversity in terms of their personalities

Though it is not yet clear whether those skills develop through experience or because of innate aptitude, some spiders are clearly in the construction business, some are hunters, others are on janitorial duty, and still others offer childcare services.

In species like A. eximius, spiders are essentially clones. Brothers and sisters mate with each other, generation after generation, so that colonies become highly inbred.

Since adolescent spiders do not go off in search of new colonies, there is no influx of new genetic material. For that reason, some suspect that spider personalities develop as a result of early life experiences, not genetics. "In spite of a lack of genetic diversity, they exhibit this exquisite diversity in terms of their personalities," Pruitt says.

In 2013, he and his colleagues looked at a social spider called Stegodyphus sarasinorum, native to India, Sri Lanka, and Nepal. They found that individual personality traits predict which job each spider performs.

S. sarasinorum is found in dry scrubland. But like the spiders I found in the rainforests of Peru, these arachnids work cooperatively to build and maintain their webs and to capture food.

Social spiders have distinct personalities

Using tiny dabs of paint applied to the spiders, Pruitt and his team were able to track 40 from each of two different wild colonies. Individuals that were bolder were more likely to be involved in attacking prey – but other characteristics, like body size or a spider's level of aggression, did not factor into their vocation.

"Responding to stimuli from a prey caught in the web elicits different responses according to a set of individual characteristics," they concluded.

In other words, social spiders have distinct personalities, which in turn help to define their roles in the community.

In a way, that is not so different from human societies. Bold, risk-taking people might be more likely to wind up fighting fires or enrolling in a police academy, while the more calculating, deliberative types wind up as lawyers or architects.

Despite lacking a backbone, invertebrates like insects and arachnids lend themselves well to studying complex social behaviour. "When it comes to a behaviour that any sort of animal can do, insects [and arachnids] can do it, if not do it better," says entomologist Phil Torres.

After all, they have been evolving on our planet for a lot longer than any bird or mammal has.