In 1888 the local Maori chieftain of the Kawhia Tribe, a man named Tane Tinorau, decided to lead a war party and attack another tribe in the Waikoto region of New Zealand's North Island.

After successfully conquering the other clan, a hunter was sent to find food. He stumbled across the entrance to a network of caves. Chief Tinorau was the first to explore the caves in detail, floating on a raft made of flax flower stalks with just a burning torch to guide the way.

However, his torch was not the only light in the cave. Above his head, the cave roof twinkled like the night sky. There were thousands of tiny insects attached to the rock, each flickering in the darkness. The beautiful effect has earned the caves their name: the Waitomo Glowworm Caves.

They are just one environment in which you can find creatures of light – one of the many species that can emit a bright glow to help catch prey, avoid predators, even to find a mate. These glowing creatures are one of the natural world's most familiar sights, yet in some cases we are only just starting to understand them.

The Waitomo Cave glowworms are not worms at all. They are the larvae of fungus gnats: small, delicate flies that feed on fungi. As soon as the larvae hatch they start to spin sticky web strings from the cave ceiling. The silky threads have inspired the insects' Latin name: Arachnocampa luminosa, which directly translates as "glowing spider-worm".

Bioluminescence has evolved separately at least 50 times

The glowworms drop globules of sticky mucus along the web fibres, which makes them look like glass bead necklaces hanging like fishing lines from the ceiling. They then hang from the silky threads and sit and wait. An orb-like gland in their tails produces the ghostly blue light: light literally shines out of their backsides.

The other insects that live in the pitch black of the cave cannot see the glowworm's sticky trap – but they can see the blue light. Attracted like moths to a flame, they fly upwards – only to be ensnared. All that is then left for the hungry gnat to do is reel in the line and devour its prey alive.

The glowworms are adept at living in the dark. As well as the Waitomo caves, they can also be found hiding out in dark, damp forest canopies. They use a chemical reaction in their bodies to create light: a process known as bioluminescence.

They are far from alone in generating light this way. Bioluminescence has evolved separately at least 50 times. The skill is scattered throughout the tree of life, appearing in insects, fish, jellyfish, bacteria and even fungi.

Although these organisms can be strikingly different from one another – it is billions of years since some of them last shared a common ancestor – the chemical reaction responsible for producing light is remarkably similar in bioluminescent organisms.

In each case the animal, fungi or bacteria takes advantage of the reactive nature of oxygen, which wants to combine with other elements in a process known as oxidation. Oxygen binds to a chemical called a luciferin and undergoes a chemical reaction, helped along by an enzyme called luciferase.

An orb-like gland in their tails produces the ghostly blue light

The high-energy compound that is formed then breaks down, releasing enough energy to excite electrons in atoms so that they jump further away from the nucleus. When they relax back to where they were, a photon is expelled and energy in the form of visible light is released.

Although all bioluminescent creatures use much the same reaction, the exact nature and structure of the luciferin and luciferase vary dramatically across different species.

In the case of the Waitomo Cave glowworms, researchers have only just begun studying how the larvae produce light. The first such study was published in 2015. The scientists discovered a remarkable similarity with perhaps the most famous of all bioluminescent animals: the firefly.

Researchers had no reason to suspect that Waitomo glowworm bioluminescence would be anything like the firefly version. For one thing, when you mix firefly luciferin with Waitomo glowworm luciferase no light is produced.

There is a huge evolutionary distance between glowworms and fireflies

The glowworm also uses an unusual part of its body to make light – organs called malphighian tubules that form part of the insect's excretory system. It is a bit like humans making light from their kidneys. No other bioluminescent insects are known to do this.

To investigate further, scientists at the University of Otago in New Zealand isolated genes from the glowworms' malphigian tubules and looked to see which ones were unusually active when compared to gene activity elsewhere in the insects' bodies.

Remarkably, three of the most active genes coded for proteins that were similar to firefly luciferase. This is strange because, although the two species are both insects, there is a huge evolutionary distance between glowworms and fireflies. One is a fly and the other a beetle. To find a common ancestor for the two organisms you would have to go back 330 million years.

It is almost impossible that a common ancestor passed on the bioluminescence genes to both bugs, as the majority of other insects that evolved from the same ancestor do not glow in the dark. Instead, the two luciferases may have evolved independently from a common enzyme inherited from an ancestor long ago.

"The two insects are evolutionarily far enough away that we expected a unique chemistry from the glowworm," says Kurt Krause, one of the scientists who studied the glowworm. "It looks like the luciferin is completely different from that of the firefly, but the enzyme luciferase has many similar features."

Nature has come up with different ways of solving the problem of making light

It is an unusual discovery, given that researchers know that other bioluminescent organisms use all sorts of different chemicals to produce a glow. The railroad worm, which is also not a worm but the larva of a beetle, uses two different luciferases to produce two separate colours - red and green like a traffic light. Single-celled plankton called dinoflagellates make their own luciferin, which is chemically very similar to the green chemical chlorophyll found in plants.

Some bioluminescent animals steal their luciferin from other creatures, effectively getting others to make their light for them.

The Hawaiian bobtail squid, for example, exploits the luminous nature of Vibrio fischeri bacteria. The bacteria do not produce light when they are by themselves floating in the ocean, but when incorporated into the squid's light organ they begin to shine a faint blue light. The relationship is mutually beneficial as in exchange for producing light the bacteria get a steady stream of nutrients.

"If you look at the chemistry of luciferins, although molecular oxygen always triggers the glowing reaction, the actual chemical luciferins used in the reaction are very different," says Krause. "Nature has come up with different ways of solving the problem of making light."

How exactly did this light-generating ability arise in the first place? One theory is that luciferins first evolved as antioxidants.

On the early Earth, before our planet had a proper atmosphere, lifeforms were bombarded with UV radiation from the Sun. This radiation would have broken apart water and released a harmful reactive form of oxygen which damages cells. Life responded by producing antioxidants – chemicals that are able to mop up this dangerous oxygen.

Between 80 and 90% of the species living 700m or more below sea level can produce their own light

Gradually, Earth's atmosphere changed. Oxygen levels rose, which meant more was available to dissolve in the oceans and organisms could begin exploring deeper levels of the ocean and still get the oxygen they needed to survive. But little of the harmful UV light filters down through the water column, so the antioxidants produced by these deep-dwelling organisms were out of a job.

Evolution responded as it always does: it improvised, finding a new role for the antioxidants.

Because there was little light at depth, species needed a way to find food and mates. Exploiting the fact that the antioxidants had already evolved to interact with oxygen, evolution gradually selected for versions that generate light as part of that interaction. Bioluminescence evolved – and it proved such a useful innovation that, as the saying goes, it spread like wildfire.

In fact, it is thought that between 80 and 90% of the species living 700m (2,297 feet) or more below sea level can produce their own light.

Even in the darkest places many animals have eyes which are straining to catch the faintest glimmer, and it seems that some animals have evolved bioluminescence to try and attract their attention.

In some cases, having a body that lights up can actually make you less likely to be seen

The frightening-looking anglerfish, for instance, has a spear-like appendage which it lights up like a lighthouse. Fish swim in for a closer look, and get ripped apart by the anglerfish's monstrous jaws.

Fireflies, on the other hand, use light to attract a mate rather than a meal. In some species the males emit a specific pattern that tells females that they're interested in mating.

But creatures of light do not always have attraction on their minds. Some generate light to distract attention – or even to avoid unwanted scrutiny entirely. In some cases, having a body that lights up can actually make you less likely to be seen.

The sparkling lights of the aforementioned Hawaiian bobtail squid allow it to camouflage itself, as the light mimics that of the dappled light filtering down the water column from the surface. Predators, including sharks, often hunt from below by looking for silhouettes in the water where an animal is blocking the sunlight filtering through. Using bioluminescence as a form of counter-illumination can foil these attacks, by making the animal as bright as the surrounding water.

Hatchetfish use this approach too. They have light-emitting organs that point downward from their bellies. They are able to fine-tune the amount of light they make so that it matches that coming from the sky, making them almost invisible to predators below.

Predators, including sharks, often hunt from below by looking for silhouettes in the water

As well as camouflage, animals can use flashing lights to startle and confuse their enemies, buying them precious time to escape. Vampire squid live deep down in the depths of the ocean. If attacked they squirt a glow-in-the-dark mucus at their assailant.

Brittle stars, a close relative of starfish, also have a unique and sly way of tricking predators. They can detach one of their luminescent arms so that the predator chases after the arm instead of the main body of the brittle star. Later they can re-grow their appendage.

Some species of sea cucumber – another starfish relative – are even cheekier and can pass on their glowing body parts to passing fish. The predator then chases after the fish while the sea cucumber makes its escape.

Other creatures can outsmart their predators. They use bioluminescence to attract animals that will attack and eat those hunters.

Dinoflagellate plankton fall into this category. They produce dazzling displays of blue light at night. The light alerts animals higher up the food chain, which are attracted by the prospect of catching and eating one of the animals that grazes on the plankton.

Brittle stars can detach one of their luminescent arms so that the predator chases after the arm instead of the main body

So how do animals know that the light they are making will attract the right kind of attention? If light can both frighten or attract, entice or distract, you want to make sure you are having the right effect on prey, mates or predators.

One theory is that organisms are generally attracted by a glow, but frightened or diverted by brief and bright flashes. For example, when the female Waitomo glowworm fly emerges from her cocoon she emits a steady glow. She is quickly surrounded by males wanting to mate with her.

The steady glow is also useful when it comes to luring insects for a meal. It is this effect that makes the roof of their caves twinkle like the night sky – captivating human attention ever since Chief Tinorau first witnessed the spectacle 130 years ago.

Join over five million BBC Earth fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.