Our relationship with fire has always been a complicated, love-hate affair.

Fire brings warmth, safety, clean drinking water and cooked food, and may have fast-tracked the expansion of humans out of Africa. But fire is also an unruly beast, bringing with it the threat of devastation and death.

Wildfires can be hugely destructive, and threaten the safety and property of people living in fire-prone regions. It seems obvious to most people that wildfires need to be prevented and extinguished, at all costs.

And they are: today, 98% of all fires in the US are successfully extinguished. But the more money we invest into stopping wildfires, the worse they seem to get. Year after year, the number of forest fires is increasing globally, along with their size and intensity.

In fact, the six worst fire seasons in the last 50 years have all occurred since 2000. This raises a tricky question: are our successful efforts to control wildfires partly responsible for this broader trend in wildfire size and intensity? And if so, how?

Any environment with relatively wet winters and long summer droughts is likely to be fire-prone. All that winter water fuels plant growth in the spring, leaving lots of plant vegetation that has the potential to dry out and become highly flammable during drought.

The most flammable ecosystems tend to be grasslands and shrublands

This means that fire has been a problem probably ever since plants first move onto the land hundreds of millions of years ago. Indeed, the impact of fire is evident in plant genomes dating back as early as 125 million years ago.

But, perhaps surprisingly, evolution has adapted the plants and animals in particularly fire-prone ecosystems to cope with the threat – and to rebound vibrantly afterwards.

The most flammable ecosystems tend to be grasslands and shrublands, because the plant stems are thinner and quicker to catch light.

In grassland habitats, such as the savannahs of southern Africa, fires are frequent. But they move through the grasses quickly, hardly heating the soil below. This allows plant root systems to survive the fire, and so the grasses re-sprout rapidly in burned areas.

The more money we invest into stopping wildfires, the worse they seem to get

Animals, from insects to birds and mammals, are usually able to survive wildfires too, by running, flying or burrowing out of danger.

What's more, as soon as the vegetation returns, so does the wildlife. The soft, young grass stems attract grazing herbivores from surrounding areas and allow the grassland ecosystem to rapidly regenerate.

In fact, without fires these grasslands would slowly turn into forest through a process known as succession. Trees can ultimately outcompete grasses when conditions are stable, but frequent fires create an environment in which grasses have the upper hand: slow-growing tree saplings are destroyed before they can really establish themselves.

This might suggest that fires pose a threat to the very existence of forest ecosystems. But this is not the case: some forests are fire-adapted, too.

Pine trees in the Ponderosa forests found across the western United States and Canada have thick, heat-resistant bark to protect the living tissues inside from rising temperatures. They also naturally drop their lower branches to prevent fire catching into the canopy.

Every five to 25 years, natural fires pass through these forests. It is vital for the forests that they do, because the flames burn leaf litter and understory plants, preventing a build-up of forest-floor vegetation.

When fires do now occur in these forests, they are high-intensity and their flames reach into the canopy

Because that vegetation is burned while it is still in relatively small quantities, the forest fires are themselves smaller. Their relatively cool, short flames leave the crowns of the larger trees intact and the forest survives.

This is an example of a fire regime: the frequency and type of fire that an environment commonly experiences.

But human intervention in the last century has disrupted the natural fire regime of the Ponderosa pine forests. By grazing livestock, logging the trees for timber and systematically fighting fires before they can run their course, humans have changed the structure of the ecosystem and encouraged a build-up of forest-floor vegetation.

As a consequence, when fires do now occur in these forests, they are high-intensity and their flames reach into the canopy.

"Livestock grazing, commercial logging, and systematic fire suppression has converted some frequent, low-severity fire regimes, such as the ponderosa pines in the interior west, into infrequent, high-severity fire regimes," explains Timothy Ingalsbee, co-director of the Association for Fire Ecology in the US.

Some forest ecosystems are actually adapted to these extreme fires

High-intensity fires in these crowded forests are devastating, killing at least 70% of trees in their path. "The plants and animals that inhabit this ecosystem are generally not well-adapted to this change in fire regime," says Ingalsbee. As a result they are unable to rebound after intense crown-fires.

The exposed, bare soil that remains is extremely prone to erosion, washing away nutrients and clogging nearby streams and rivers.

While high-intensity crown fires are devastating in many woodlands, some forest ecosystems are actually adapted to these extreme fires.

Lodgepole pine forests and the redwood forests of the Californian coast are two such examples. Every 80 to 200 years, wildfires with extremely hot, high flames whip through these forests, and very few trees survive.

Wood ash is a great fertiliser, giving seeds a rich environment in which to grow

But the trees in these forests produce resin-sealed pinecones, which can survive the extreme temperatures – and which will open after the fire has died down.

"Plants have evolved different strategies to cope with fire," says Çağatay Tavşanoğlu, a fire ecologist at Hacettepe University in Turkey. "One strategy is to survive fire as a seed, not as an individual plant."

Indeed, a surprisingly large number of plants use serotiny – germination triggered by fire – to survive in fire-prone environments. In lodgepole pines, "seeds are protected in closed cones in the canopy during a fire, and then dispersed to the burned soil," he explains. Wood ash is a great fertiliser, giving seeds a rich environment in which to grow, and allowing pine trees to resprout rapidly after a fire and regenerate the forest in a matter of decades.

The Australian bush is also adapted to high-intensity fires. Here, Eucalyptus trees thrive because of their pyrophylic traits.

Eucalyptus trees actually produce flammable oils in their leaves, encouraging fire when they drop to the floor. As soon as a fire catches, the trees' bark peels away in long streamers, providing more fuel for the flames.

Some plants are able to survive using underground buds that remain protected under the soil surface

The fires burn hotter and reach the canopy, killing almost all the trees, but leaving the environment open for new saplings. Seeds of the Eucalyptus tree are only released from their seed capsules by burning, and they quickly germinate and thrive in the fire-fertilised soils.

Because Eucalyptus trees actively encourage high-intensity fires, few other species can tolerate living near them, giving the trees very little competition in an extremely nutrient-rich environment.

The Mediterranean maquis shrubland is adapted to a similar fire regime, experiencing intense fires about once every 20 years. Here, plants such as the strawberry tree use protective underground buds to keep their seeds safe from scorching soils.

"Some plants are able to survive using underground buds that remain protected under the soil surface, and then re-sprout quickly after a fire," says Tavşanoğlu.

These examples show that intense fire, while obviously destructive, is vital for maintaining biodiversity in some regions.

"Certain ecosystems such as lodgepole pine are still functioning within their natural fire regimes of infrequent, high-severity fire," explains Ingalsbee. "The problem is that many people think high-severity fires are scary and 'unnatural', so land managers try to prevent them."

Intense fire, while obviously destructive, is vital for maintaining biodiversity in some regions

We might think that by controlling intense wildfires in these regions we are protecting ecosystems. But paradoxically we are actually increasing the vulnerability of species like the Sequoia redwood trees of the western US. By removing the rare but intense fires these species have adapted to exploit, human activity is allowing other species to muscle in and outcompete these iconic giants.

What's more, as species adapted to a low-severity fire regime move into these formerly high-severity fire regime forests, it leaves the ecosystem more vulnerable if and when firefighters fail to prevent one of those high-severity fires sweeping through the forest.

Unfortunately, firefighters might increasingly find themselves on the losing side.

Part of the problem here is climate change. Global warming will increase air temperatures and make droughts more frequent, creating more opportunities for wildfires to ignite.

"An increase in temperature and decrease in rainfall in the Mediterranean Basin may result in larger fires in the next few decades," says Tavşanoğlu.

In the US, federal wildfire suppression and protection costs have trebled since the 1990s

A recent study found that fire seasons globally lasted 18% longer in 2013 than in 1979, exposing twice as much fire-prone land to the conditions needed for fire.

On top of this, fire suppression efforts over the last century mean that forests have become unnaturally overgrown, and that they contain plenty of kindling just waiting for an excuse to ignite. An increase in the frequency of forest fires has been reported in California and Nevada, where the area burned each year now exceeds that before fire suppression efforts began.

This leaves authorities with a difficult problem. After decades of successful wildfire suppression, we can now expect fires to be far worse when they finally happen.

This is already becoming evident: as wildfires increase in frequency, the cost of keeping them at bay is soaring. In the US, federal wildfire suppression and protection costs have trebled since the 1990s, and now account for nearly half of the Forest Service's annual budget.

Something has to change.

If we are to reduce the risk of devastating high-severity fires in ecosystems that are not adapted to that sort of regime, we must change our management strategies.

The key to restoring fire-prone habitats seems to be introducing variety

There are three main options on the table. The first is the least controversial: use machines to clear out understory vegetation, dead trees and non-native plants to prevent a build-up of potential kindling.

The other two options are more contentious: either use regular controlled burning – or, even more controversially, manage natural fires and allow them to run their course.

Mechanical thinning may be the least dangerous and most politically acceptable strategy, but it can also be very costly. Although the timber removed from some forests can be used commercially, offsetting the costs of removal, in other forests it is just not valuable enough. Mechanical thinning also requires huge time investments. Many ecologists feel it is simply not worth it as a sole strategy.

"The way forward is of course to utilise all of the tools available," says Ingalsbee. In certain situations mechanical thinning might make sense. But more generally, for really effective forest management, "there is no substitute for fire".

New schemes incorporating natural fire, controlled burning and mechanical thinning are beginning to take hold in the US and Australia, but changing opinion has been a slow process.

"Some survey research indicates a big shift in public attitudes favouring acceptance of prescribed burning," says Ingalsbee.

Fire suppression policies have tended to destroy spotted owl territory

However, while the public have become more accepting of controlled burning schemes, they remain sceptical of allowing natural wildfires to burn. "Unfortunately, it is only a few politicians who understand contemporary fire ecology science," he adds.

Whichever strategy you choose, the key to restoring fire-prone habitats seems to be introducing variety. Forests should be thinned and managed so that they contain a mosaic of individual trees, clusters of trees, and openings. These patches can be arranged to match natural features of the landscape – for example, cooler slopes facing north-east naturally support more trees than warmer, south-westerly slopes, and are less prone to fire.

By matching the management of an environment with its geographical features, we can even create vital habitat for rare wildlife and plants.

The spotted owl is native to redwood and other hardwood forests in the western US. Their natural habitat of dense forest with a high canopy can be extremely fire-prone, meaning that fire suppression policies have tended to destroy spotted owl territory. However, creating dense patches of hardwood forest on wetter, cooler slopes provides habitat for this endangered bird, without causing a serious fire risk.

The western world may have championed a no-tolerance policy on forest fires over the last few centuries, but people have not always been so keen to extinguish fires.

Aboriginal Australians used traditional fire management techniques, such as 'early dry season burning', to reduce the intensity of natural forest fires. However, these practises were discouraged and then largely forgotten with the arrival of European settlers in the late 1700s.

Similar traditional fire management practises have been used to control wildfire in South Africa and South America for thousands of years. They continue to be effective to this day, leading several researchers to advocate their use more widely.

We have become detached from the importance of fire in the landscape

In the US, wildfire prevention became popular after the Great Idaho fires in 1910, which burned 3 million acres and killed 87 people. In Europe and North America, they have "tried to exclude wildfires from the landscape, and this has had a detrimental impact on ecosystems that evolved with recurring wildfires," says Ingalsbee.

What's more, our evolutionary history has been shaped by fire.

A study published in April 2016 suggests that changes in the vegetation in Africa, from grasslands to woodier, shrub-like vegetation around 3 million years ago caused an increase in natural wildfires. These wildfires may have sparked our ancestors' imaginations, first using natural wildfires to their advantage when hunting, and eventually harnessing fire around 1.8 million years ago – about the same time early humans first left Africa.

It is really only within the last few centuries that we have become detached from the importance of fire in the landscape, and lost sight of the value it has for the proper functioning of nature. If researchers like Ingalsbee are persuasive enough, we may learn once more to live with fire rather than to always fight it.

Join over five million BBC Earth fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.