In Alex Garland’s Annihilation, The Shimmer appears as a lovely, iridescent wall, a gel-like substance of alien origin enclosing a recently evacuated area. It is enticing, poetic and ominous, like this cerebral sci-fi thriller itself, which takes Natalie Portman and four other scientists inside the area to investigate what the Shimmer holds and why all previous missions have disappeared inside it.

It is philosophical about human nature and identity without ever being pretentious

Following his sharp focus on artificial intelligence in Ex Machina, Garland has written and directed a film that is broader but scene-for-scene just as masterful. Annihilation is philosophical about human nature and identity, without being in the least pretentious. It addresses the psychology of self-destructive behaviour. And it drops in effective jump-scares. Mutated beasts, including a bear and an enormous white alligator, pounce on the team apparently out of nowhere. Throughout, Garland fills the screen with images that become ever more ravishing as the team gets closer to some answers.

- Isle of Dogs is stop-motion at its best

- Fifty Shades Freed is a total disaster

- Black Panther: The most radical blockbuster ever?

Garland cleverly lays out the plot, from Jeff VanderMeer’s bestselling novel. We first see Portman being questioned by men in HazMat suits after her return from The Shimmer, and learn she is the lone survivor of her expedition. The film is a long flashback in which we learn that Portman’s character, Lena, is an Army veteran, now a biology professor at Johns Hopkins University.

Her husband, Kane (Oscar Isaac), who disappeared on a secret military mission a year before, comes home as a nearly catatonic, changed man. The deadened look in Isaac’s eyes is evidence of how much the actor can do with a glance, and of how effectively Garland uses extreme close-ups. Eyes matter in this film, from the characters’ opaque or revealing gazes to abstract images that make it feel as if we’re staring into the eye of the sun or the moon.    

When it turns out that Kane is the only survivor of his own trip into The Shimmer, Portman joins the next team in, led by a terse psychologist, Dr Ventress (Jennifer Jason Leigh). Gina Rodriquez gets the splashiest role and comes through as the tough, quick-to-anger paramedic, Anya. Tuva Novotny plays Cass, an anthropologist. Josie, a physicist, is played by Tessa Thompson, who is always so good that you wish her role here were not so underwritten (or over-edited).

The fact that this is a female team is handled perfectly in an aside. “All women?” Lena asks when she meets the other members. “All scientists,” she is told, and that’s the end of that.

Into the zone

Within The Shimmer, the women wield guns and at times experience group memory loss. They find horrifying evidence of the previous team’s experience, which only deepens the mystery. And Lena quickly discovers that all life forms, not just the threatening animals, are mutating. Multi-coloured flowers of different species climb a tree trunk. White deer have antlers made of pink-flowering tree branches. Or are the women experiencing shared hallucinations? As Lena approaches a lighthouse assumed to be the source of The Shimmer, she encounters a beach on which trees made of shards of glass or ice rise out of the sand. Rob Hardy, who was also the cinematographer on Ex Machina, makes each image graceful and each scary moment visceral.

The further the team explores, the more we see of each character’s particular vulnerability. Cass is grieving for a dead child. Anya is a sober addict. In flashback we learn that Lena’s marriage was not as perfect as it seemed at the start. “Almost none of us commit suicide,” Ventress says about the team’s apparent suicide mission, extending it to a sweeping assessment of human nature. “Almost all of us self-destruct.” 

Garland playfully borrows from classic genre films and makes those references and influences his own. There are scenes that evoke 2001, Close Encounters of the Third Kind, Alien and any number of Terrence Malick films. The minimalist, electronic score by Geoff Barrow (of the group Portishead) and Ben Salisbury adds a subtle layer of mystery.

The film’s mind-bending conclusion is wonderfully open-ended. It suggests how much of what we have seen has been shaped by the characters’ points of view, and how delicately Portman has shaped her role. The ending and all those poetic images make Annihilation ripe for allegorical interpretation, but that ham-fisted approach would reduce Garland’s expansive vision.

In the run-up to the film’s opening, it was widely reported that Garland and his producer, Scott Rudin, had resisted changes based on negative responses to a test screening. It’s easy to see the source of that problem. Annihilation has an auteur’s indie sensibility, a mid-range budget of around $40 million, and a wide studio release in the US. (Paramount sold it off to Netflix internationally.) That’s never an easy combination, but a film this ambitious, suspenseful and beautifully made is certain to find its audience.

★★★★☆ 

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.