Robert Pattinson, sci-fi and sex in outer space – if ever the audacious, brilliant French director Claire Denis were making a bid for a mainstream audience, High Life would seem to be it. It turns out, the reverse is true. Before and especially after his Twilight years, Pattinson has sought out roles in smaller, artistic films, apparently on a mission to establish himself as a serious actor. The plan is working.

One simpatico director, David Cronenberg, cast Pattinson in the misbegotten Cosmopolis and underrated Maps to the Stars. His recent turn as a low-life criminal and protective brother in Josh and Benny Safdie’s Good Time was among his most layered roles yet. Denis is his latest kindred spirit, using his quiet presence to great effect.

-        Film review: Two stars for Venom

-        Roma is ‘the most exquisite film of the year

-        Colette film review: How radical is the new biopic?

High Life, Denis’ first film in English, is set on a spaceship full of prisoners sent on an almost certain suicide mission to explore a black hole. Despite the starry casting and high-concept plot, Denis has not compromised the scathing view of humanity or elliptical style of her early films such as Beau Travail and recent ones such as Bastards. High Life is provocative, eerie and above all weird. But even at its most willfully enigmatic it is anchored in this world by Pattinson’s terrific, humane, understated performance.

His character, Monte, and a baby named Willow, who is quite likely his daughter, are the ship’s last survivors. Early in the film, Monte puts the bodies of his dead colleagues in their spacesuits and sends them floating into the void.

Pattinson gives his character a resigned yet determined attitude. Monte forges ahead even though he no longer has contact with Earth, and even tries to teach the baby right from wrong. In this world “taboo” is among baby’s first words, and it comes laden with irony.

The spaceship is a minimalist design of beige and grey corridors, often cast in blue shadows. Within that cold, austere setting Pattinson’s scenes with Willow are gentle and endearing. The baby actor is the child of one of Pattinson’s best friends, a hint that casting directors should look close to home if they want to duplicate that comfortable father-daughter warmth.

Denis seems profoundly uninterested in space itself

As the film lucidly flashes back and forth in time Denis fills in details of the ship’s journey. There is never much of a plot. She creates a jagged narrative that reflects the characters’ unease, enhanced by Stuart Staples’ electronic score. All the prisoners had been sentenced to death or to life in prison, and chose space as an alternative.  They did not realise they would be used as human guinea pigs by a controlling doctor named Dibs (Juliette Binoche).

The rules of the ship forbid sex with another person, but solo activity is fine. Men are encouraged to provide sperm samples for Dibs’ experiments. She calls Monte a monk because he refuses, saying chastity will make him strong.

Technically, High Life is a no-sex-in-space movie, and Denis uses that restriction to explore thwarted desire and loneliness in a crowd. Binoche enters a chamber designed for sexual privacy, where she rides a device that is a variation on a mechanical bull, the camera roaming over parts of her body in close up. Denis has said that in an earlier version of the script, the device was called ‘the love machine,’ but everyone around her said the name was “too French.” Too wry is more like it. Binoche’s scene is not about physical fulfillment, but emotional frustration.

Most of the crew’s crimes are not fully explained. Apart from Monte, the one sympathetic character is played by André Benjamin, who stands out in a short scene. Digging through earth in a garden where the crew grows food, he explains that he chose to die in space so his son would be proud of him for helping further scientific knowledge. Mia Goth plays Willow’s mother Boyse, who is tough and unrepentant. The way Dibs gets Monte’s sperm and uses it (probably) to impregnate Boyse recalls Denis’ brutal view of sexuality in Bastards. Dibs’ crime on Earth, when it is revealed, turns out to be horrific.  

Eventually Monte’s T-shirt is in tatters, his cropped hair has a white patch, Willow has become an adolescent and Denis heads towards a deliberately ambiguous ending.  Throughout, Pattinson carries the film with few words and a sobering sense of restraint, conveying Monte’s unfulfilled longing and unstated regrets.

One of the oddities of High Life is that it did not really have to be set in space. Despite the haunting blackness outside Monte’s window, it is not a special effects extravaganza like Gravity, and doesn’t have the futuristic chill of 2001: A Space Odyssey. At times it evokes other last-person-on-the ship films like The Martian, especially in the gardening scenes. But Denis seems profoundly uninterested in space itself. Instead, the isolated setting allows her to focus on the essence of human behaviour, on the themes of love, guilt and desire that have propelled her long and stunning career, and that her newest lead actor make so captivating.

★★★★☆

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.