Just a few hours’ drive from Dubai, the little-visited Musandam Peninsula is a remote patch of land where ancient fishing villages give way to treacherous mountain roads.

As the little boat drifted perilously close to jagged rocks, I gaped upwards, lost in wonder at the scale of the honey-coloured cliffs that rose from nothing and soared to loom more than 200 metres above my head. Gulls circled, arcing out over the sea and back towards their perches on the barren crags. The sea was full of life, but these bluffs, weathered and cracked into dramatic shards, were utterly devoid of even the hardiest plant or shrub. They were like cliffs at the edge of a glorious Martian sea.

The skipper had been napping. Suddenly, he leapt up as the boat scraped rock, swore at himself in Arabic, and quickly pushed us away from the cliffs with his foot. He guided us into a narrow cove almost hidden among the giant slabs. The water darkened and the temperature dropped. The surroundings looked like a Norwegian fjord, but with no green – as if life itself had been scraped from the rocks with a wire brush.

I was visiting Oman’s Musandam Peninsula – a remote patch of planet separated from the rest of the country by a corner of the UAE – while filming a new BBC series, Indian Ocean, which took me through 16 countries around the edge of our most glorious ocean. The region juts out into the Strait of Hormuz, the key choke point at the eastern end of the Persian Gulf, through which chug scores of oil tankers taking vast quantities of the world’s supply of black gold to Europe, the US and Asia.

Long before oil was discovered in Arabia – creating some of the richest countries on Earth and apparently endless demand for Swiss watches and German sports cars – Omanis had built an empire that stretched down Africa’s east coast to Zanzibar. Many will tell you that the Queen of Sheba had her palace in Oman, and that Sinbad the Sailor set sail from one of its ancient ports.

It is hard to overstate the difference between Oman and its neighbours. While the latter have most of the oil and grand building projects and bling, Omanis remain wedded to 5,000 years of history and culture, preserving traditional buildings, opening museums and occasionally bemoaning the flash modernism of their neighbours in the UAE. “Can you believe they built a ski slope, indoors!” an Omani businessman told me with disdain.

In the midst of all the glitz of the modern Gulf, home to vast shopping malls and some of the most luxurious hotels on the planet, Musandam is a place where life itself moves at a different pace: it lacked roads until just a decade ago. From my bobbing boat, I could see why. With metal-melting temperatures during summer and terrain that needs Thor’s hammer to flatten and tame, the landscape was one of the most dramatic and inhospitable I have ever encountered.

Yet Musandam is just a few hours’ drive from Dubai, and reachable for a weekend break. Strike out towards the peninsula along the coast from the brashest of the Emirates and the tip of the Burj Khalifa – the world’s tallest building – fades behind as you motor through dirty roadside sprawl along tarmac where I once saw a group of 18-year-olds racing Bugattis and Bentleys.

You pass flaming oil wells, through the emirates of Sharjah and Ras al-Khaimah, before the endless development withers and the straight road begins to buckle towards the Musandam border, where a cliffside track winds around the fractured coast. It is the Gulf, but not as you know it: otherworldly Musandam is not a place for those wanting fusion food and FIJI Water, fine hotels or a Lamborghini showroom.

Khasab, the region’s capital, has a splendid fort, built by the Portuguese to keep their sailors supplied with dates and fresh water. The influences of other, more modern, arrivals were being felt during my visit – locals hummed with excitement about the imminent opening of their first supermarket. Yet the little town is not asleep. As I wandered, the still air was broken by the sound of roaring engines, while at the harbour, a dozen speedboats powered their way out to sea. My guide, Badr, a beaming bear of a man, smiled. “They’re Iranians, smuggling fridges.” It is surely one of the most bizarre international trade routes. US sanctions have apparently created a shortage of household appliances in Iran, so smugglers race across the Strait of Hormuz to buy white goods in Musandam and shift them back to the Islamic Republic, dodging giant tankers and the gunboats of the Iranian coastguard. I watched them waving and laughing as they rocketed past with washing machines, microwaves and plasma TVs.

The BBC team and I followed the smugglers out of Khasab harbour in a boat, behind dolphins that streaked through the water like torpedoes. Despite the tanker motorway a few miles to the north, the waters around Musandam are pristine. Just an hour from Khasab are dive sites with schools of blue and yellow Indian Ocean angelfish, Arabian butterfly fish, snappers, lionfish, groupers, stingrays and turtles. Some experts think the coral of Musandam is among the best in the world due to the absence of heavy industry along the coast and few fishing fleets in the area.

Rounding a rocky island outcrop, we slipped into a sheltered bay. At the far end, smoke wafted lazily from the ancient village of Kumzar – a cluster of low houses beneath the cliffs. Badr had arranged for us to meet a local fisherman, Abdul Salim, who was sitting with his men on a rocky beach as we drifted to shore. A shout rang out and his men leapt into action, dragging a long net out into the water.

‘Why are you doing this now? What have you seen?’ I asked.

‘Ah, we have eyes in the heavens,’ said Abdul with a conspiratorial grin.

The Kumzari have a unique language – the only one of Persian origin on the south side of the Gulf – and they began to quietly sing in it as they spread the net in a 50-metre arc out into the water then back towards the shore. As the net was pulled tight the singing became louder. Soon it sounded like a bawdy shanty, accompanied by cheeky laughs from the men. As Abdul’s troops tugged and the fish tumbled to the shore, I spotted a metallic glint on the sheer cliff face. “See!” he said, “those are our eyes!”

An old man emerged from a canvas hide with a pair of binoculars and a kettle. His job was to shout when he spotted a shoal of fish enter the cove, so the team below could trap and catch it. “We have fished this way for generations,” said Abdul. “Minimum effort, but maximum reward.”

The next day, the dusty heart of Musandam beckoned me away from the coast. Until recently, the stony interior could only be explored on the back of a mule. Now, a road climbs steadily into the great Hajar Mountains that stretch for more than 300 miles, reaching 3,000 metres at its highest point. The road soon became a track carved into the hillside, with extraordinary drops off the side into oblivion. We crested one hill and the entire landscape opened before us. Mountains hewn, carved and hacked from the ancient foundations of Earth swept to the horizon.

Driving down from the heights, we came across a wedding. Several hundred men wearing white thobes – traditional ankle length robes – and headdresses had gathered under a tent by the road on the outskirts of an arid town. Suddenly there was the crack of gunfire. “Are they allowed to have guns at weddings?”  I shouted to Badr as another volley of shots rang out. “Not in the rest of Oman, and certainly not somewhere like Dubai. But remember, out here,” he replied with a smile, “we’re in the wilderness.”

The article 'Oman’s sleepy Musandam Peninsula' was published in partnership with Lonely Planet Magazine.