Beyond the futuristic architecture and Olympic hype lies a city quietly evolving, with art galleries taking over old factories and cafés staging literary festivals.

As the city looms ahead, the sun blasts through the morning smog. Watching it filter through the glass skyscrapers, I feel I am entering a forest of mirrors, struggling to recognise anything of the city I last visited 15 years ago. “Too big, too noisy and too fast,” I think to myself.

Close to the airport there are pockets of American-style suburbia – housing estates with names like Yosemite and River Garden, home to Beijing’s expatriate communities and the Chinese elite. Beyond these stretch endless industrial compounds and public housing blocks, all intersected and encircled by the city’s relentlessly expanding network of highways; the bicycles of old Beijing have been replaced by five million cars. On first encounter, this new Beijing can be a powerfully alienating place – the ultimate Asian megacity choking on its own success. Nearly 20 million people live here and the thrum of human noise is constant.

When I worked here as the BBC’s Asia correspondent in the early 1990s, Beijing had already escaped the constraints of strict Maoism but it was still an austere city. The headlong rush to riches had yet to begin. In those days, the great shopping highlight was haggling at the old antique market at Panjiayuan or browsing the stalls on Yabao Road, jostling with Russian traders who came south on the Trans-Siberian Express to buy clothes by the bulk load.

Now, walking along the street near my hotel in Sanlitun, the city’s business centre, I contemplate an entirely new Beijing: one brimming over with Western brand names and trendy bars, a place of concrete, glass and sharp angles. Luxury cars sweep by carrying the new masters of China. Some have made it through hard work, luck and business acumen. Others belong to the class known as “princelings”, relatives and friends of the party elite who have made their millions thanks to powerful connections.

In the years I have been away, the bulldozers have razed many of the city’s hutongs, the traditional narrow streets that survived the rise and fall of dynasties but could not compete with capitalism.

A few minutes later I am slurping my way through lunch in a noodle shop, where the main dish is Malan, a hand-pulled noodle from northern China, listening to my old friend, the writer and guide Lijia Zhang, telling me I have it all wrong: “Give it a chance, Fergal. Wait a few days. I promise you there’s another city.”

Lijia lives in the poetically named village of Jiuxianqiao, or “Wine God’s Bridge”, an area of narrow streets and low-rise brick houses barely 15 minutes’ walk from the mad bustle of Sanlitun. Here, you enter a portal into a city stubbornly resisting the aggressive advance of modernity. Old men gather on street corners to play chess. Families wander in the dusk. A man stands outside his home brushing his teeth. Escaping from an open window, the fumes of garlic frying in hot oil catch in my chest and bring water to my eyes. This is the hour when the city exhales – a slow release of the tension of the day.

Like so many of the capital’s residents, Lijia is not a native. She was brought up in the old imperial capital Nanjing, in the Yangtze Delta. Her story is typical of modern China’s narrative of change. She was a promising student with a love of English literature, but poverty forced her to leave school at 16 to become a factory worker. Within a decade, though, as China’s economic reforms created opportunities for the young and energetic, she quit the industrial drudgery and moved to Beijing.

I met her when she was translating for foreign journalists at the beginning of her own career as a writer; she was a young woman with a hearty laugh and an irreverent sense of humour. Fifteen years later she is the respected author of a best-selling memoir, the ironically titled Socialism is Great, is writing a novel on prostitution and also acts as a guide for travellers who want to experience Beijing’s rich intellectual heritage. “If you come here as a tourist, the danger is that you only get the big hotels, the shopping malls and a quick tour of the Forbidden City or the old Summer Palace,” she says.

That evening, at her home, writers and thinkers mingle and gossip. The dinner table is laden with dishes from all across China. There are Sichuan-style fried green beans with minced pork; a spicy gong bao (cubes of chicken fried with peanuts, peppercorns and chillies), a dish once denounced by Maoists as politically incorrect because of its associations with imperial China; and steamed egg with five spices, which is fragrant with the scent of aniseed and cinnamon.

I sit beside the journalist Raymond Zhou, whose combative columns in the China Daily reach millions of readers every day. Mild-mannered and genial in person but with a barbed wit, Zhou regularly scrutinises the pressures of a city undergoing rapid transformation.

“China is gripped by hatred, which is almost like an out-of-control machine-gun firing indiscriminately at any moving target,” he writes. The list of hate figures, according to Zhou, includes Westerners, officials, celebrities, the old, the young, the rich and the poor: “And we hate ourselves, because our relentless drive for a better life seems to go nowhere.”

The Bookworm is the city’s first English-language bookshop. Young Chinese mingle with expats over coffee, wine, prose and poetry. Manager Alex Pearson, whose father was a diplomat in Beijing in the 80s, also founded Beijing’s first international book fair. “There is a confidence in people which didn’t exist before,” she tells me. “It feels as if time is on fast-forward and there is this constant influx of interesting and stimulating people. It’s a fantastic city in which to experiment with ideas, a great place to establish – and occasionally lose – businesses.”

On the book festival’s opening night, Chinese “sensualist” Hong Ying, who was once sued for allegedly libelling a dead short-story author, shared the stage with an Argentine crime writer. Also that night, a Hong Kong novelist summoned up the universal loss of parental death while a Hungarian, Peter Zilahy, related tales of totalitarian madness in his own land.

In Beijing, one is constantly struck by the tension between old and new, between official constraints and the striving for artistic freedom. A filmmaker friend who has lived in the city for decades put it like this: “The artist is constantly stretching his arms out to see how close are the bars of the cage.” The events of the Arab Spring have heightened official fears of a revival of pro-democracy activism, as the recent arrest of the artist and activist Ai Weiwei illustrates. Yet I feel I am witnessing something very special and enduring – a cultural flowering overlooked by a Western world obsessed with economics. I head to District 798 in Dashanzhi, home to the city’s avant-garde artistic movement. Located in a former industrial complex built by the East Germans in the Bauhaus style in the 1950s, it is at first glance an unlovely place: an expanse of old workshops and factory compounds.

But inside is a warren of artists’ studios, galleries and performance spaces, as well as bars, restaurants and bookshops. There are striking Modernist sculptures and delicate ink paintings with artistic roots that lie in a Chinese tradition of calligraphy dating back several centuries before Christ. Artist and poet Fan Xueyi, who calls herself Sunlight in English, works with water and ink, her verse and imagery each complementing the other. “I think that by experiencing Chinese poetry and art the foreigner can see something of the essence of our culture, which is about achieving a harmony between life and nature,” she says. Fan is pragmatic when considering the rampant materialism of the new Beijing. “You can get rich quickly but you can’t have taste or sophistication quickly. But as people become rich, they’ll need finer things in life and will learn to appreciate art more. It will drive art forward in China.”

It is impossible to separate food and art in Beijing – Chinese cuisine is a great national art form. “To the people, food is heaven,” the old proverb says. In my time, I have sampled fiery pork dishes in the Cultural Revolution Restaurant, an emporium of questionable moral taste given the vast numbers of people persecuted in that era. I have devoured Mongolian hotpot while listening to Beijing opera in the upstairs room of a small café, where devotees proudly showed me photo albums of their favourite stars. And I have eaten a delicate turtle soup while watching dancers from the Dai ethnic group, of subtropical Yunnan province, flutter mothlike to an ancient melody wafting forth from a battered cassette player.

One of the jewels of Beijing’s traditional architecture and cuisine, the Mei Mansion, is set in a 200-year-old courtyard and named in honour of the great Beijing opera singer, Mei Lanfang. Mei always played women’s roles and is said to have maintained his feminine appearance by avoiding fatty foods. The resident chef here belongs to the same family that cooked for Mei. The cooking stresses the original flavours of the ingredients: there will be no hot spices to mask the stewed pig’s head, the meatballs and crab, or steamed fish.

But my favourite dining experience is to wander along the prosaically named “snack street” in Dongcheng district, or through the nearby night market, which vibrates with life after dark. The air is filled with the calls of the stallholders offering everything from the ubiquitous noodles to spicy Sichuan soup and Shangdong-style pancakes, deep-fried crickets and scorpions. With Lijia as my guide, I can eavesdrop on the back-and-forth banter of cook and customer in accents from all over China.

In purely economic terms, Beijing in the 21st century resembles what Manhattan meant to the “tired, poor, huddled masses” of Europe in an earlier epoch. It is a giant magnet, pulling people from across China’s vast hinterland in one of the greatest migrations in human history.

According to a recent estimate, one in three Beijingers is a migrant worker and many are employed in service industries. They form a silent army that sweeps the streets, waits on tables and cleans the hotel rooms of tourists. The state has tried to stem the flow, steadily rolling back the number of residence permits given to newcomers. Yet they keep coming.

One of the great challenges they face is education. By one estimate, there are 300 migrant schools catering to 500,000 children. Beijing is the great crucible of social reinvention, and every child learns that hard work at school is a potential escape route from poverty.

The road to Bowen School cuts across dusty fields and through grim industrial zones in southeast Beijing. As we arrive in the yard I spot two boys, aged about 10, racing ahead into the school. It is a cold, two-storey building of the uniform drabness that characterises the China of the toiling masses. Forget the tyranny of first appearances: the boys have alerted their fellow pupils to our arrival and I am greeted by a vibrant mass of children. They clap, they cheer, they surge around us like a living sea. “Hello.” “What is your name?” The English words are carefully pronounced and with relish. Clearly much effort has gone into their learning.

I am here on a Beijing Buddy visit, one of several programmes run by the Migrant Children’s Foundation, which promotes cultural exchanges between foreigners and the schools. Helen Boyle, who runs the programme, has watched children grow in confidence: “When we started teaching over a year ago, they weren’t able or very willing to speak at all and were quite shy. Now they can make sentences and are always very enthusiastic.”

Knowing the Chinese fondness for legends, I summon up some Celtic tall tales from my childhood that involve ferocious warriors, vicious kings, an improbably large potato, a brave young prince and the inevitable triumph of good. An enthusiastic Chinese volunteer translates. My heroes are cheered. My villains are hissed at. And the laughter and smiles stay with me all the way home.

Late in the afternoon, I go to see the old Summer Palace, the Yihe Yuan (“Gardens of Nurtured Harmony”). Here, the Empress Dowager Cixi spent vast sums of money reconstructing the complex after it was attacked by vengeful allied troops during the Boxer Rebellion in 1900. It’s a powerful symbol of restored Chinese national pride. Parties of schoolchildren pass by, doubtless being regaled with stories of iniquitous foreigners in times past; they pay me no heed. I sit by the edge of the Kunming Lake and, beside that wide expanse of water, after all the stimulation of the previous days, I feel a sense of peace.

Beijing may not capture your heart immediately. It is a proud, quirky, challenging place, a city of hustlers and of poets, where hard realities and dreams collide every day. On its streets you wander through yesterday and tomorrow, from ancient history to accelerating future, sometimes in the same shimmering minute, and there is nowhere, absolutely nowhere, quite like it in our world or in our time.

The article 'A Beijing state of mind' was published in partnership with Lonely Planet Magazine.