Suddenly, they are everywhere: stretched across balconies, flapping from car antennas, and pinned to coat lapels the world over in a moving display of solidarity with the community that was brutally terrorised on Sunday after a bigoted assault at a gay nightclub in Orlando, Florida. At first glance, the rainbow flag’s joyous refraction of colour may seem a strangely sunny response to the dark savagery of the deadliest mass shooting in US history. But as we watch its jubilant stripes bind together communities around the globe, it is worth pausing to reflect on the origin of a cultural symbol that was propelled into iconic status nearly 40 years ago by heartbreaking tragedy.

According to the US gay activist Gilbert Baker, who is credited with creating the emblem in the late 1970s, the idea behind the flag’s bold design emerged in 1976 – the year the United States celebrated the bicentenary of its independence as a republic. Still reeling from the twin traumas of withdrawal from the Vietnam War in 1973 and the first ever resignation of a US President in 1974, following the Watergate scandal, America strove to conjure from national malaise a feeling of patriotism. Key to the summoning of such spirit was the restorative display across the country of the Stars and Stripes, whose simple geometry masked the intensity of the psychological, political, and social turmoil seething underneath.

“Flags are about proclaiming power” – Gilbert Baker

It was against this turbulent backdrop that San Francisco City Supervisor Harvey Milk, the first openly gay person ever to be elected to public office in California, encouraged Baker in 1977 to devise a unique symbol for the gay community – an insignia of pride capable of affirming social independence. “Flags”, Baker has since asserted, “are about proclaiming power.” As a drag queen whose appetite for flamboyant clothes in the glam-obsessed 1970s was bigger than his wallet, Baker became a dab hand with a sewing machine – a skill he would later put to use in producing political banners. Hypnotised by the talismanic intensity of the US flag and its ability to transmute itself in art and fashion – from pricey Pop Art paintings by Jasper Johns to grungy denim patches – Baker was drawn to the deceptive simplicity of a field of stacked stripes as a symbol for many stitched together as one.

In contemplating how, precisely, he should reinvent that pattern, Baker was aware that any design he produced would compete in popular imagination with a painful, if resilient, logo by which the gay community had long been identified. In Nazi concentration camps, men imprisoned because of their homosexuality were marked out by a pink triangle affixed to their clothing. In the decades following the end of World War Two, gay communities around the world stripped the pink badge of its intended humiliation and defiantly re-inflected it with pride. But however heroic that reclamation of meaning may have been, in Baker’s mind the symbol was still haunted by the ghosts of Hitler and the Holocaust. The gay community, he believed, deserved a fabulous emblem entirely of its own fashioning. “We needed something beautiful,” Baker concluded, “something from us.”

Blue sky thinking

Journalists and historians have expended a great deal of energy speculating how, exactly, the rainbow eventually suggested itself to Baker in 1978 as an appropriate phenomenon for conversion into the dimensions of a flag. A frequently cited hypothesis links the rainbow to the resplendent aura of the actress Judy Garland, long considered a gay icon (her role in the film The Wizard of Oz has yielded the colloquial slang-phrase “friend of Dorothy” as a term for a gay man), and to her famous performance of the song Over the Rainbow.

Other writers have noted that strong bright colours (such as the green carnation Oscar Wilde wore to signify his sexual orientation) have for centuries served as shorthand for homosexuality. But in an interview that Baker gave last year to the Museum of Modern Art, after the flag’s prototype was acquired for its permanent collection, he suggested that the rationale for selecting the rainbow was more elementary than these theories assume: “It’s a natural flag”, he insists. “It’s from the sky.”

It also has echoes in history, though its use in earlier cultural contexts around the world did not deter Baker from persevering with his sprightly spectral design. Since at least the late 15th Century, and the incorporation of the rainbow flag by the German theologian Thomas Müntzer in his reformist preachings, the symbol has been seized upon by religious and social activists to draw attention to their causes. A version was unfurled in the 16th Century during the German Peasants’ War to signify the promise of social change. In the 18th Century, the English-American revolutionary and author of the influential political tract Rights of Man, Thomas Paine, advocated for the adoption of the rainbow flag as a universal symbol for identifying neutral ships at sea. The flag has since been flown by Buddhists in Sri Lanka in the late 19th Century as a unifying emblem of their faith, by Indians annually on 31 January to commemorate the passing of the spiritual leader Meher Baba, and, since 1961, by members of international peace movements.

Colour therapy

In its initial incarnation, Baker’s rainbow flag consisted of eight colours – two more than the version now recognised internationally as an emblem for the LGBT community – and each colour was assigned a symbolic meaning. A band of hot pink (representing sexuality) ran across the top of the flag in the original scheme, followed by red (which stood for life), then by orange (for healing), yellow (sunlight), green (nature), turquoise (magic), indigo (serenity), and violet (spirit) at the bottom.

Displayed for the first time in the United Nations Plaza in downtown San Francisco in June 1978, this eight-striped version was produced by a team of 30 volunteers commandeering the washing machines of a public laundromat in order to rinse the dye from the fabric and the wide attic space of a gay community centre, where the individual strips were ironed and sewn together. This is the version that Harvey Milk would have known, if all too briefly, in the few short months before he and the Mayor of San Francisco, George Moscone, were shot to death in City Hall on 27 November by a deranged former colleague of Milk’s.

Demand for the rainbow flag accelerated in the aftermath of the murders for display at gay pride parades and at events organised in honour of the slain champion of LGBT rights. For a variety of practical reasons, Baker was forced to whittle the design down, first by dropping the top band of hot pink (a pigment that proved difficult to source) and then turquoise (inclusion of which, for reasons of symmetry, made displaying the flag as a vertically-hung banner from lampposts awkward). In the 38 years since Baker offered his invention as a universal symbol of gay pride, proliferation of its prismatic joy has been impressively steady. In 1994, a mile-long version of the flag, which streamed through the streets of New York to mark the 25-year anniversary of the 1969 Stonewall riots in Greenwich Village (a crucial moment in the gay liberation movement), set a record for the longest flag ever produced and helped plant the symbol permanently in social consciousness.

Baker’s creation is threaded by perseverance and tinged by pain

Now the flag is everywhere. On 12 June, it was waved by marchers in the first major gay pride rally ever held in Ukraine. Between 24 and 26 June, the flag will be flown for the first time above the UK Parliament to celebrate the London pride weekend. In 2015, Facebook introduced a rainbow filter after the US Supreme Court decision ruling that same-sex marriage was legal nationwide. But the backlash to that in Russia and the Middle East, and events in Orlando, are reminders that this is not just a flag for celebration. Though ostensibly ebullient in its design, Baker’s creation is nevertheless threaded by perseverance and tinged by pain. “Flags”, he is said to have exclaimed on the 20th anniversary of the symbol’s birth, “are torn from the soul of the people.”

100 Works of Art That Will Define Our Age by Kelly Grovier is published by Thames & Hudson.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.