In 1986, a year before the artist’s unexpected death, an exhibition of new work by Andy Warhol opened at a gallery in London. Commissioned by the British art dealer Anthony d’Offay, it contained five monumental self-portraits, each measuring 9ft by 9ft (2.7m by 2.7m). In all of them, Warhol’s gaunt, spectral face topped by a spiky, silver ‘fright wig’ appeared to float against a dark background.

Among the crowd at the private view was a 20-year-old art-history student called Jeremy Deller, whose encounter with Warhol that evening would prove fateful. Ultimately, Warhol inspired Deller to become a successful artist in his own right. In 2004 he won the Turner Prize, and in 2013 he represented Britain at the Venice Biennale.

More recently, he has been curating an unusual new exhibition at Modern Art Oxford.  Opening on 6 December, the show will compare Warhol’s achievements with those of the 19th Century British polymath William Morris (both, for instance, were prolific printmakers). Recently I met Deller to hear how Warhol changed his life almost three decades ago.

“I was a first-year art historian at the Courtauld Institute of Art,” he recalls, “but I was already very aware of Warhol – he was one of the first contemporary artists who I had really connected with.” Eager to meet his hero, he secured an invitation to the opening of Warhol’s show at d’Offay’s gallery off Bond Street in London.

At the party, he briefly met the American Pop artist, who signed a baseball cap for him. But he had a more fruitful conversation with one of Warhol’s friends, who told him: “Oh, do ring us up – we’re at the Ritz.” “And I thought, ‘there’s no way I cannot do this’, so I did.”

Two days later, Deller and a friend arrived at the Ritz hotel in central London where they were ushered up to Warhol’s suite. There they were greeted by Warhol and five hangers-on: “An advertising guy, his dealer from Milan, a guy from Holland, a fashion designer, and a photographer – all male,” Deller recalls. “They were sitting around watching [British TV comedian] Benny Hill with the sound turned down and listening to Roxy Music, which I thought was really cool.”

Realising that “we were, in a sense, that night’s entertainment”, Deller and his companion accepted an offer of a drink and began chatting to the group. Things, he says, remained “relatively innocent”, until the moment when Deller had his picture taken next to Warhol. At that point, Deller recalls, Warhol became a little bit… He trails off. “What?” I ask – “touchy-feely?” “Yes,” he says, before pausing. “Well, actually he groped me.” Seriously? “Yes, a proper, between-the-legs grope. It’s true.”

How did Deller react? “I was surprised. But I was also incredibly flattered. I thought, ‘that was Andy Warhol who just did that to me – amazing.’ Besides, he was more of a voyeur than a doer. He liked seeing other people do it [have sex], and he’d draw and photograph them quite a lot, but he wasn’t really a participant. So that’s probably as far as it would have gone. He wasn’t predatory in that way.”

Factory line

Beguiled rather than threatened by this attention, Deller was delighted when a member of the party suggested that he and his friend should travel out to New York that summer to help at Warhol’s studio, which was known as the Factory. “I was like: okay, I will take you up on that,” says Deller. “So my friend and I went out there and hung around the Factory for a couple of weeks.”

By this point in Warhol’s career, the Factory, which had already moved around New York City several times, was located in an unusual office building on East 33rd Street. The days of the ‘Silver Factory’ of the 1960s were long gone, when the studio had been full of “crazy, druggy people”, as Warhol once put it, “jabbering away and doing their insane things”. In place of the crazies were slicker executive types, employed to manage the sprawling operations of Andy Warhol Enterprises Inc. These included producing television shows and publishing Interview magazine, as well as creating art.

“There were all these interconnecting rooms,” Deller recalls, “with a door that went into another building behind that was the headquarters of Interview. So the whole studio set-up was like being in Warhol’s mind: you had the publishing section, filmmaking on the top, a painting studio, a business part, a dining room. He’d created a world.” Deller was struck by the casual atmosphere in the office: “It was a very cool, funky, work/play environment, which is now de rigueur for tech companies like Google.”

During his time at the Factory, Deller was able to observe Warhol at close quarters. “He was a mythical character by then, really, because of how he looked: jeans, the big wig, big glasses, black polo neck, a very expressionless face, almost like a mask. It was like he was ageless. But he was very curious about things. Whenever I talked to him, he really wanted to know what was going on: where I’d been, who I’d seen, what had happened. He was a kind of intelligence-gatherer.”

Artist-in-training

Deller recognised that despite some awkwardness, Warhol was a strongly social being. “And I liked his gossipy persona. It was mischievous and subversive – and funny. Perhaps people haven’t picked up as much as they could have done on how funny he was. He never really played the game, but always wanted to cause a bit of cultural chaos.”

Spending time in Warhol’s company was intoxicating. “It kind of ruined me, really,” Deller says. “Afterwards, when I went back to the Courtauld, I would look at drawings by, say, Annibale Carracci, and even though I loved that world, it didn’t seem relevant anymore.”

In the years that followed, Deller decided to become an artist himself. “I didn’t go to art college, but walking around the Factory was my training,” he says. “Looking into rooms and seeing things going on made me realise that as an artist you could do whatever you wanted. So, for me, it was a life-changing time.”

Almost three decades have elapsed since Deller met Andy Warhol. These days, he is less starry-eyed about this colossus of 20th Century culture. But he remains convinced of Warhol’s greatness.

“Everyone assumes that Warhol was just surface and shallowness,” he says. “But there was much more depth to him than that. I think of him almost as an imperial artist. He chronicled the American empire when the country was at its absolute height between the ‘50s and the ‘80s. Everyone who was anyone had their portrait made by Warhol, and those portraits of powerful people in America are brilliant and beautiful objects. He documented an age, really, and was super ambitious not only for himself but also for art with a capital ‘A’.”

Alastair Sooke is art critic of The Daily Telegraph

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.