Lady C's agent claims she left I'm a Celebrity...Get Me Out Of Here! because she was being bullied.
ITV says that the 66-year-old left the jungle on medical grounds.
She'd argued with some of the other contestants and has been a controversial figure in the show's 15th series.
Because of her departure, the latest viewer vote has been cancelled. If you voted by phone since Tuesday night's show, you can get a refund.
Speaking to Newsbeat Lady Colin Campbell's agent Mike Hollingsworth said: "At 3am I got word that Lady C was coming out of the jungle.
"The reasons given were that she'd suffered a medical problem."
He says he thinks there's more to it than that. Mike didn't go into details but told us: "I could see the tension in the camp was building up.
"There was a lot of bullying going on and I think there was an element of misogyny as well."
During her time in the Australian camp Lady C clashed with some of the other contestants,.
She called singer Tony Hadley "a Dumbo, a chippy oik, a bully, a liar and the mouth of diarrhoea".
On another occasion she fell out with former Dragons' Den entrepreneur Duncan Bannatyne over emu poo.
She also decided against trying some Bushtucker Trials last week and refused to do the laundry.
Speaking of her relationship with many of the other people in the jungle Mike Hollingsworth said:
"She doesn't suffer fools gladly and if you cross her you know what you're going to get in return.
"And we've seen some of that on the programme."
Mike spoke with Lady C as soon as she arrived in her hotel after leaving the jungle. He told us:
"I was amazed at how lively she was, she had a glass of champagne in her hand and was doing well."
The latest celebrity to be voted out of the jungle, Chris Eubank, was one of Lady Colin Campbell's few allies in the jungle - along with Kieron Dyer.
"She's a mother figure, so naturally I gravitated towards her," he said.
"And she's wise, and she's experienced and knowledgeable, and all of that I am attracted to... Her conversation was invigorating and surprising."
The ITV show finishes on Sunday, when the king or queen of the jungle will be crowned.