The UK should withdraw aid from Rwanda if its government is threatening opponents living here, says the former Foreign Office Minister Kim Howells.
The BBC has learned at least three exiled Rwandans have been told their lives are at risk from the Rwandan government.
The UK is Rwanda's biggest direct aid donor, giving £83m a year.
The Rwandan High Commissioner in London has dismissed the allegations as "bogus".
Earlier this year two Rwandans living in London received warnings from the Metropolitan Police that the Rwandan government posed an imminent threat to their lives.
Jonathan Musonera is a founding member of the Rwanda National Congress, an organisation launched last year in opposition to President Kagame's government, which is accused of carrying out human rights abuses, and suppressing political freedom.
The BBC has now identified a third UK resident who has also been notified of a similar threat. This third man, Noble Marara, says he is aware of two others who have also been warned of danger, suggesting at least five Rwandans may be at risk.
Mr Marara believes he is in danger because he gave evidence to a French judicial enquiry that was detrimental to Mr Kagame.
Since arriving in the UK in 2005, Mr Marara has moved house more than seven times, and changed his car three times. He will not eat in restaurants for fear of being poisoned.
"I stay away from the Rwandese community," he says. "I may speak to them, but not meet them or show my address, and I cook for myself because I cannot trust anybody - that's for sure.
"Many people have been poisoned."
Since receiving the warning that his life is in danger, Jonathan Musonera has doubled the locks on his door, changed his daily routine, and forbidden his wife and daughter to leave the house unaccompanied.
"I was very scared. The Rwandan government, they can use anything, they can use anyone."
The British government is a strong supporter of Rwanda, and one of its biggest direct aid donors - giving the country £83m a year.
Network of spies
But former Foreign Office Minister Kim Howells, who was head of the Intelligence and Security Committee until last year, says President Kagame is becoming increasingly autocratic.
He argues these death threats are the latest in a series of events that raise a question mark over Britain's continued support: "If there's any hint at all that these people are threatening people whether they're British citizens or residents, then we must say to them 'I'm sorry this aid is going to be cut off immediately', and that's a threat they certainly could not afford to ignore."
Paul Rusesabagina of Hotel Rwanda fame was honoured for sheltering more than 1,000 Rwandans during the genocide.
Now living in exile, he has become one of President Kagame's most high-profile critics. While he welcomes the humanitarian aid provided by the international community, he argues that Britain is wrong to give the Rwandan government direct financial support:
"My message to the British people, the British administration would be to stop what they call direct aid, this money injected into nation's budgets.
"Since the UK is among the few nations which are giving cash to today's Rwandan government, I would advise [them] to stop".
The BBC understands that British ministers are taking the death threats extremely seriously and they called in the Rwandan High Commissioner in London, Ernest Rwamucyo, to give an explanation.
The Foreign Office refused to comment on the outcome of those discussions, but made it clear that the UK does not tolerate such activity.
However, Mr Rwamucyo told the BBC: "We were really quite shocked and found the Metropolitan Police warnings are bogus, baseless and absolutely untrue."
"There's no way Rwanda would ever think of doing such a thing, and of all places, in the UK. We have such a strong relationship, one of confidence and trust, with the UK. We don't have any reason whatsoever to even think of doing such a thing."
Despite those denials, many of those File on 4 spoke to believe there is a network of spies at the heart of the Rwandan refugee community in Britain, and claim it includes students, asylum seekers and former Rwandan Patriotic Front soldiers.
Mr Mugenzi used to work for a refugee organisation in London and says he has seen the documents of individuals who claim to be fleeing the Rwandan government but then become fervent supporters.
He is convinced the Rwandan government furnishes its spies with documents to help prove the case for asylum in the UK.
Rwandan High Commissioner Ernest Rwamucyo dismisses these allegations as pure nonsense: "There's no basis for that. Whoever is making those allegations should bring the evidence."
The International Development Minister Andrew Mitchell declined to speak to File on 4.
In a statement, the Department for International Development stressed the strength of Britain's commitment to Rwanda, pointing out that the aid given will help many of the country's poorest people.
The statement adds: "The UK-Rwanda relationship is a candid one and we raise issues where we have concerns on a regular basis and at senior levels.
"We take every opportunity to raise with the Rwandan government our concerns over political space, media freedom and extra-judicial killings.
"We continue to urge the government of Rwanda to address these issues and to bring the perpetrators of human rights abuses to justice."