Property multimillionaire David Rowland has said he will not become Conservative Party co-treasurer, having previously accepted the job.
He said he was "honoured" to be chosen but wanted to focus on his "developing business interests" instead.
The Conservative Party refused to comment on reports David Cameron was urged not to appoint him by senior party figures.
Mr Rowland has faced criticism of his former tax exile status.
According to The Spectator, former Tory deputy chairman Lord Ashcroft, who gave up his non-dom tax status to remain in the Lords, and the party's current co-treasurer Michael Spencer had both warned the Conservative leader that Mr Rowland's appointment could generate adverse publicity and that other donors would not want to deal with him.
Mr Rowland, who according to The Sunday Times Rich List has an estimated fortune of more than £700m, lived in Guernsey for tax purposes for many years.
He returned to the UK last year to enable him to make a series of donations to the Conservative Party, totalling more than £2m in the run up to the general election.
Political parties are banned from accepting donations from foreign donors.
The Daily Mail has run a number of articles on Mr Rowland's business interests and private life which has led some commentators to speculate that the party had failed to anticipate the sort of coverage his appointment would generate.
Tim Montgomerie, co-editor of the grassroots ConservativeHome website, which is part-owned by Lord Ashcroft, said: "Big questions need to be asked about CCHQ (Conservative headquarters) failing to carry out due diligence on this appointment."
The Conservative Party refused to comment on the allegations.
In a statement, it said Stanley Fink, currently the Conservative party's co-treasurer with Mr Spencer, will continue as sole treasurer when Mr Spencer steps down in the autumn.
A Conservative spokesman said: "We understand Mr Rowland's reasons for not taking up the role and remain grateful for his support."
Mr Rowland said: "I was honoured to be asked to become treasurer. Unfortunately my developing business interests mean I will not have the time to give that role the focus and attention it deserves.
"I remain deeply committed to the future success of the Conservative Party. David Cameron is a great leader of the party and of our country and I will continue to actively support him and the party in the years ahead."
It comes amid Lib Dem anger at the coalition government's appointment of retail tycoon Sir Philip Green to advise on Whitehall efficiency savings.
The Topshop magnate pays taxes in the UK but has faced criticism because his company is owned by his wife, who lives in the tax haven of Monaco.
Lib Dem peer Lord Oakeshott, who advises Business Secretary Vince Cable, told BBC News: "In principle it must be right that anyone in a prominent position in public life is firmly resident and domiciled in the UK for tax purposes."
Deputy Prime Minister Nick Clegg played down the importance of Sir Philip's role when he was asked about it on a walkabout in his Sheffield constituency.
He said: "He is going to come in for a few weeks, doing a very short piece of work, to see how money was spent in the past and how it could be spent better in the future.
"Like everybody, I am looking forward to seeing his report."