The owner of Banksy's Season's Greetings said he was taking a step back from talks to re-house the art in Port Talbot, after they were making him depressed.
John Brandler bought the graffiti for a six figure sum in January.
Neath Port Talbot council said it was working to find a satisfactory solution for all parties in the talks.
A Welsh Government spokesperson said: "Discussions with Mr Brandler and Neath Port Talbot council are ongoing".
Mr Brandler wants to house the art in an empty old police station in Port Talbot, making it a street art museum open to the public, but this has not been formally confirmed by the council.
The Essex-based gallery owner lives more than 200 miles from Port Talbot and has made three trips to Wales to discuss where to re-house the graffiti and funding plans.
But every day away from his gallery is costing him "an awful lot of money", according to Mr Brandler.
"It is costing my business a lot of money just to go to Wales to have these conversations," he said.
"I can't afford to bankrupt my business to discuss things which haven't happened. It is my business that is suffering.
"I am not going to be coming to Wales until I know what is happening."
He added that the lack of progress in the negotiations over agreeing funding and planning the project was making him frustrated and depressed.
Meanwhile, pictures from Amsterdam destined for the potential Port Talbot street art museum are sitting in his garage, Mr Brandler added.
"I certainly wouldn't have started this if I had known it was going to be this stressful," he said.
Mr Brandler bought the graffiti from garage owner Ian Lewis and promised it would be kept in Port Talbot for a minimum of two years.
He said it had been worked out how to move the graffiti, but parties were waiting on a formal agreement as to where it would go.
The graffiti is still protected on Mr Lewis' garage after the Welsh Government offered to take on security arrangements.
A council spokesperson said: "We are in discussion with Mr Brandler and partner agencies and are working to find a satisfactory solution for all parties."