Wales defence coach Shaun Edwards says the national side would benefit if some star names returned from France.
But Edwards would welcome moves from any players returning to play their club rugby in Wales.
He said: "You've got control over the conditioning and you've got control over the games they play."
Wales head coach Warren Gatland has said the Welsh Rugby Union has not "instigated contact" aimed at bringing scrum-half Phillips, centre Roberts and back-rower Lydiate back from France.
|Wales defence coach Shaun Edwards|
|"When players go to France, there's not many of them who come back better players."|
But Gatland did say that "one or two of them are keen to return to Wales".
The Racing Metro trio are among a number of Welsh internationals who have joined clubs in France over the last few years, with Jonathan Davies at Clermont Auvergne and Leigh Halfpenny at Toulon joining the exodus during the summer.
Former Great Britain rugby league captain Edwards feels the demands of playing in the Top 14 did not necessarily benefit players.
"There's no doubt that when players go to France, there's not many of them who come back better players," Edwards added.
"[Steffon] Armitage is a big difference, he's been there many years.
"But from a Welsh point of view I would definitely say the fitness levels when they come back to the Welsh team after being in France are probably not what we expected them to be."
British and Irish Lions pair Phillips and Roberts have been linked with a move from France along with club-mate Lydiate.
But Phillips tweeted on the speculation: "Not sure why my name was mentioned today.
"I'm very happy at Racing Metro and the coaches are very happy with me also."
Roberts tweeted: "Funny few days. Contrary to fictitious 'press' belief, I'm not going home this week. Merci."
The speculation arose when Racing coach Laurent Labit told French newspaper L'Equipe: "We have raised the subject [leaving] with them. They have never really been that effective since their arrival."
The Paris club declined to make further comment when contacted by BBC Sport Wales, while the agents of the three players and Regional Rugby Wales were also asked to comment.
A further report in L'Equipe on Wednesday speculated that former Newport Gwent Dragons flanker Lydiate could leave by November.
Roberts and Lydiate joined Racing ahead of the 2013-14 season while Phillips moved there in December 2013 after being dismissed for disciplinary reasons by former club Bayonne.
If they were to play for potential new teams at the start of this season's European competitions on Friday, the trio would have had to move before 12:00 BST on Tuesday, 14 October.
And if they play in the European Rugby Cup at all for Racing Metro - who host Northampton on Saturday in their pool opener - the players will not be able to represent another team in the tournament this season.
Should any of the trio return to play for a Welsh region it could be on one of the dual-contracts that were created under the new Rugby Services Agreement signed by the regions and the WRU in August.
Wales captain Sam Warburton is the only player on such a deal at the moment.
More Wales stars are expected to be placed on similar deals.