Rural groups say police guidance that people can drive to the countryside to exercise will cause "untold anxieties".
The National Rural Crime Network and other groups said it risks spreading the virus through unnecessary journeys.
Driving to the countryside for a walk is "likely to be reasonable" if more time is spent walking than driving, the guidance says.
Police groups say the advice is not for the public - it is meant to help officers decide when to charge someone.
The letter challenging the guidance is signed by the National Rural Crime Network, the Countryside Alliance, the National Farmers' Union, and the Country Land and Business Association, who say they represent "many millions of residents and thousands of businesses" in England and Wales.
They said they receive "hundreds of concerned messages a day" about people flouting the laws restricting movements, and say there are serious concerns this guidance will "encourage even more people to carry out unnecessarily long journeys".
They have written to Justice Secretary Robert Buckland demanding a change to the pandemic advice.
"The key message needs to remain: stay home, save lives. Anything which complicates that message is unhelpful," the letter says.
The letter demands that the police guidelines are "urgently reviewed".
The guidance was collated by the National Police Chiefs' Council and the College of Policing, but was based on advice from the Crown Prosecution Service.
The College of Policing said the advice was for internal purposes - not for the public - and "was designed to help officers remain consistent with criminal justice colleagues" when deciding when to charge someone.
It follows complaints that officers were being heavy-handed in enforcing the law restricting people's movements during the coronavirus outbreak.
Derbyshire Police were criticised for filming walkers in the Peak District using a drone in an attempt to warn people away from visiting the countryside. Some people, such as former justice secretary David Gauke, said the walkers appeared to be observing social distancing measures.
Under the guidance, officers are advised that driving to the countryside for a walk is "reasonable" if "far more time" is spent walking than driving.
It also says that police should not intervene if people stop to rest or eat lunch while on a long walk, but short walks to sit on a park bench are not permitted.
But the rural groups say the advice needs to be strengthened or it will cause "increased pressures on rural police forces and communities".
Human rights group Liberty says the wide-ranging powers need careful scrutiny, however.
"Times of crisis can create the conditions for our rights to be swept away," says director Martha Spurrier.
"The government has responded to the coronavirus pandemic by giving police sweeping powers resulting in some heavy-handed policing. This needs to be urgently addressed."
Update and Correction 22 April 2020: The College of Policing has since clarified that this guidance is for internal use and our article has been updated. An earlier version of the article incorrectly reported that the guidance was published in the week following the Easter bank holiday weekend and this has been amended to make clear that it was made available to police forces on Thursday 9 April.