GPs are warning they risk spreading coronavirus if they follow NHS calls to see more patients in person.
The Royal College of GPs said it was "bemused" by guidance on seeing patients in person after replacing many in-person appointments with online and telephone calls.
NHS England has warned doctors they face fines if they do not make clear their patients can see them in person.
Age UK said some elderly people felt their GP was not "open for business."
Doctors' surgeries have changed the way they operate during the pandemic, after the government urged practices to increase the use of phone and video appointments.
In January, 80% of all consultations were carried out in person.
Latest NHS Digital figures from July, however, show only half of appointments were face-to-face, while around 45% were over the phone.
This figure varies depending on the NHS region. North Warwickshire had reduced face-to-face interactions to 28% by July, according to the published data, though health bosses there have disputed the official NHS figures.
Earlier this week NHS England wrote to every GP surgery in the country to remind practitioners they had a duty to also offer face-to-face appointments as it was feared many with serious conditions were being missed.
The charity Age UK is among those who believe the inability to walk into a practice and book an appointment has been putting some off attending.
Chairman of the Royal College of GPs, Professor Martin Marshall, said those in the profession had been left "bemused" and "angry" from the NHS England letter, however, as previous guidance had been to limit in-person contact with patients.
Doctors understood the inability to see patients in person could lead to missed diagnoses, he said.
He said this was particularly the case if the patient displayed signs of cancer and needed a physical examination, but GPs had had to balance that with the risk of waiting rooms becoming a site for Covid-19 transmission.
"To suggest that GPs should open up their doors on the basis that a small number of patients are unhappy that they can't see their GP just seems frankly silly to me," he said.
BBC analysis shows the overall number of appointments recorded by GPs fell from 26.8m in January to 22.4m in July, although July's figures were still higher than in previous months of lockdown.
Age UK director Caroline Abrahams said, while recent surveys had shown about half of older people would be "comfortable attending an online appointment", some struggle to use digital services and others are concerned about catching the virus by attending in person.
"We have also heard from older people struggling to get through to GP surgeries, facing long waits on the phone to be able to speak to a receptionist, and feeling like their GP hasn't really been open for business," she said.
"Others faced extra difficulties because they had hearing problems, communication difficulties, or other health conditions which made a phone appointment all but impossible."
Prof Marshall said: "At the peak of the pandemic, our own figures showed about one-in-ten appointments were face-to-face, now it is about one-in-three and I think that is about right.
"No matter how much we try and socially distance people, if people are coughing and spluttering in the waiting room there is a risk to both clinicians and patients."
Prof Marshall said he believed the reason for the overall fall in appointments was largely due to factors outside of GPs' control, including that in some surgeries, GPs themselves have had to self-isolate.
Many patients did not want to expose themselves to the risk of infection, he said, while others did not want to be a burden on the NHS.
"We have been saying all along that if you have symptoms suggesting cancer please come along," Prof Marshall said.
"That message has been hard to get through when the overriding message has been to 'protect the NHS'."
An NHS England spokesman said it was "entirely right" more GP appointments were available by phone and online.
He said: "That is completely consistent with also saying that there still needs to be options of face-to-face appointments - which these figures confirm are available everywhere."
More about this story
The Shared Data Unit makes data journalism available to news organisations across the media industry, as part of a partnership between the BBC and the News Media Association.
Reporting team: Anna Khoo, Paul Lynch, Rob England and Alex Homer.