An Audit Scotland report has called for a fundamental review of community health partnerships (CHPs).
The bodies were set up to join together work previously carried out by hospitals and social work departments.
The watchdog said there was limited evidence that the establishment of CHPs had brought about widespread sustained improvements in services.
Doctors' representatives said CHPs had "spectacularly failed to bridge the gap between health and social care".
They described it as a "damning report" and said CHPs were "bureaucratic monoliths".
CHPs were established under the National Health Service Reform (Scotland) Act 2004.
By 2010, 36 CHPs had been set up in Scotland, managing a total budget of more than £3bn.
They were intended to improve health and quality of life by joining up health and social care services, and moving more services from hospitals into the community.
But the Audit Scotland report said that, despite controlling a quarter of Scotland's budget for health and social work services, they had faced a number of barriers to achieving their aims.
It highlighted examples of good practice but said they were local initiatives on a small scale.
Some health trends have been worsening, the report said.
It pointed out that more older people, and more people with problems such as chronic respiratory illnesses, were being admitted to hospital as emergencies.
Doctors' representatives said the findings were deeply disturbing.
Dr Dean Marshall, who chairs the BMA's Scottish general practitioners committee, said: "This is a highly critical report which confirms our experience of the management and performance of these organisations.
"GPs have turned their back on CHPs because they have become bureaucratic monoliths caught up in their own processes rather than influencing the planning, funding and development of local services to meet patient needs.
"They have also spectacularly failed to bridge the gap between health and social care, a challenge that must be met in order to care for out increasingly elderly population."
The chairman of the Accounts Commission for Scotland, John Baillie, said: "For a decade there has been a focus on improving partnership work between councils and NHS boards.
"However, the added value of these partnerships is not clear.
"The report makes strong recommendations for all partners to address this together."
Responding to the Audit Scotland report, Health Secretary Nicola Sturgeon said: "Community health partnerships have to change.
"Planning care for increasing numbers of older people is one of our biggest national challenges and health boards and councils need to work together far more closely.
"We want to see health and social care for adults delivered in an integrated way by NHS and council social work staff and community health partnerships will have to change to adapt to this."
Scottish Labour's health spokeswoman Jackie Baillie said: "This damning report highlights the drastic need for reform."
She added: "The purpose of CHPs was to ensure better joint working between NHS and social care services but this report highlights that they have largely been allowed to wither on the vine."
"I urge the Scottish government to undertake a wide-ranging review of the structure and function of CHPs to ensure they are fit for purpose - doing nothing is simply not an option."