Only two people were arrested for terrorism-related offences last year after being stopped and searched under anti-terrorism powers.
They were among 91,500 such searches of people and vehicles in England and Wales in 2009-2010, Home Office statistics show.
Liberty said the figures demonstrated "what a blunt and counterproductive instrument" the powers can be.
Last year, restrictions were placed on the measure after a court ruling.
The statistics show that apart from the two terrorism-related arrests, a further 436 people were arrested for other reasons.
The previous year there were 10 terrorism-related arrests - but more than double the number of searches.
The figures also show that almost a third of all stop and searches by police involved ethnic minority groups: 16% of those stopped were black, 11% Asian, 3% mixed race, 1% Chinese or other ethnic group.
Having been searched, however, white suspects were most likely to be arrested.
Fifty-five per cent of all Section 60 searches - where police search in anticipation of violence to look for knives and other weapons - involved ethnic minority suspects.
In total, police carried out 1,360,167 searches of people and vehicles in 2009-10 - a drop of 10% on the previous year.
The total number of arrests fell by 5% to 107,444, amounting to 9% of all searches.
The figures also show that the number of people given a fixed penalty notice for using a mobile phone while driving in England and Wales went up by 8% in 2009 from 116,000 to 126,000.
James Welch, legal director for civil liberties organisation, Liberty, said: "Only 8% of the UK population are from minority ethnic groups, yet they make up a third of those stopped and searched - even more when the police use search powers that don't require suspicion.
"Those in politics and the media who constantly call for broader police powers should be mindful of the negative long-term impact they have on community relations and public trust."
In 2010, the European Court of Human Rights ruled the power to search people without suspicion under Section 44 of the Terrorism Act 2000 was illegal.
The government announced restrictions on their use, and the police are now not allowed to use the power unless they "reasonably suspect" a person of being involved in terrorism.
A Home Office spokesman said stop and search was a "vital tool for police officers" but it had to be deployed in "an intelligence-led way".
"Any inappropriate use of stop and search powers would be a concern and we want to work with the police service and other interested groups to ensure every stop and search is fair, efficient and justified," he said.
"We have already amended guidance on Section 60 stop and search powers to emphasise that ethnicity should never be the sole reason for their use."