The UK is operating a "parallel, unfair and secretive shadow justice system" for terror suspects, according to human rights campaigners.
Amnesty International said the effect of control orders had been to "by-pass the ordinary criminal justice system".
The group called for the orders to be scrapped by the Home Office which is due to complete a review of counter-terrorism legislation in November.
The Home Office said it had already pledged to review control orders.
Nicola Duckworth, director of Amnesty's Europe and Central Asia programme, said: "The effect of the control order regime has been to by-pass the ordinary criminal justice system in order to impose severe restrictions on the rights of individuals.
"Five years on, it is time for an end to the control orders regime. The measures used under the Prevention of Terrorism Act have created a parallel, unfair and secretive shadow justice system for individuals who are suspected of terrorism-related activity."
Last month, the Court of Appeal paved the way for terror suspects to sue for damages after ruling the control orders of two men were unlawful.
The three appeal judges rejected a government challenge to a High Court ruling that quashed the men's control orders, which it said were legally flawed. The Home Office said it would strongly resist paying damages.
Control orders were introduced in 2005 after it was ruled that suspected extremists could no longer be held in prison without charge.
Amnesty said the orders could impose restrictions including house arrest, electronic tagging, bans on contacting others and limits on using bank accounts.
A Home Office spokeswoman said: "We have already made clear by the home secretary's statement of the 13 July to Parliament our intention to review the control order system while taking whatever steps necessary to protect the public.
"That work is under way as part of the review of counter-terrorism legislation and is due to report back in the autumn."