The US Secret Service prostitution scandal involved as many as 20 women, 11 American agents and some military personnel, senior US officials say.
Senator Susan Collins, briefed by the Secret Service director, said 20 women were found at the US hotel.
The incidents took place in Cartagena, Colombia, ahead of last weekend's Summit of the Americas.
On Monday, the head of the US armed forces said the Secret Service and the military had "let the boss down".
Secret Service Director Mark Sullivan was "rightly appalled by the agents' actions and is pursuing a vigorous internal investigation", Ms Collins said in a statement.
"He ordered all the agents to return to Washington immediately, and all have been interviewed," said Ms Collins, the top Republican on the Senate's Homeland Security and Governmental Affairs Committee.
Speaking to reporters on Tuesday, Ms Collins, a Republican who represents Maine in the Senate, also said she had asked Mr Sullivan a number of questions during her phone briefing.
"Who were these women? Could they have been members of groups hostile to the United States? Could they have planted bugs, disabled weapons, or... jeopardised [the] security of the president or our country?"
Speaking to reporters afterwards the senator said some agents "were uniformed personnel who are assigned to building security". Others were "these specialised agents who do security details", she said.
US congressional homeland security committee chairman Peter King told Reuters news agency that "so far there is no information that any of them were involved with any narco-terrorist group or any organised crime".
Two of the agents were senior personnel with salaries at the top end of the government's pay scale, the Washington Post reported.
All 11 agents have been placed on administrative leave and had their security clearance revoked. They were not directly involved with presidential security.
A Marine Corps spokesman said on Tuesday that among the military service members being investigated were two Marine dog handlers assigned to support the Secret Service.
The White House meanwhile said it had confidence in the director of the Secret Service to investigate the incident, adding that he had addressed the matter quickly.
On Monday a senior official told the BBC that at least 10 military service members were also under investigation, double the five that was originally reported.
General Martin Dempsey said the military did not know exactly what had happened in the Colombian city of Cartagena, a colonial city on the country's Caribbean coast and venue for the Summit of the Americas.
"What we do know is that we distracted the issue from what was a very important regional engagement for our president," he said, adding it was an embarrassment for the agency.
On Sunday, the US president said that he expected a "rigorous" investigation.
"If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," Mr Obama said.
Details of what happened on Wednesday night are still coming to light. The group of military and Secret Service agents were partying at Cartagena's Pley Club, which has been described as a high-end strip club in an industrial part of the port city.
Members of the Secret Service paid $60 (£38) each to the club's owners to bring at least two women back to the Hotel Caribe, where they were staying, the Washington Post reports.
The next morning one of the women demanded more money and a dispute ensued.
ABC News has said the agents were bragging about their work for the president, telling their company that evening: "We're here to protect him."