UDPATED 08/18/16 12:40 a.m.

RIO DE JANEIRO (CBSNewYork/AP) — Two American swimmers who said they were robbed during the Olympics have been removed from their flight to the U.S. by Brazilian authorities.

The United States Olympic Committee said Wednesday night swimmers Jack Conger and Gunnar Bentz were removed from their flight.

They will not be allowed to leave Brazil until they testify about robbery, according to an attorney.

“We are gathering further information,” USOC spokesman Patrick Sandusky said.

The two, along with Ryan Lochte and Jimmy Feigen, said they were robbed at gunpoint in a taxi as they returned to the athletes village from a party, several hours after the last Olympic swimming events were held.

Earlier Wednesday, Brazilian Judge Keyla Blank ordered that Lochte and Feigen stay in Brazil as authorities investigate their claim they were robbed during the Olympics.

Steve Lochte told The Associated Press by phone from his Florida home that his son called him Tuesday after arriving in the United States. The 32-year-old swimmer was going to pick up his car and buy a new wallet to replace the one that he said was stolen in the robbery.

“I’m just happy he’s safe,” the elder Lochte said. “It was an unfortunate experience for him and the other three. I don’t know what all the controversy is. They were basically taken out of the taxi and robbed. The main thing is he’s very lucky that he’s safe and that all they got was his cash and wallet.”

The elder Lochte said his son’s Olympic credential and cell phone weren’t taken during the incident early Sunday morning. He said he was sure Ryan had his passport or he would not have been allowed to board a plane.

Feigen’s whereabouts could not immediately be confirmed, though he told the San Antonio Express-News he was still in Brazil.

MOREMedal Count

The U.S. Olympic Committee said police went to the athletes village Wednesday morning to try to collect the passports, but the swim team had already moved out.

“We will continue to cooperate with Brazilian authorities,” spokesman Patrick Sandusky said.

The U.S. State Department issued a statement Wednesday encouraging those involved to cooperate with Brazilian law enforcement.

A USA Swimming spokesman said he couldn’t comment on athlete whereabouts because of security concerns. Lochte’s attorney, Jeff Ostrow, did not immediately return messages seeking comment.

CBS2’s Otis Livingston reported Brazilian officials said their story isn’t adding up. Police said they have found little evidence so far to support their accounts, and said the swimmer, who admit they were intoxicated, were unable to provide key details about what happened when they were returning from the French Olympic team’s hospitality house.

“Why would anybody fabricate anything?” Steve Lochte said. “It’s just ridiculous.”

A police official with knowledge of the investigation told the AP that police cannot find their taxi driver or witnesses. The official spoke on condition of anonymity because the investigation is ongoing.

Ostrow has said there is no question the robbery occurred.

Traveling with Lochte and 26-year-old Feigen were 21-year-old Gunnar Bentz and 20-year-old Jack Conger. Lochte swam in two events at the Rio Games, winning gold in the 4×200-meter freestyle relay. Bentz and Conger also earned gold for swimming in the preliminary heats of the 4×200 relay. Feigen received gold for swimming in the heats of the 4×100 free relay, which was won by the U.S.

Bentz’ father, Steve Bentz, declined comment when reached by the AP.

The group did not call police, authorities said, and officers began investigating once they saw media reports in which Lochte’s mother spoke about the robbery. Police interviewed Lochte and one other swimmer, who said they had been intoxicated and could not remember what type and color of taxi they rode in or where the robbery happened, the police official said. The swimmers also could not say what time the events occurred.

Later in the day, Lochte described the incident to NBC’s “Today” show .

“We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing just a police badge and they pulled us over,” Lochte said. “They pulled out their guns, they told the other swimmers to get down on the ground — they got down on the ground. I refused, I was like we didn’t do anything wrong, so — I’m not getting down on the ground.

“And then the guy pulled out his gun, he cocked it, put it to my forehead and he said, `Get down,’ and I put my hands up, I was like `whatever.’ He took our money, he took my wallet — he left my cellphone, he left my credentials.”

Word of the robbery initially created confusion between Olympic and U.S. officials. An International Olympic Committee spokesman at first said reports of the robbery were “absolutely not true,” then reversed himself, apologized and said he was relying on initial information from the USOC that was wrong.

Lochte told USA Today that he and his teammates didn’t initially tell the U.S. Olympic officials about the robbery “because we were afraid we’d get in trouble.”

(TM and © Copyright 2016 CBS Radio Inc. and its relevant subsidiaries. CBS RADIO and EYE Logo TM and Copyright 2016 CBS Broadcasting Inc. Used under license. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press contributed to this report.)