NEW YORK (CBSNewYork) – President Donald Trump planned to focus on infrastructure Tuesday as he continues his working vacation in New York City, but he faced more fallout over his response to the recent violence in Charlottesville, Virginia

For the first time since his inauguration, Trump awoke in New York City, where signed an executive order to streamline the infrastructure permitting process. He called the current system a “massive, self-inflicted wound on our country.”

“Just blocks away is the Empire State Building. It took 11 months to build the Empire State Building. But today, it can take as long as a decade and much more than that,” he said. “[There are] many, many stores where it takes 20 and 25 years just to get approvals to start construction.” 

Questions from reporters quickly turned to the president’s initial silence on white supremacist groups in Charlottesville.

When pressed why he waited two days to condemn the groups, Trump said, “I didn’t wait long.”

“I wanted to make sure — unlike most politicians – that what I said was correct,” he said.

“This event just happened. In fact, a lot of the event didn’t even happen yet as we were speaking. This event just happened. Before I make a statement, I need facts. So I don’t want to rush into a statement. So making the statement when I made it was excellent,” he added. “It takes a little while to get the facts. You still don’t know the facts. And it’s a very, very important process to me, and it’s a very important statement. So I don’t want to go quickly and just make a statement for the sake of making a political statement.”

The president went on to place some blame on the counter-protesters.

“What about the alt-left that came charging at the – as you say – the alt-right? Do they have any semblance of guilt?” he asked.

“You had a group on one side and you had a group on the other, and they came at each other with clubs, and it was vicious, and it was horrible, and it was a horrible thing to watch,” he said. “But there is another side. There was a group on this side — you can call them the left, you’ve just called them the left — that came violently attacking the other group.”

“I think there’s blame on both sides,” he added.

After the news conference, former KKK leader David Duke tweeted: “Thank you President Trump for your honest & courage to tell the truth about #Charlottesville & condemn the leftist terrorists in BLM/Antifa.”

A short time after Trump’s remarks, Virginia Gov. Terry McAuliffe tweeted a statement, saying, “this was not ‘both sides.'”

“Our Commonwealth and the nation are still reeling from one of the largest outpourings of hatred and violence we have experience in recent history. We need real leadership, starting with our president,” he said.

Trump arrived in New York City Monday night and was met with both supporters and protesters outside Trump Tower.

“What happened in Charlottesville was done with Trump’s endorsement, with his permission. And he can barely criticize it,” Elissa Krauss said.

Protests continued Tuesday night outside Trump Tower on Fifth Avenue, though crowds were far smaller than they were on Monday night when Trump arrived.

“Most people don’t understand. That’s the problem. Most people are unwilling to get out of the comfort zone and come out and do something,” one man named Marty told WCBS 880’s Mike Smeltz.

There was only one organized protest Tuesday night, focusing on immigrant rights. But Marty said it was important nonetheless.

“It makes me feel good in my soul. I must be here. I left the comfort of my home, got on a train. I was here yesterday. I’ve been involved in every demonstration since he’s been elected, and I will continue to demonstrate,” Marty said.

Ellen from Brooklyn said diminished numbers should not be interpreted as diminished passion.

“I don’t think we can judge the numbers based only on what we see right here,” she said. “People are very upset about what’s going on.”

Activist and filmmaker Michael Moore also held a vigil outside Trump Tower tonight along with the audience of his Broadway show, “Terms of My Surrender,” and actor Mark Ruffalo joining as a special guest.

The group called on Trump to fire members of his administration they claimed are affiliated with white supremacists, including White House Chief Strategist Steve Bannon, senior policy advisor Stephen Miller, and deputy assistant Sebastian Gorka.

Trump did explicitly name white supremacists and other hate groups two days after the Charlottesville attack on Saturday, after first condemning what he called an “egregious display of hatred, bigotry and violence on many sides, on many sides.”

“Racism is evil, and those who cause violence in its name are criminals and thugs, including the KKK, neo-Nazis, white supremacists,” Trump said Monday.

Yet, the president declined to explain his delay in denouncing the hate groups.

“They have been condemned. They’ve been condemned,” he said.

Monday night, the chief executive officers of Intel and Under Armour announced they were leaving the president’s American Manufacturing Council. They followed Kenneth Frazier, the head of Merck, who left because of the president’s initial response to Charlottesville.

One day later, Walmart Chairman Douglas McMillion said the president missed a chance to unite the country, but said he will not quit the council, CBS2’s Dick Brennan reported. Scott Paul, of the Alliance for American Manufacturing, also stepped down Tuesday, while two representatives from the AFL-CIO said; “we cannot sit on a council for a president who tolerates bigotry and domestic terrorism.”

“They are not taking their job seriously as it pertains to this country,” Trump said Tuesday.

Web Extra: Scaramucci Blasts Bannon On ‘Late Night’: ‘Up To Me, He’d Be Gone’

Also Tuesday, political watchers are focused on another possible shakeup back in Washington, CBS2’s Chris Wragge reported.

Multiple sources tell CBS News that chief political strategist Steve Bannon’s job is in serious jeopardy, because Chief of Staff John Kelly is working to restore order to the White House.

When Trump was asked about Bannon’s status Tuesday, he said he’s a good person and we’ll have to wait and see what happens, Brennan reported.

In a tweet Tuesday morning, the president appeared to endorse the possibility of a pardon for former Arizona sheriff Joe Arpaio. The 85-year-old was convicted of illegally detaining Latinos because he suspected them of being undocumented immigrants. He’s scheduled to be sentenced in October.

Security around Trump Tower is tight, with police conducting random checks of vehicles and bags along Fifth Avenue.

“We’ll be checking inside the vehicle – in the truck – before they’re allowed to proceed down Fifth,” Chief of Department Carlos Gomez told Diamond. “We have a good number of traffic agents assigned to the area.”

The president will be returning to his home in Bedminster at some point Wednesday.