NEW YORK (CBSNews) — The fallout continued Friday for Papa John’s, as business partners and even universities that have accepted large sums from the pizza chain’s founder scrambled to distance themselves after John Schnatter admitted using a racial slur during a business call.

The University of Louisville announced that the school’s football stadium will no longer carry the Papa John’s name. The stadium is to be redubbed “Cardinal Stadium” effective immediately, university president Neeli Bendapudi said Friday.

Later that day, Schnatter admitted to using the N-word on a conference call and apologized, but said it was taken out of context.

In a radio interview with WHAS in Louisville, Schnatter said he feels “sick” about the incident. Explaining how he came to use the slur, he said was “just talking the way that the Colonel talked,” according to The Associated Press — a reference to the Colonel Sanders of KFC. But “I said it, and it’s wrong,” he told the station.

Cutting ties

The University of Kentucky, which houses the “John H. Schnatter Institute for the Study of Free Enterprise,” on Friday said it would no longer use Schnatter’s name.

“We appreciate Mr. Schnatter’s understanding that his unacceptable language is contrary to the values of the University of Kentucky,” it said on Twitter. However, believing that maintaining ties with him would create a “barrier,” the statement said, “neither the Gatton College of Business and Economics nor the Institute will recognize Mr. Schnatter in any way.”

Schnatter helped found the center in 2005 with a he made an $8 million gift to the school. The Charles Koch Foundation contributed $4 million to help start the center.

Even Papa John’s Pizza is moving to distance itself from the man who started the company. The chain on Friday said it was pulling Schnatter’s image from its marketing materials. It also plans to launch a listening tour and hire an outside auditor on matters of inclusion, it said.

“We want to regain trust, though I know we need to earn it,” CEO Steve Ritchie said in a statement.

Papa John’s does not plan to change its name, according to the Associated Press.

The face of the company

Schnatter has long been the face of the Papa John’s brand, and the company has acknowledged in regulatory filings that its business could be hurt if his reputation was damaged. That scenario seemed to play out last year after Schnatter blamed disappointing pizza sales on the outcry surrounding NFL players kneeling during the national anthem. At the time Papa John’s was an NFL sponsor. He resigned as CEO soon after.

This week, Schnatter stepped down from his role as chairman of the board of the pizza chain. He remains the company’s largest shareholder, however, owning about 30 percent of Papa John’s shares.

Sports teams are also pulling the plug on partnerships with Papa John’s in wake of Schnatter’s comments.

The New York Yankees said on Friday they were ending their relationship with the franchise. “In response to the reprehensible remarks made by Papa John’s founder and owner, the New York Yankees are suspending their relationship with the company,” the MLB team said on Twitter.

Previously, the Miami Marlins said they were ending their relationship with the chain, including ending a promotional deal and closing the Papa John’s concession stand at Marlins Park, according to the Miami Herald. The Tampa Bay Rays made a similar move on Thursday, ending a promotion offering fans half-price pizza after a game in which the team scored six or more runs.

Also on Friday, the NFL’s Atlanta Falcons and Atlanta United of Major League Soccer said they are suspending their partnerships with Papa John’s and reexamining the relationship, according to the Atlanta Journal-Constitution.

“The divisive comments made and acknowledged by the company’s founder are reprehensible and do not align with our core values,” AMB Sports and Entertainment, a group representing the franchises, said in a statement.

Hometown fallout

Fallout from the incident is having local repercussions as well. Schnatter’s hometown of Jeffersonville, Indiana, is putting distance between itself and the man it once called a “hometown hero.” The mayor of Jeffersonville, Mike Moore, returned a $400,000 donation from Schnatter to restore the Nachand Fieldhouse, a historic gym.

Schnatter had pledged $800,000 to renovate the building, which also carried his name until earlier this week.

“I’m not going to allow that name with those comments to be attached to anything in the city of Jeffersonville,” Moore told CBS News affiliate WLKY. “An $800,000 gift to the city of Jeff is enormous, but the city of Jeff is worth more than $800,000. It’s not for sale,” Moore said.

Schnatter is also a frequent donor to Republican candidates in Kentucky and nationwide.