NEW YORK (CNN)Harvey Weinstein‘s defense attorney Benjamin Brafman plans to withdraw from representing the former producer in his criminal case, according to a source familiar with the situation.

The source would not provide further details Wednesday.

When contacted by CNN, Brafman said he had no comment.

Weinstein is accused of raping a woman in a New York hotel room in 2013 and forcibly performing oral sex on another woman at his Manhattan apartment in 2006.

He faces five felony charges: two counts of predatory sexual assault, one count of criminal sexual act in the first degree and one count each of first-degree rape and third-degree rape. Weinstein has pleaded not guilty.

The case hasn’t been completely smooth for either side. In October, one of Weinstein’s six felony charges was dismissed after an NYPD officer was found to have mishandled evidence.

Weinstein’s petition to dismiss the remaining charges was declined in December.

Brafman is one of New York’s most prominent criminal defense attorneys. His past clients include Martin Shkreli, Plaxico Burress, Sean Combs and Dominique Strauss-Kahn.

In a series of interviews for a profile by Esquire, Brafman spoke about representing Weinstein.

“I’ve spent forty years trying to get to the top of my profession, and this is the most high-profile case in the United States,” Brafman said. “It’s flattering to be picked by someone like him.”

Brafman also addressed how the #MeToo movement has impacted Weinstein’s high-profile case and others.

“One day it’s Charlie Rose. Then it’s Kevin Spacey. Matt Lauer. Les Moonves. Suddenly they have something about Judge Kavanaugh from when he was in high school? Give me a break,” Brafman told the magazine. “I have a daughter; I have granddaughters. I’d like them to not have to deal with harassment, discrimination, and an unequal pay scale. But a movement becomes dangerous when it generates the kind of hatred that keeps a person from getting a fair trial.”

When speaking to CNN in August, Brafman said it would be “difficult but not impossible” to seat an impartial jury in Weinstein’s trial.

It’s not yet known who would take over for Brafman.

A representative for Weinstein declined to comment.