Papa John's founder John Schnatter has resigned as chairman of the board.
The company made the announcement late Wednesday, hours after Schnatter apologized for using a racial slur during a conference call in May.
Forbes said Schnatter used the N-word during a media training exercise. When asked how he would distance himself from racist groups, Schnatter reportedly complained that Colonel Sanders never faced a backlash for using the word.
In a statement released by Louisville, Kentucky-based Papa John's, Schnatter said reports attributing use of "inappropriate and hurtful" language to him were true.
"Regardless of the context, I apologize," the statement says.
The University of Louisville also said Wednesday that Schnatter resigned from its board of trustees, effective immediately.
Schnatter stepped down as CEO last year after blaming slowing sales growth on the outcry surrounding football players kneeling during the national anthem. He remains chairman of the company he started when he turned a broom closet at his father's bar into a pizza spot.
Papa John's shares fell nearly 5 percent Wednesday after the report, closing at $48.33.