Julie Chen Moonves Credits Her 'Strong Bond' with Husband Les for Surviving His Sex Scandal (Exclusive)

Five years ago, Julie Chen Moonves involuntarily left The Talk in the wake of her husband Les Moonves being accused of sexual harassment and assault by 12 women.

The But First, God author, 53, decided to stand by her husband since 2004. She began to use her married name when she singed off of Big Brother and issued a statement in the wake of the allegations calling the former CBS exec, 73, “a kind, decent and moral human being.”

Chen Moonves tells PEOPLE in this week’s issue: “I never took his last name when he was running the media company where I was working, where it could be intimidating. It kind of felt obnoxious, but legally, I was ‘Julie Chen Moonves,’ and I thought, ‘OK, now that name isn't so hot, this is when I take it.’”

At the time, Chen Moonves thought her hiatus from The Talk would only last a few days or weeks.

“But then when I watched the [season] premiere and I saw what was said, it became so clear to me like, I can never go back,” she recalls. “It was like watching my funeral.”

Following her departure, “I was feeling lost and angry and confused and just at a loss,” Chen Moonves adds. “I didn't know what to do next. I just felt like I wanted some answers and I didn't know what to do with myself.”

Chen Moonves badly wanted her personal life to stop making headlines.

“There were just too many rumors about me moving out and me leaving my family. It was so not true,” she says. “They had pictures of moving boxes, but it was like, we were moving back into our house because we had done some work [on] it. I just wanted all the speculation and the false reports to end.”

During that tough period, Chen Moonves developed a relationship with God. But she credits her resilient connection with Moonves, rather than God, for carrying them through that time. (Last November, Moonves had to pay $2.5 million to CBS stockholders whom New York state Attorney General Letitia James believed were kept in the dark initially about the allegations against him — though he never admitted any wrongdoing.)

“We have such a strong bond,” says Chen Moonves. “We leaned on each other and we knew that nothing could break us. It was our love and our foundation that got us through.”

Chen Moonves continued hosting Big Brother because “it's not a show where you're delving into your personal life,” she explains. “It's not about me, it's about the house guests, and I'm just the facilitator.”

For more on Julie Chen Moonves, pick up the latest issue of PEOPLE, on newsstands Friday, or subscribe here.

While Chen Moonves says in her audio memoir “I was collateral damage” for her husband’s mistakes, she reveals that her issues on The Talk predated 2018. She felt betrayed when her co-host Leah Remini and some other colleagues went to CBS and told the network they wouldn’t work with Chen Moonves because they found her to be too uptight. As a result, Chen Moonves iced out Remini, 53, for years.

"At the time, it felt like a betrayal,” Chen Moonves says.

Chen Moonves admits that Remini apologized one year later, but the mom of 13-year-old son Charlie didn’t feel ready to reconnect until she began her walk with God.

“She has such a big heart,” Chen Moonves says of the King of Queens star. “She's one of the funniest people I know and she's very passionate in her beliefs. She is a good person.”

Chen Moonves, who got baptized in 2022 and now belongs to five churches, believes that her story serves as proof that “it is never too late for anyone to start a personal relationship with God,” she says. “Once you do, you'll have peace that transcends all understanding. You're not alone.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

But First, God is out now wherever books are sold.