Giuliana Rancic: I Know I'm Too Thin

Giuliana Rancic finally speaks out about her Fashion Police controversy, the health problems she s kept secret, and why she s become shockingly thin. Subscribe now for instant access to the exclusive interview, only in PEOPLE.

For months, she’s stayed mum on the subject of her body, but now, Giuliana Rancic, who says the firestorm of attention has “gotten out of hand,” is finally speaking out.

The E! News anchor’s slim appearance at the Jan. 11 Golden Globes sent the Internet abuzz with rumors of eating disorders – and worse.

“Some people were saying, ‘The cancer is probably back,’ ” Rancic, who was diagnosed with breast cancer in 2011 and underwent a double mastectomy that year, tells PEOPLE. “And they were accusing me of every eating disorder. I thought to myself, ‘God, if someone really thought I had an eating disorder, what a horrible way to approach it.’ ”

Thankfully, Rancic, 40, is healthy, though she says the cancer-suppressing medication she has been taking since 2012 has had an effect on her body.

“I started noticing that I was eating a lot, but not gaining weight at all,” Rancic says. “I was concerned.”

Her oncologist, Dr. Dev Paul of Rocky Mountain Cancer Centers in Denver, Colorado, informed her that the drug, which can alter metabolism, could cause weight loss.

“It’s really hurtful,” says Rancic of the attention, adding that she had hoped criticism about her appearance would quiet down on its own. “I’m sorry that some people think I’m disgustingly skinny, as they put it, but there’s nothing I can do. I’m lucky that I even have the type of cancer that reacts to the medicine.”

When it comes to her appetite, “I eat more than any of my friends,” she says. “I eat a very robust, healthy, balanced diet and dessert almost every night. I’m not hiding from anything.”

Though the mom of son Duke, 2½ (with husband Bill), admits that the physical change has been difficult to bear.

“I look in the mirror and it’s hard for me,” she admits. “I am really thin. I want to look fit and beautiful and sexy, and I can’t.”

RELATED: Giuliana Rancic on Fertility Struggle: We Lost Our Last Embryo

Speaking out now, Rancic, whose memoir, Going Off Script, hits shelves April 7, hopes the tides will turn.

“I never want my weight to distract people from what I do,” she says. “But it is becoming a distraction, [whenever] I wear something sleeveless, show my arms, my back.”

In fact, Rancic says a major publication was interested in featuring her but backed out, telling her she was “too skinny.”

“That was really hurtful,” she says. “I thought, ‘Wow, I’m on cancer medication. I’m doing the best that I can.’ ”

Despite the public backlash, that mindset has served Rancic well.

“I refuse to be broken [by what people say],” she says. “I have so many beautiful things in my life.”

For much more from Rancic, including a very personal reveal and the truth about the Fashion Police controversy, pick up the latest issue of PEOPLE, on newsstands Friday