Minnie Driver Recalls 'Agony' of Breakup with Matt Damon in Public Eye After 'Good Will Hunting'

Minnie Driver is looking back on her rise to fame over 25 years ago and the romance she once shared with her Good Will Hunting costar Matt Damon.

The 52-year-old actress — whose new book Managing Expectations: A Memoir in Essays was released Tuesday — reminisced about that time with Entertainment Tonight this week, discussing the first time she met Damon and fellow costar Ben Affleck.

"They were really late, they were like an hour late. And then they exploded into the suite, and they were so sorry, and there was something about a broken foot… and they were really hung over," she said of meeting the famous pair of friends. "It was like a bunch of really sweet labradors kind of tumbled into this thing, and I was like, 'Oh wow, these kids, these guys wrote [Good Will Hunting]?' "

When asked about her eventual romance with Damon that began while they filmed, and how those close to Driver "warned" her not to get carried away, she explained how her family helped her gain perspective.

"My family loved Matt — it wasn't that. It was that they could see that this young man was rocketing really fast and so was I, and when you're young, it's pretty hard to keep your head on straight and to maintain a grounded sense of deportment," Driver said.

"They were like, 'This may well end badly for reasons that are to do with all these things coming together in a perfect storm.' "

"And also, like, you shouldn't date someone you work with," she added of their advice.

RELATED VIDEO: Ben Affleck Says Matt Damon Was 'Principal Influence' on Helping His Career After Justice League

Good Will Hunting went on to become a highly acclaimed hit, earning nine Oscar nominations at the 1998 Academy Awards — including two for Damon, for Best Actor and Best Screenplay (which he won alongside Affleck), and one for Driver, for Best Supporting Actress.

The pair's romance, though, fizzled before the film hit the awards circuit, leading to a "surreal" experience for the Speechless star.

"I don't care who you are, that is agony and it's like a strange, surreal dream," Driver said of having to witness her breakup play out in the tabloids at the time. "But I know he didn't put that picture there. It's so tricky, because it's not deliberate, he couldn't have helped how famous he became and how his life was being picked over, in the same way that mine was."

"It's funny now, when I think about walking down the magazine aisle in my local supermarket at midnight with my best friend, we were like, 'Are you seeing this? This is super weird,' " she added.

"My friend, Alexandra was like, 'Oh my god, I can't believe this is happening, like, I feel like we're in a movie. I would totally watch this movie, even though I know you're super heartbroken, but this is really weird.' "

Driver credits her late father Ronnie Driver with giving her advice that ultimately helped her out of heartbreak.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"I was obviously brokenhearted over this boy that I loved. He was now with someone else," she recalled, going on to describe how one morning she was sitting on the steps outside her house in London when her father came out for a jog.

"He was like, 'Listen. You're not going to want to hear this, but the best thing I can tell you is, you're definitely going to feel this way about somebody else.' "

"And I was like, 'That is a terrible thing to say! I don't feel better. It's awful!' " she continued, adding: "He was so right!"

Also this week, Driver opened up to PEOPLE about her nearly 30-year acting career and why it hasn't always been easy to get roles.

When the actress got her first big break on the 1995 film Circle of Friends, she got paid $10,000 and thought she was set.

"I was like, 'Well, if I got this job, then I'm definitely going to get another one,'" she says in the latest issue of PEOPLE. "And then I didn't. I've had to fight really hard for every job I ever got. I feel like that was the first lesson in. It's never just going to come easily. And I think that's okay because the payoff is huge, like emotionally being able to do what you love for a living and make money at it."