Was Prince Charles's Boarding School Experience as Awful as 'The Crown' Depicts?

The second season of The Crown gave us a closer look at Prince Charles‘s early years —including those at Gordonstoun, the Scottish all-boy’s institution where he attended secondary school.

In The Crown, Charles is sent to Gordonstoun at the insistence of his father, Prince Philip. Queen Elizabeth and Charles’s great-uncle, Lord Mountbatten, both preferred he go to Eton College, which is just a short distance away from home at Windsor Castle. The Crown even shows Charles being fitted for his Eton clothes, preparing to start school. But Philip is determined that his son follow in his footsteps and attend his alma mater.

When Charles arrives at the Scottish Highlands school, he’s quickly miserable. He’s bullied by his classmates, can’t keep up with the physical fitness aspect of his education and struggles with the more Spartan aspects of life at Gordonstoun, like windows that won’t shut in the rain — especially when compared to his life at home.

Philip, however, also had a difficult time adjusting to the school at first. The episode of The Crown, entitled “Paterfamilias,” featuring Gordonstoun shows Philip clashing with his peers and resisting some of the responsibilities he’s given at school, such as building walls. (At the time of Philip’s attendance, Gordonstoun was a new institution, founded by Kurt Hahn, a Jew who fled Germany due to the rise of the Nazis.)

Despite these early challenges, Philip eventually learned to love Gordonstoun, and his time at the school became something that he felt shaped who he was in the future. The Crown depicts the turning point coming after the untimely death of Philip’s sister Cecilie, who died in a plane crash in Belgium on her way to a wedding in London. On the show, Philip says the school “put me back together” after his sister’s tragic death.

“For Philip it is a wonderful, pioneering moment and in many ways was the making of him,” historian Robert Lacey tells PEOPLE.

It’s for this reason that Philip was so adamant about Charles attending the school. Lacey says his intentions were good: He simply wanted Charles to have the same sort of life-altering experience that he had at Gordonstoun.

“Philip has been depicted as cruel for sending him to the school, but Philip did it the best of motives,” Lacey says. “It was a fantastic, developing stage in his life, after his broken background. He thought it would be the making of Charles, but the school had changed. When Philip was there, he found the hardships, the challenges of the climate and countryside uplifting. By the time Charles was there, the school has become a much more conventional private school. It’s a poignant, powerful story.”

Charles’s time at the school is widely thought to have been unhappy, at least at the start. He’s even said to have nicknamed the school “Colditz in kilts” and reportedly called it an “absolute hell,” a quote The Crown includes in the episode. This is much of what the showfocuses on, including a particularly traumatizing experience during the school’s annual challenge. But in the midst of his time at Gordonstoun, he spent two terms studying in Australia. When he returned to the school, he was given the position of “Head Boy.” (Paging Percy Weasley fans.)

In the years since he graduated, Charles has even spoken fondly of Gordonstoun. He’s even said that talk that he detested the school to be “exaggerated” in a 1974 interview with Observer Magazine, and says his education there helped him grow as a person.

“I am always astonished by the amount of rot talked about Gordonstoun and the careless use of ancient clichés used to describe it,” he said in a 1975 speech to the House of Lords, almost a decade after he left the school. “It was only tough in the sense that it demanded more of you as an individual than most other schools did — mentally or physically. I am lucky in that I believe it taught me a great deal about myself and my own abilities and disabilities. It taught me to accept challenges and take the initiative. Why else do you think I am brave enough to stand up before your Lordships now?”

However, though Charles doesn’t seem to bear any resentment towards the school — at least publicly — his own sons, Prince William and Prince Harry, went to Eton.

• With reporting by Simon Perry