The Real Reason Tom Baker Left Doctor Who
From 1974 until 1981, there was only one Doctor Who.
That's saying something, considering the franchise has existed in one form or another since 1963, with over a dozen actors occupying the role of the titular, time-traveling character. Baker, while merely the Fourth Doctor, was also the longest-serving one, and continues to be one of the most beloved. His booming voice, storm cloud of curly hair, and manic grin remain synonymous with the character to this day, to the point where it remains practically a right of passage for every Doctor Who fan to own an unnecessarily long, Baker-style scarf.
However, no one — not even Baker — can be the Doctor forever. Just like every other actor to occupy the role, both before and since, Baker eventually moved on. What you might not know, though, is the real reason Baker left Doctor Who after nearly seven years, was the result of some behind the scenes messiness.
Tom Baker conflicted with the Doctor Who producers
As you know, Tom Baker was the Doctor longer than any other actor. What you may not know is that Baker has often said that he tried to quit the show nearly ever year. Rumors have persisted for years about behind-the-scenes conflict, and in the 501st issue of the official Doctor Who Magazine, Baker spoke openly about his different experiences with producers over the years, stating how he "clicked" with Philip Hinchcliffe — "He was young, and handsome, and beautifully dressed, and witty, and not long down from Cambridge. We got on like a house on fire" — but that he started having more a more difficult time during the era when Graham Williams took over as Doctor Who's producer.
Baker acknowledges that he deserves some of the blame for these problems with Williams, admitting that he wasn't always easy work with: "By then, I was rather proprietorial." These feelings, though, came from a deep love for Doctor Who. "I cared so much about the show," Baker explains, "about making it the best it could be. Although I used to affect that I didn't do very much work, in fact I thought about the part all the time, marking up the scripts."
It was the next producer, though, who Baker truly struggled with.
Tom Baker left Doctor Who ... for a while, anyway
There was a more significant turning point during Tom Baker's third and final era of Doctor Who, which was produced by John Nathan-Turner. As Baker explained in a 2018 interview, recorded by Digital Spy, he and Nathan-Turner had some major disputes. "I didn't like his approach to anything very much," Baker said. "His approach as a producer, to the scripts and to my performance...he managed somehow — how terrible — to diminish me. He made assumptions about how I should do things, or what lines meant, or how it should be shot, which diminished me, and I found that unbearable."
One unique point of contention occurred when Baker's costume was changed, with question marks being added to the lapels — a touch that Baker found, in his words, "insufferably vulgar and cheap."
In retrospect, Baker thinks he handled this the wrong way, and he's humble about it. Others seem to agree, though: Lalla Ward, who both played Romana on Doctor Who and was briefly married to Tom Baker, famously quipped that her favorite monster on the program was Tom Baker, according to the BBC. Surprisingly, Baker believes that his conflicts with Nathan-Turner (and subsequent departure) actually led to a better TV program, stating the producer "nudged me towards the realization it [Doctor Who] had run its course and I should go somewhere else. I think, in a way, when I said when I wanted to go, he was relieved, that he wouldn't have to have that fight."
Baker left the show, and didn't return until 1993's Dimensions in Time. Although, in a rather happy ending, Tom Baker now regularly reprises his role of the Fourth Doctor through audiobook dramas produced by Big Finish.