Elia Kazan: biggest rat of the pack

One way and another, in the middle decades of the 20th century, Elia Kazan became a central figure in American culture. He offered himself as the archetypal despised immigrant who had made it in the new world. He had been born in Istanbul in 1909 and he arrived in New York in 1913 as the son of a rug merchant. He was smart, hugely ambitious, drawn to women, and an inspiring conspirator (and betrayer) with guys. He went to a good school, Williams College in Massachusetts, but felt looked down on by the upper-class white girls. So he slept with them, and he excelled at every competitive urging. In the early 30s he apprenticed to the Group Theater as an actor, a possible director and a handyman who could fix physical problems – chairs, plumbing, cars that wouldn't start. So they called him "Gadget", and the nickname "Gadge" stuck. He was a member of the Communist party for just over a year, but could not endure its discipline and secretive hierarchy. Gadge needed to run things.

So, in the 40s, he started to direct plays and discovered his own phenomenal, seething intimacy with actors, a way of getting to smothered psychological truths that had been buried in the gentility of previous theatre. He established himself with the first productions of Thornton Wilder's The Skin of Our Teeth (1942), Arthur Miller's All My Sons (1947), Tennessee Williams's A Streetcar Named Desire (1947) and Miller's Death of a Salesman (1949), with Lee J Cobb in the lead.

His influence went far beyond the credits. On Streetcar, for instance, he took the play (a metaphor for gayness) and made Stanley Kowalski as important but more vital on stage than Blanche DuBois (the central character on the page). In doing this, he boosted the career of one of his discoveries, Marlon Brando, and caused a theatrical sensation with a new level of sexuality, of hot, ugly life being lived, and a kind of acting that left audiences aware of feeling close to dangerous people. In the same year as Streetcar, Kazan helped found the Actors Studio, the centre for this new psychological realism in acting, which linked the teachings of the Russian director Konstantin Stanislavsky with carrying the hesitations of life into Hollywood close-ups.

Though Kazan was into movies as much as theatre, at first he was frustrated by the Californian art because he knew he wasn't getting his own passion and needs on to the screen in the way he had forced himself as an author on Streetcar. Those early films look decent but ordinary now – A Tree Grows in Brooklyn, The Sea of Grass, Boomerang, Gentleman's Agreement (which won the best picture Oscar), Pinky, Panic in the Streets, and the film of Streetcar, which was reduced by censorship's anxiety but remains the living record of how Brando felt as Stanley and won an Oscar for Vivien Leigh as Blanche.

Martin Scorsese loves those films, and he revered the impact Kazan had on American life. In the new 60-minute documentary A Letter to Elia, which he has made with Kent Jones, Scorsese assesses the example Kazan offered to him as a kid, mad on movies and yearning to become a film-maker. So it's a film that has to confront the famous crisis in Kazan's life. In 1952, as an ex-communist, desperate to make more personal movies, he came under pressure to testify at the House Un-American Activities Committee. This was a curse of the age, and the ruin of many careers. But Kazan mattered to HUAC because he was so prominent. The screws were tightened: he was told that if he did not name names he might not be allowed to make more films – for the pernicious compromise in what we now call the "Black List" was that the studio system bowed to the Committee's pressure and agreed to enforce the boycott.

So Kazan testified. If you read his 1988 autobiography, A Life – one of the great books written about American show business and directing – you will feel his agony, you will hear of the old friends who never spoke to him again, and you realise how Kazan was haunted by the incident as long as he lived. But if you look at the films that followed – On the Waterfront, East of Eden, Baby Doll, A Face in the Crowd, Wild River, Splendor in the Grass, America, America – it is hard to deny the argument that testifying and being rebuked matured Kazan's creative vision. The way he identified with Cal in East of Eden (the effective movie debut of James Dean) is what drives the emotionalism of that film. It is a defence of the spirit of Cain, and an unfair dismissal of the good brother "Abel".

For 10 years up to 1952, Kazan had had unmatched success. But he was the kind of guy who knows success can kill you – in America this is an essential paranoia. So in a way, he engineered his own moral downfall – where he became hated among his own people. And he thrived on the contempt. It's an open question whether he became a great artist or just the most natural, ruthless and successful director of actors. The Actors Studio style produced more than Brando, Dean, Karl Malden, Eli Wallach, Rod Steiger and Lee J Cobb. It is the basis for the work of Warren Beatty (he was in Splendor in the Grass), Robert De Niro, Al Pacino and Sean Penn. Mostly guys, you might note, though Jo Van Fleet, Eva Marie Saint and Ellen Burstyn are actresses within the Method tradition. The Actors Studio is also the tomato sauce in which the meat of The Godfather films is cooking.

Kazan died in 2003, but his importance has not faded yet. This is because of his films and the influence of psychological realism in acting (though that has faded recently, as camp undermines sincerity). It is also because of the emblematic role he filled in a national crisis. In 1955, in his novel The Deer Park, Norman Mailer had a character, a director named Charles Eitel, who owed a lot to Kazan. A Life may outlast many of the movies, for it is a portrait of burning ambition and a shameless dwelling on shame, arguing with itself over tactics, scalps and coups, and a long list of conquests. Scorsese's personal tribute is, in its way, just as naked – for here is a great director who longed to be one of his own gang of male studs who behave like brilliant actors. Kazan defined modern acting and helped turn it into a lifestyle for all of us.

But there is still a test of conscience at the heart of the story. If the crisis gets tough enough, would you betray old friends or behave with silent honour? The Black List seems like history, but how easily it could come again. The intimidating cries against being "un-American" are still heard in a land that has never been more insecure.

The crisis found its last act in 1999, when the Academy chose to give Kazan an honorary Oscar. This was hardly necessary: he had won twice as director, for Gentleman's Agreement and On the Waterfront (with three other nominations). There was picketing outside the awards theatre and many in the audience did not join the applause as the Oscar was presented by Scorsese and De Niro (who had acted in Kazan's The Last Tycoon, one of the director's poorest films).

Would Kazan give a word of apology? No one who knew his combative soul expected that. So he said thank you and defied his enemies. The real opportunity missed was that of the Academy not apologising on behalf of the film community for the cowardice that consented to the Black List. This could never have happened if Hollywood had stood by the Constitution and its own people. But it was a time of fear and strife, and 50 years from now, Elia Kazan's story may tell us more about his era than will the lives of presidents and civic leaders. He made himself and his story central, and played the lead to perfection. The classic American heroes can be scoundrels, too, for it is a nation where the outlaws often get to run the show.

A Letter to Elia screens on 28 May at Curzon Soho as part of the London International Documentary Festival. Details: lidf.co.uk