Jesse James is a scumbag. That’s the consensus that was reached after the 42-year-old with the imposing tattoos and greased-back hair cheated on his loving wife, Sandra Bullock, with an inked-up vixen harboring a predilection for all things Nazi. To make matters worse, the news broke just days after Bullock took home the Best Actress Oscar for her riveting performance in The Blind Side. James retreated to a rehab facility in Arizona for a month, before moving from his home turf of California to Austin, Texas—a short drive from Bullock’s home there—in an effort to win her back.

But why did James cheat on one of the most cherished actresses in Hollywood?

In his memoir, released May 3, James recounts his rise from childhood-abuse victim living in a broken home to building the motorcycle-customizing brand West Coast Choppers from the ground up to reality TV stardom with Discovery Channel’s Monster Garage. The father of three also opens up about his three failed marriages—to Karla, a dancer, porn star Janine Lindemulder, and finally, A-list actress Sandra Bullock.

James sat down with The Daily Beast for an exclusive, tell-all interview where he opens up about his abuse, why he cheated on Sandra Bullock, his Nazi costume controversy, and much more.

What compelled you to write your memoir now?

I’ve had offers for a decade to write my memoir, but I thought I was too young. I feel like my life has changed a lot in the last year, and it’s my way of putting a period on everything and saying, “That part of my life is over now. If you want to read about it, read about it. Me, I’m moving forward. I’m trying to not live in the past.”

The first portion of this book, strangely enough, really reads like The Blind Side. You’re a high school football star, and you move out of your crazy home life to live with another family for a year.

Yeah, it’s very similar. I think I kind of associate with Michael Oher to that respect. I’m not a 300-pound black dude, but I lived in poverty and a not too ideal upbringing.

It seems like the abuse you suffered at the hands of your father really haunted you, and governed many of the decisions you made later on in life.

I never wanted to be one of those dudes that used his childhood as a crutch, so I kind of forgot most of it. But writing the book made me remember stuff I hadn’t thought about since I was 8 years old. What happens to your kids definitely affects them down the road. I think rehab made me have an elevated sense of honesty with myself, cause the only person you’re bullshitting there is yourself. I would talk about my dad breaking my arm when I was 7 and I noticed I’d laugh, but that’s like a coping mechanism. Also, spurred on by the whole scandal that’s happened, so many people claim to know what I’m like. “Oh, he’s a Nazi! He’s this! He’s that!” You want to know what I’m like? Here it is.

Are you in contact with your father still?

No. Me and my father don’t talk. It’s unfortunate and I love him, but this goes back 25 years when we kind of parted ways. I think the stuff that happened with my brother dying and my father never calling him or anything like that made me distance myself. But maybe it’s salvageable. I don’t know.

“I think it was several factors: ego, loneliness, distrust, which probably led to resentment.”

Early in life, you had a lot of anger from your family situation and you seemed to release that on the football field, then later, you seemed to use work as your emotional outlet.

I had to have these things to make me feel complete, or happy, or loved. If I built these big, shiny bikes, people would say, “Oh, Jesse is so great!” But not without that bike in front of me with a big jukebox to distract me from being a beat-up little kid.

There’s a metaphor here: you had broken relationships at home, and you became a welder.

I never thought about that! For me, welding is the most soulful thing that I do. And it’s mood-reflective, so if I do a weld in a bad mood, it looks terrible. But I never really thought about the connection between fusion and broken home.

Your first girlfriend cheated on you—twice—and your stepmothers verbally abused you. How did these relationships affect you later on in life? Did you have trust issues?

I think I definitely had trust issues, and that carried on later in relationships where I didn’t trust anyone. They could treat me like gold and tell me they loved me, but I didn’t believe it. And it wasn’t them, it was all me. I’ve only gained that trust lately; that I need to blindly trust and not associate the current with the past.

In the book, you say you had trouble trusting Sandra Bullock.

I thought of our relationship as small. I didn’t see her as “Sandra Bullock,” she was just Sandy. I knew that was a huge part of her life, but that’s not why I wanted to be with her. I’m sure subconsciously some part of my ego thought that would be cool, but when I thought about her, I thought about the desire to be normal. I think I’ve always longed for a family and the normalcy that I’ve never experienced.

But it struck me as odd that you had trust issues with Bullock even after she stood by you during your child custody dispute with your ex-wife, Janine Lindemulder.

[Sandra] was there and gave me moral support, but as far as sitting in court twice a month for four years, that was all me. But the way Sandy was with Sunny was amazing and Sunny still loves her and considers her mom. It just sucks that because of the divorce, things have played out the way it has, because I would have no problem having her maintain a relationship with Sunny. There’s a big hole there in Sunny’s life without Sandy there.

How have your kids taken the divorce and subsequent media shitstorm?

They’ve taken it a lot better than me. They’re clear-headed; see it for what it is. They would get frustrated with the paparazzi and be like, “Why don’t you ram them!” We tried to make the best out of every situation and laugh at the way they looked or something. But it got to a point where we said: “We don’t need this in our lives anymore. Let’s move.”

Between moving to Austin, right by Sandra, and going to rehab, it seemed like you went to some lengths to try and patch things up with her.

For a while I tried to do everything I could, but I had to come to the realization that thing was broken beyond repair, and trying to fix it or make it better is something I’ve always done, cause I’m a welder and mechanic, but it was just my ego I was trying to fix. I’m remorseful that I hurt her so bad, but I just needed to accept things the way they were and move on. I just had to let go, forgive myself, and move on with my life.

Have you and Sandra had a sit-down and patched things up?

Oh yeah, we’ve talked quite a bit. I think we’re both feeling the same thing. We both realized it was over. I know she forgives me and I’ve apologized to her and tried to do everything possible to convey to her how sorry I am that things happened the way they did, and that I never wanted to hurt her.

Why did you cheat on her?

I think it was several factors: ego, loneliness, distrust, which probably led to resentment. When you have a certain amount of shame and guilt in your life that spurs from other stuff, you can manufacture resentment in any situation; because I’m not getting the attention I need or the love that I need, when all I had to do was ask for it and she would give it to me. But just because I didn’t get it when I didn’t ask for it, I was like, “Oh, she doesn’t love me.”

Was she too nice to you, or for you?

I don’t know. It’s easy for me to look back now and say, “Here’s the list of problems with her and problems with me.” But when you’re in the middle of it, you’re in the middle of it. Without the tools I needed to love myself, I could never love anyone. It’s daily maintenance to maintain a good relationship with someone. You really got to take care of each other. I didn’t have the ability or the knowledge to know that back then.

One of the things that seemed to drive you into the arms of Sandra was your hellish relationship with Janine Lindemulder.

Calm, stable, level-headed, good conversation, has her own job. I was like, “Wow, that’s what I want.” I didn’t necessary pick [Sandra] because we were a perfect match. I picked her because I thought that’s what I needed. But Karla, Janine, Sandy—they weren’t the problem. I was the problem.

It seems like you have a type, and you tried something different—a good girl—and it didn’t work out.

Well, with Janine, I hadn’t been to a strip club since 2001, and I really tried to turn things around from that whole lifestyle. It ended and blew up worse than ever, but I have a strong desire to constantly change and make myself better, and I really believed I was doing that with Sandy. I feel like I was a little bit persecuted for something that happens to everyone. I think people see stuff on E! and think, “Oh, she’s so beautiful! She made that one movie! I know she’s a great person!” For me, it’s reversed. I build the best motorcycles in the world, but I’m an asshole. Of course they’re going to vilify me, but they also have a desire to drag Sandy down and bring her to their level. It’s a weird commentary on our society, that people have the need to drag others through the mud to make themselves feel better.

Was the backlash amplified because of your and Sandy’s contrasting physical appearances?

People never really fully accepted us together. I just always heard comments for years like, “I don’t know what she sees in him. He’s a grease monkey.” It’s kind of sad, because I think there’s a certain amount of discrimination against working-class people in this country. I’m a working-class guy and have always worked a blue-collar job, and that doesn’t fit with people’s idea of stardom, so he’s not cool enough to be with Sandra Bullock.

You say in the book you felt “trapped” being with Bullock and you needed freedom.

Mentally, I felt like I had to conduct myself a certain way. I’m a beer drinker/biker/don’t have a problem punching someone, but all that ended. I couldn’t do anything to bring any kind of embarrassment. It was a very controlled way of living, and I don’t think it was very healthy. It’s not like she made me that way, I made myself that way because I felt like I needed to do that to fit into the lifestyle I stepped into.

Do you think a lot of the flak you got from the media was because Michelle “Bombshell” McGee, your mistress, was the polar opposite of Sandra looks-wise.

Oh yeah, I think it’s looks and people judging a book by its cover; the way I looked, the way [Michelle] looked. All of a sudden, my life became like a Roger Corman movie: Jesse James and the She-Wolf of the SS. If I had cheated on her with a supermodel, I imagine things would have been different. But what the public perceives as bad, visually, is bad on every level.

Why did you go for girls that were the polar opposite of Sandra?

There was no rhyme or reason why I picked her. It wasn’t really something I obsessed about or thought about. It could have been anyone. Sandy is a one of a kind woman. Any woman seems different from Sandra.

And you also got a lot of flack for those pictures of you posing as a Nazi.

You know, people were out for blood. People love Sandy because she’s such an inspiration to people, and with the way I look and the way the woman I cheated on her with looked, they were like, “How could you do this?” And Us Magazine gave them that photo, and it looked horrible.

Prince Harry had the same thing happen to him in the U.K.

He’s a prince though. I’m a “bad dude.” He can brush it off as just wearing a costume or something, and I was just wearing a hat and making a funny joke. I have so many friends of color—black friends, Mexican friends. They all called me and are like, “This is so fucked up. What are you going to do?” And you can’t do anything. People see that and really think I’m a Nazi. Keith Moon of The Who showed up to press conferences in an SS outfit. I didn’t think he was a Nazi. Lemmy from Motorhead has a huge WWII SS uniform collection. But none of those dudes cheated on their wife, who was an Oscar winner.

And in the book you say Robert Downey Jr. reached out to you.

He was really cool and shot me a text. Being through the stuff that he’s been through, it was really valid what he said. There wasn’t too much encouragement coming from many places in those days.

He sent me a text that said, “This is the most glorious shitstorm ever! Don’t worry dude, you’re going to be just fine.” Somehow, I believed him.

Now, you’re engaged to Kat Von D, and the public was upset that you seemed to move on so quickly.

We were friends for eight years before. We just had mutual respect for each other and talked from time to time. We weren’t “pals.” I could wait five years and people would still think it was too soon. Nobody is inside of mine and Kat’s lives, so everybody will just speculate. They don’t know that we are methodical and taking it slow, they just assume they know what’s going on. I married Sandy six months after meeting her, so the old me would have probably already been married. But she reached out to me when things were at their worst, just as a friend, and we agreed to just meet up and have dinner. I really enjoyed talking to her and there was just a spark. We hung out more and more and fell in love. And, as a changed person and seeing things differently, seeing qualities I like that I wasn’t able to see before, I see her and can’t spot any flaws in her.

But why get married? You’ve been in three marriages before.

I mean, the way I am now, I don’t think those earlier ones really counted. Karla is still a big part of mine and the kid’s lives, but any of those marriages could have probably worked—except for Janine—if I was in the mental place I am now. They weren’t the problem, it was me. And I don’t need to marry Kat to love her forever.

It must be difficult to not see Louis after going through a four-year process to try and adopt him with Sandra.

Sometimes I see tabloid stuff online and it tugs at my heartstrings. I’ve cried about that probably more than I’ve cried about anything in my life. But I know Sandy’s a good mom, and he’ll be treated like a king. I just have to find solace in that. I still miss him, though. I just need to focus on the kids I have. I’m pretty lucky. My kids are rad. They look the same, like Village of the Damned—blond-haired, blue-eyed demon kids.

Do you regret saying Kat was better in bed than Sandra on Howard Stern?

That was just him being Howard. It was a bad answer to a sarcastic question, and it’s not what I meant. We were just screwing around. I regret saying what I said, but if you listen to the whole thing it wasn’t supposed to be taken seriously. It’s Howard.

Do you think you’ll ever have a relationship with Sandra and Louis, or is that door closed?

I have no idea. If that opportunity ever arises, I’m a Dad first and foremost and I’d take that opportunity seriously and be a great Dad to him.

Marlow Stern works for The Daily Beast and has a master's from the Columbia University Graduate School of Journalism. He has served in the editorial department of Blender magazine, as an editor at Amplifier magazine, and, since 2007, editor of Manhattan Movie Magazine.