Oliver Hudson Says He's Now 'Keeping in Touch' with His Estranged Father Bill Hudson

After having a contentious relationship with his famous father, Oliver Hudson says he and his dad Bill Hudson are finally reconnecting.

“Well, we’re trying right now,” Oliver, 41, said on Larry King Now. “We’ve shot some texts back and forth.”

The Splitting Up Together star attributes the reconciliation to a controversial Instagram post he made in 2015. “There was an incident that happened via my Instagram,” he explained.

“I had posted something that was darkly comedic and it blew up into something, and actually it helped us with our relationship. So now we’re kind of keeping in touch,” he added.

Back in 2015, Oliver got some heat forposting an old photo of himself with his sister Kate Hudson, 38, and Bill, 68, on Father’s Day with the caption “Happy Abandonment Day…@katehudson.”

“I have a very weird sense of humor I guess,” he previously told PEOPLE of the incident. “My Instagram account is strange. It’s a great place for me to express my insanity.”

When they first spoke to one another after Oliver posted the image, the actor told Larry King their relationship was “contentious.”

“He did an interview and talked s—,” the actor said of his dad. “He felt hurt by it. I called him and we got on the phone and talked. It was really productive. Then we saw each other, had breakfast and it was the first time I had seen him in 12 years and it was really amazing to see him. We talked it out and we had a really great three-hour conversation.”

• Want to keep up on the latest from PEOPLE? Sign up for our daily newsletter to get our best stories of the day delivered straight to your inbox.

He also opened up about his relationship with his sister, which he said was much improved since their childhood. “We have an amazing relationship, my sister and I,” he said.

“Now it changed, because when we were kids I couldn’t stand her,” he added. “She came out of the womb acting and it bothered me. I couldn’t stand it. Please stop saying, ‘Look at me.’ And now I can honestly say she’s one of my best friends. Kate and I get along very well.”

Oliver also said he really gets along well with Kate’s boyfriend, Danny Fujikawa. “She’s got an amazing boyfriend, and she’s content,” he said. “We just have the greatest time together. And our kids are best, best friends. Strong family unit for sure.”

He has three children with his wife Erinn Bartlett: Wilder, 10, Rio, 4, and Bodhi, 8. Kate has two kids: Ryder Robinson, 14, with Chris Robinson, and Bingham Bellamy, 6, with Matt Bellamy.

Oliver, who is close with his mother Goldie Hawn’s longtime beau Kurt Russell, said that it wasn’t always easy growing up with famous parents. “The toughest part, especially as a young kid, is people wanting a piece of your parents,” he explained.

“We’re not cognitive enough of it at that age to understand what it is, so all we — or I — saw was these fans taking my mother away, meaning taking her time and attention away from me,” he continued. “It bothered me. It really, really bothered me. Now it’s a different story. But as a kid, I remember feeling anger and frustration that these people were infringing on our lives. So that was tough.”

Now, he’s embracing the family business, and says the whole family wants “to do something together.” He even told King that having some family members guest star on Splitting Up Together isn’t out of the question.