Prince Harry and Prince William's 'Awkward' Outing After Queen's Death: 'Both Couples Found It Hard'

Prince William and Prince Harry's show of unity outside Windsor Castle after Queen Elizabeth's death was a surprise for many — and it was not easy for the brothers.

"It was awkward. Both couples found it hard," an insider tells PEOPLE in this week's issue of William, his wife Kate Middleton, Harry and his wife Meghan Markle. "They were in a stoic spirit of just getting through it for the Queen."

The outing on September 10 came two days after William and Harry's grandmother Queen Elizabeth died at age 96 at Balmoral Castle in Scotland.

Along with Kate and Meghan, Prince William and Prince Harry stepped out together for a walkabout, greeting well-wishers and taking in the flowers and other memorial items left for the late monarch.

Kensington Palace said at the time that William invited Harry and Meghan to join him and Kate. Another source added that security was not expecting both couples.

Prince William thought the walkabout to greet the crowds "was an important show of unity at an incredibly difficult time for the family," a royal source previously told PEOPLE.

Added a palace insider, "It's such an extraordinary historical moment and also a deeply personal one for the family that you'd hope and think that all members of the family would unite and support [King Charles III] especially. And perhaps some of those wounds can be healed in the process."

For more on the Royals, listen below to our daily podcast PEOPLE Every Day.

The reunion among the foursome was poignant, as relations have been strained the Duke, 38, and Duchess, 41, of Sussex and members of the royal family in recent years.

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

They did privately introduce King Charles, 73, to his granddaughter Lilibet, 1, for the first time, which also gave him the chance to see his 3-year-old grandson Archie. (The family resides in California after Meghan and Harry stepped back as senior members of the royal family in 2020.)

"It was a fantastic visit. The prince was delighted to see his grandson and meet his granddaughter for the first time," a royal source said, noting that it was "wonderful" to have Harry and Meghan back in the United Kingdom.

Prince Harry also spoke about Archie and Lili spending time with Queen Elizabeth in a tribute to his grandmother following her death: ""Granny, while this final parting brings us great sadness, I am forever grateful for all of our first meetings — from my earliest childhood memories with you, to meeting you for the first time as my Commander-in-Chief, to the first moment you met my darling wife and hugged your beloved great-grandchildren."

RELATED VIDEO: Prince Harry Walks Beside Prince William During Queen Elizabeth II's Funeral

But at Queen Elizabeth's funeral on Monday, the rift was never fully out of mind.

"When families have a falling-out, you long for them to hug one another," royal biographer Penny Junor tells PEOPLE. "It broke my heart."

Those close to King Charles and Queen Camilla hope, for the sake of his new reign, if not salvaging ties, that William and Harry will now follow their grandmother's lead.

"You'd think that all members of the family would unite and support the King, especially," says a source close to King Charles. "Perhaps some wounds can be healed in the process." It would likely have been Elizabeth's dying wish. Adds the close source: "She knew that conflicts were a part of life, and she didn't hold grudges. Most of all, she wanted to see her family happy."

For more on the royal family following Queen Elizabeth's death and funeral, pick up the latest issue of PEOPLE, on newsstands Friday.