Aubrey Anderson-Emmons Joins 'Modern Family' as the New Lily

When viewers tune in for the new season of Modern Family, they’ll notice a new Lily in Cam and Mitchell’s house, a.k.a. 4-year-old actress Aubrey Anderson-Emmons.

Aubrey replaces Jaden and Ella Hiller, the identical twin girls who played Lily for the Emmy-winning sitcom‘s first two seasons. The reason for the re-casting, a show source tells PEOPLE, is because the Hiller twins didn’t enjoy acting. When they were required to be in scenes, they became agitated and unhappy.

In a nutshell, the girls just wanted to be kids, their mother, Michelle Hiller, tells PEOPLE.

“Ella and Jaden are enjoying new toddler adventures in swimming, dance, gymnastics, and rock climbing,” says Michelle. “They are happy with the freedom of being 3 [years old]!”

While Ella and Jaden are enjoying their early retirement, Aubrey is having a field day at her new job. The Santa Monica resident is comfortably adjusting to her new role as Cam and Mitchell’s adopted daughter, says the source.

Aubrey will also bring a new dimension to the character – lines. The previous actresses did not speak in their role, but the new Lily “has a lot of lines,” a show rep tells PEOPLE. “She’s adorable and fans are really going to like her.”

The new season of Modern Family kicks off Wednesday night on ABC.