 Born in Denver, Colorado, Dillon was one of six children of Judge Henry Clay Dillon.She was the younger sister of opera singer Enrica Clay Dillon.Dillon's father became a district attorney in Los Angeles while her mother was a prominent socialite.