 Elgart was born in 1922 in New London, Connecticut, four years younger than his brother, Les, and grew up in Pompton Lakes, New Jersey.Their mother was a concert pianist; their father also played piano, though not professionally.Larry and Les both attended Pompton Lakes High School.