 He was a son of David Mannes and Clara (Damrosch) Mannes, and a brother of writer Marya Mannes.His parents founded the Mannes College of Music in New York.His maternal grandfather was conductor Leopold Damrosch, and his maternal uncles were conductors Walter Damrosch and Frank Damrosch.