 Schwartz married twice.His first wife, advertising executive Clara Apfel, died in 1964; they had one son, Paul Waldo Schwartz, an art critic in Paris.In 1966, he married Anne Schnall, the former wife of investor, M. Elliot Schnall.