 He was the son of Henry Scudder (1778–1863) and Elizabeth (née Hewlett) Scudder (1792–1870).His paternal grandparents were Henry Scudder (1743–1822), of Crab Meadow in Long Island, and Phebe Carll Scudder (1743–1821).His nephew, Townsend Scudder (July–1960), was a judge and also a U.S. Representative from New York.