 He was a son of Rev.Ephraim Peabody (1807–1856) and Mary Jane (née Derby) Peabody (1807–1892).His older sister, Ellen Derby Peabody, was the wife of Charles William Eliot, the 21st President of Harvard University.