 Whaley was born in Utica, New York on May 6, 1821.He married Louisa Mary Perdue (1828-1908), who survived him.Their sons included Charles Monroe Whaley (1850-1918), Warren Clayton Whaley (1852-1931), Harlan L. Whaley (1854-1936), Fulton Morse Whaley (1856-1907), Carlisle Landers Whaley (1859-1944) and James Whaley (b.