Elmer's mother Elizabeth was born into slavery; her family had moved to Oberlin when she was a child, after the American Civil War and emancipation. Imes had two younger brothers: Albert Lovejoy Imes and William Lloyd Imes. The latter became a minister and was later pastor of St. James Presbyterian Church in New York City. He held degrees from Fisk, Union Theological Seminary, and Columbia University.