Elmer S. Imes was born in 1883 in Memphis, Tennessee to Elizabeth (née Wallace) and Benjamin A. Imes, both of whom were college educated. They had met as students at Oberlin College in Ohio. They married there in 1880. His father earned a divinity degree at Oberlin Theological Seminary in 1880. Benjamin was descended from free people of color, who had been established in south-central Pennsylvania by the time of the Revolution.