Bacher was born in Loudonville, Ohio, on August 31, 1905, the only child of Harry and Byrl Fox Bacher. In 1908, the family moved to Ann Arbor, Michigan, where Harry worked as a banker and Byrl as a voice teacher on the University of Michigan faculty. Bacher attended the W. S. Perry School, and later Ann Arbor High School. While there he met Professor Harrison M. Randall, the head of the physics department at the University of Michigan, who encouraged him to study physics.