 Fisher was born on 29 August 1862 in Crosshouse, a mining village west of Kilmarnock, Ayrshire, Scotland.He was the second of eight children born to Jane (née Garven or Garvin) and Robert Fisher; he had one older brother, four younger brothers, and two younger sisters.His younger sister died at the age of 10 in 1879, the only one of the siblings not to live to adulthood.