 Fisher was born in Crosshouse, Ayrshire, Scotland.He left school at a young age to work in the nearby coal mines, becoming secretary of the local branch of the Ayrshire Miners' Union at the age of 17.Fisher emigrated to Australia in 1885, where he continued his involvement with trade unionism.