 Fisher's mother was the daughter of a blacksmith and worked as a domestic servant.On his father's side, he was descended from a long line of Ayrshire coalminers.According to family tradition, his paternal grandfather was persecuted for his involvement in the fledgling union movement, and on one occasion was left homeless with five young children.