 Miller was raised in the East End of Glasgow, attended St Mungo's Academy and grew up supporting Celtic; his father (also Joseph) had been a professional footballer with Hamilton Academical and Swindon Town.He was a 12-year-old playing for St Marys Boys Club in the Calton (the same team which produced Tommy Burns) when Alex Ferguson, the manager of Aberdeen, was alerted to his talents through Miller's uncle who had previously worked as a scout for Ferguson.He signed provisionally for Aberdeen while continuing to play in Glasgow for Celtic Boys Club, a team with close ties to the professional club and coached by legendary player Jimmy Johnstone, but not formally connected to them.