 Milne was born in Buffalo, New York, but his family moved to Pasadena, California, while he was still young.While working as a messenger deliver boy for Western Union he saved his money and bought a motorcycle, converting it to a speedway bike.By the early 1930s, Milne had started to earn some decent money in racing.