 His father Richard worked as a mechanic, and his mother Hilda in a department store.Downes boxed as a junior for the Fisher ABC.He moved with his parents to the United States in 1952, while still a teenager, to live with his trapeze artist sister Sylvia, who had lost an arm in a traffic accident, going on to serve in the US Marine Corps from 1954–56, being recruited after boxing against them for the YMCA.