Stafford was born in England in 1920 but moved to South Africa at the age of 8. He attended the University of Cape Town where he came top of his class for Physics and subsequently did an MSc researching cosmic rays in 1941. He joined the South African Naval Forces as a Lieutenant Electrical Officer concerned with degaussing in the southern hemisphere.