 Born in Vancouver, British Columbia, Elson was the son of Robert T. Elson, a newspaper reporter who became an editor at "Time" and "Life".He attended St. Anselm's Abbey School (then known as the Priory School) in Washington, D.C. and received an undergraduate degree from Notre Dame in 1953.He also received a master's degree in English from Columbia University in 1954.