 Downes was born in West Roxbury, Massachusetts, the son of Olin Downes.He was educated at Columbia College, graduating in 1933, and the Manhattan School of Music and received a Ph.D. in musicology from Harvard University in 1958.Downes served in the Army during World War II.