 Downes was born in Birmingham, England in 1924, the son of a bank teller.He began to learn the piano and violin at age five and was also a choirboy at King Edward's School, learning the organ and becoming choir master when his voice broke at 13.He left school at the age of 14 to earn his living in a low-paid position at the City of Birmingham Gas Department.