 Brown is a Jewish American born in New York City to Gertrude Cohen and A. H. Brown.He had previously served in the Lyndon Johnson administration as Director of Defense Research and Engineering and Secretary of the Air Force.While Secretary of Defense, he insisted in laying the groundwork for the Camp David accords.