Johnson was born in Nashville, Tennessee to Grace and Bishop Joseph A. Johnson Jr. He attended Fisk University for his undergraduate studies, where he studied physics and graduated summa cum laude. He moved to Yale University for his graduate studies, where he earned a master's in 1961 and PhD in 1965. Johnson was the second African American to obtain a Ph.D. in physics from Yale. His research considered plasma dynamics and the production of pions in antiproton-proton collisions. He worked under the supervision of Henry Kraybill. After graduating, Johnson worked at the Sikorsky Aircraft company and Bell Labs.