 Born in Uriah, Alabama, Madison attended Escambia High School, graduating in 1949.He enrolled at Texas Tech University on an athletic scholarship, but left after his freshman year to join the United States Army.He returned to Texas Tech in 1952, starting four games in the 1952 college football season before an injury cut short his career.