 Toler attended the University of San Francisco and was a linebacker.Despite his physical gifts and strapping size, Toler never even played a down of prep football at Manassas High School in his native Memphis, but quickly emerged as a dominant force once recruited to the Bay Area.While at City College of San Francisco, Toler teamed with future Dons teammate and NFL Hall-of-Famer Ollie Matson to lead the school to the “Mythical” Junior College National Championship in 1948.