 Frazier transferred to Ball State Teachers College—now known as Ball State University—Muncie, Indiana, where played for three seasons as a fullback and lettered in track.After graduating from Ball State in 1940 with a Bachelor of Science in physical education in 1940, he briefly taught and coached basketball in Guilford, Indiana before joining the United States Navy.During World War II, he was stationed at Norman Naval Air Station in Norman, Oklahoma.