 He resides with his wife Mary Sue Willie in Concord, Massachusetts.He has three children who have careers in government (James Theodore Willie), construction (Martin Charles Willie), and academic (Sarah Susannah Willie-LeBreton).Willie became the first African American professor at Syracuse University where he taught from 1950 to 1974.