Lawrence P. Ward, vice president of student affairs at Babson, said in a letter to students that the men drove by Wellesley College’s Harambee House, which serves as the focal point for African-American students.