 Dillard was drafted by the San Diego Padres in the 13th round of the 1970 Major League Baseball draft, but did not sign as he had been awarded a full scholarship to the University of Mississippi.After two years of college, he was drafted by the Boston Red Sox in the second round of the 1972 MLB draft.Dillard began his professional career in 1972 playing for the Winston-Salem Red Sox of the Class A Carolina League, with whom he played the entirety of the 1973 season.