 Williams' grandfather, Joshua Singleton, was the son of the black separatist leader Benjamin "Pap" Singleton.Joshua Singleton moved his family to Allensworth when the colony was founded in 1909, and the Singletons became prominent members of the Allensworth community.Virginia Louise Williams, Midge's mother, helped run Singleton's grocery store and helped obtain a Girl Scouts of the USA charter for the Allensworth settlement.