 Williams was known by her nickname "Midge" to distinguish her from her mother, also named Virginia Louise.Her father was John Williams.Born in Oregon, she grew up in the African American agricultural community of Allensworth, California, United States, in Tulare County.