 Bivins was born free on May 8, 1862, in Accomack County, Virginia; his parents Severn S. and Elizabeth Bivins were established as free people of color before the American Civil War and farmed on Virginia's Eastern Shore.Bivins grew up working with them, but decided to attend Hampton Institute in Virginia.In 1887 he joined the United States Army as a private.