 Fitzgerald was born free about 1843 in New Castle County, Delaware, to Thomas and Sarah (Burton) Fitzgerald.His father Thomas was mulatto or mixed race, of African and Irish ancestry, and manumitted from slavery, likely by a white father and master.His mother was white and of English ancestry; she decided to raise her sons proudly within the community of free people of color, rather than "passing" as white, although they were of majority European ancestry.