 Walker was born to an Irish Protestant father and a mother of South Carolina background in Charleston, South Carolina in 1839.In 1841, after his father died, Walker's family remained in Charleston where Walker grew up.