 Walker, who was of Loyalist ancestry, was a farmer's son in the Kingston peninsula area.His family had settled there in 1786, placing them in the first group of black settlers on the peninsula, which is upriver from Saint John.Walker grew up in a large family in the rural community of Kars, and attended the one-room schoolhouse there.