 Bailly's great-grandfather Nicholas Antoine Coulon, Sieur de Villiers was a trader and Army officer who was killed by Fox Indians on the shore of Green Bay, Wisconsin in 1733.His uncle, Father Charles-Francois Bailly de Messein, spent more than twenty years as a Catholic missionary to the Mi'kmaq Indians of Nova Scotia, and was appointed coadjutor Bishop of Quebec in 1788.Following the family interest in the fur trade, Joseph Bailly received an advanced education in Montreal, and served a clerkship with the North West Company.