 John grew up near Quebec City on the family estate which was one of the properties his father, Henry Caldwell, had obtained from Governor James Murray by way of a 99-year lease.He studied to become a lawyer and was called to the bar in 1798.He never practiced law as he was administering his father's assets and ongoing business affairs.