 His maternal grandfather, Cadwallader Colden, and his uncle, James DeLancey served as Colonial Governors of New York.He served as sheriff of Westchester County from 1769 to 1776 and as an officer in the militia.Because of his loyalist sympathies, he was forced to leave the area and went to New York City, where he and his uncle, Oliver De Lancey, raised a loyalist unit known as "De Lancey's Brigade", "De Lancey’s Cowboys", and "De Lancey's refugees".