 On 6 February 1760, following the death of Edward Drummond, sixth Jacobite-jurisdiction Duke of Perth, James' father became heir to the Earldom of Perth, which had been forfeit since 1716 owing to the attainder of James Drummond, 2nd Duke of Perth (The first Earl of Melfort was the younger son of James Drummond, 3rd Earl of Perth).His father, therefore, assumed the surname of Drummond and styled himself 10th Earl of Perth, and in 1776, following the death of Jean Drummond, Duchess of Perth in 1773, he took up residence at the Drummond estate of Stobhall in Perthshire.Upon his father's death on 18 July 1781, James succeeded to his father's claim to the Earldom of Perth (his elder brother Thomas, who moved to America, had died the previous November), but did not use the title.