 James Ogilvy was the son of James Ogilvy, 6th Earl of Findlater (ca.1714-1770), and Lady Mary Murray (1720–1795), daughter of John Murray, 1st Duke of Atholl.Ogilvy was born at Huntingtower Castle in 1750.