 On 6 December 1766, he married Penelope Pitt, daughter of George Pitt, 1st Baron Rivers.Her wanton intrigue with Vittorio Amadeo, Count Alfieri, provoked a duel between her husband and her lover in Green Park  on 7 May 1771, and Ligonier was able to obtain a divorce by Act of Parliament on 7 November 1771.He married Lady Mary Henley, daughter of Robert Henley, 1st Earl of Northington, on 14 December 1773.