 Noel was the second son of the Reverend Baptist Wriothesley Noel and Jane Noel (née Baillie).His father was the tenth son of Sir Gerard Noel and Diana, Baroness Barham and brother of Charles Noel, 3rd Baron Barham (later 1st Earl of Gainsborough).Noel married three times and was widowed twice