 Lord Hylton married his second cousin, Lady Agnes Mary Byng, daughter of George Byng, 2nd Earl of Strafford, in 1858.Their divorce was a "Cause célèbre".There were children of this marriage, sons and a daughter, Agatha Eleanor Augusta Jolliffe, who married Ailwyn Fellowes MP.