 Lord Penrhyn married, firstly, Juliana Isabella Mary, daughter of George Hay Dawkins-Pennant, in 1833.They had two sons and three daughters.After her death in 1842 he married, secondly, Maria Louisa, daughter of Henry FitzRoy, 5th Duke of Grafton, in 1846.