 Cavendish married firstly Anne Pyne, the daughter of Henry Pyne and Anne Edgecumbe, on 9 January 1730.Together they had six children.He married secondly Catherine Prittie, daughter of Henry Prittie and Elizabeth Harrison, on 4 October 1748, and they had one son.