 James married firstly, Susanna Strangways, daughter of leading civil war politician Colonel Sir Giles Strangways of Melbury, Dorset, a wealthy neighbour at Athelhampton.They had three daughters (Susan, Dorothy and Anne), and three sons (Robert, Giles and James) but the marriage was not a happy one.James was a gambler and a drinker, and Susanna and the children often retreated to her parents home at Melbury.