 Little is known of Lathom's personal life.In 1797 he married Diana Ganning, daughter of a wealthy Norfolk lawyer, and the pair had four children, three of which survived, a baby boy dying in infancy.However, despite Lathom's burgeoning literary career and his growing family, some unknown cause led him to leave Norwich in 1810 and end his literary career.