 He claimed to have had an affair with his sister-in-law, the poet Anne Wharton, in the early 1680s.He never married, but he was persuaded by Parish that Hezekiah Knowles, the son of an associate of hers, was his illegitimate son.Wharton's mental instability seems to have gone unnoticed outside his family circle, but he was out of favour under James II for his pronounced Whiggery, despite making representations to his consort, Mary of Modena (and fantasizing about having an affair with her).