 He lived with Maria Skerrett in both Richmond and Houghton Hall in Norfolk while Lady Walpole was still alive.She herself courted controversy when it was noted that the Walpoles' youngest son Horace, born 10 years after his siblings when the marriage was cool, did not share looks or character with any siblings or his father.Lady Walpole's lover at that time was reported to be Lord Hervey.