 Malicious gossip that he was the Queen's lover was not taken seriously even at the time, and is entirely discounted by historians.It was his position as an extreme Tory, and his strong opposition to the Reform Act 1832 which made him unacceptable to the Government, and Lord Grey eventually insisted on his dismissal, much to the Queen's distress.Subsequent negotiations to reinstate him came to nothing.