Many Conservative voters I met in Richmond did not like the tricksy way in which Goldsmith presented himself as an independent candidate in nothing more than name, openly admitting he was likely to rejoin the Conservatives if he won.