 He built a mansion called "Elvills", now known as Castle Hill, in Englefield Green, Egham, Surrey between 1758 and 1763.It was designed by Stiff Leadbetter.Sir John Elwill, 4th Baronet, died without male issue on 1 March 1778, and was interred at
Egham, Surrey.