 Holwood House is on the site of an earlier building owned by William Pitt the Younger, and the grounds contain the remains of an Iron Age fort known as a "Caesar's Camp", which is a Scheduled Ancient Monument.Pitt is thought to have caused the Fort remains to be levelled in order to landscape the estate's gardens.The house was described in Thomas Wilson in his "Accurate Description of Bromley in Kent" of 1797 as " a small, neat, white building; it is more simple than elegant, and built on a rising ground, which commands one of the most fertile, variegated, and extensive inland prospects in the whole county".