 It ultimately ended up in the possession of the Earl of Hopetoun and from the eighteenth century onward remained unoccupied.During this and later periods large parts of the stonework were re-used in other structures, and the later courtyard buildings were almost razed to the ground, leaving only the tower and north courtyard wall remaining significantly above ground-floor level.It became Admiralty property in 1903 and as the result of land reclamation lost its waterfront position, becoming marooned within the dockyard.