 The estate is the location of the Aboriginal settlement of Elizabeth Town, established by Governor Macquarie and under the leadership of Bungaree and is a significant Aboriginal-European contact site.Physical evidence of this is extant in the form of a midden behind blocks of flats between Onslow Avenue and Billyard Avenue.Elizabeth Bay House has representative associations with John Verge, the most fashionable architect in NSW during the 1830s and the artist Conrad Martens who executed views of the house and other commissions for members of the extended Macleay family.