 The Dyffryn Estate dates back to 640 A.D. when the Manor of Worlton (also known as Worleton), which included St Lythans and St Nicholas, was granted to Bishop Oudoceus of Llandaff.In the 16th century the Manor of Worlton was rented under copyhold by the Button family, who are believed to have first settled at the manor at Dog Hill in Dyffryn.The family's next residence, Columbar, was thought to be built on the location of Dyffryn Gardens.