 The Button family occupied the estate for a number of generations, producing Admiral Thomas Button who become a notable early explorer.The name of the Manor of Worlton was changed to the Manor of Dyffryn, St Nicholas in the 18th century when the Dyffryn Estate was sold to Thomas Pryce, who built the second building to be known as Dyffryn House, a Georgian manor, on the site in 1749.