 The politician and writer John Penn built the adjacent Pennsylvania Castle, a Gothic Revival mansion overlooking Church Ope Cove, between 1797 and 1800.Penn's new estate encompassed both Rufus Castle and that of the former parish church of St Andrews.At this time Penn made alterations to Rufus Castle to transform it into a picturesque folly.