 After Philip's assassination, Otto, crowned Holy Roman Emperor in 1209, stayed at the fortress several times.Upon Otto's death at the nearby "Harzburg" in 1218, Harly Castle passed to his Welf heirs.At the 1290 Imperial diet in Erfurt, the Hildesheim prince-bishop Siegfried II of Querfurt accused Duke Henry I of Brunswick of using its favourable location for ambushes and highway robberies of bypassing merchants.