Scottish kings came to favor the site, and it was James IV who decided to transform the simple lodgings into a true royal palace.
Many kings came to the place and loved it, but none thought it worthy enough to transform due to how far it was.