Why does archive.org block access of old domain URLs that have been covered with robots.txt after the old site (not protected by robots.txt) closes down?
When I input site plus the domain, some pages are not seen, Google says it is due to the robots.txt. But why?