Prevent robots from indexing a restricted access sub domain
<p>I have a sub domain setup for which i return a 403 for all but one IP. 
I also want to avoid the site being indexed by search engines, which is why I added a robots.txt to the root of my sub domain.</p>

<p>However, since I return a 403 on every request to that subdomain, the crawler will also receive a 403 when they request the robots.txt file.</p>

<p>According to <a href="https://developers.google.com/webmasters/control-crawl-index/docs/faq?hl=en#h1c" rel="nofollow">google</a>, if a robots,txt returns a 403, it will still try and crawl the site.</p>

<p>Is there anyway around this? Keen to hear your thoughts.</p>