Can I ban robots?
<p>Say I have a website with content I want to protect.</p>

<p>Say I want people to access that site — but only manually, while sitting at their computer typing the keys — not by writing some computer program that somehow accesses the data remotely or automatically by robots.</p>

<p>Is that even possible to do? Say in the Terms and Conditions agreement maybe I include a clause that says something to the effect:</p>

<blockquote>
  <p>You agree not to use this site by copying anything automatically or by using any robots; You will only access the site manually.</p>
</blockquote>

<p>I seem to recall somewhere on this site somebody mentioned a case that ruled that once someone made the information available, they had no right to control <em>how</em> others would use or access the information? Is that right or am I imagining I read something here that never was?</p>

<p>Pragmatically, on your server side, there is nothing about an http request from a computer program that is any different than one typed manually, so if the "robot" doesn't intend to honor your demand, there's little or nothing you can do.</p>

<p>Most sites implement a "robots.txt" file in their root directory that defines the site's policy, and many/most of the web crawlers and spiders respect that policy...but it's an honor system.</p>

<p>You can use sophisticated metadata methods to block programmatic access to your site...refuse new connections less than five seconds apart, for example.</p>

<p>Edit:   There's no law that says you have to honor an http GET request. You can allow or refuse them at your most capricious whim.</p>

copyright internet