Is it legal to use Web crawler for data gathering?
<p>I live in Uk, I am programmer and I have built a personal system that goes to some foreign housing web sites and extracts data from adverts going add by add (using crawler/scraper/web spider), I then process this data and do some analysis to find me a 'good deals'. Since system has already served it purpose for myself I was thinking of making this analytical data public. Basically making a web site and continuously posting results there. In my understanding data that I am gathering is publicly available so I can use that data in whichever way I want. Another is that there is no way to stop me or claim damages as they would be impossible to estimate.</p>

<p><strong>What are legal implications of using Web crawler for data gathering?</strong></p>

<p>You would need to see a lawyer for a definitive answer - generally, if it is on the web and not protected by action requiring acknowledgement of terms of service/use it is "fair game".  A greyer area would be if your bot follows the "Robots Exclusion standard" (ie honors the contents of robots.txt file)</p>

<p>Relatedly it may also depend on the amount of data you are spidering - if you spider to quickly it could be construed as a DoS attack, and get you in to hot water (or used as an excuse to get you in to hot water).</p>

<p>In most places you can't copyright a fact - so you are OK on that score - however if you present a layout similar to the original site you take the data from (though it sounds like thats not the case here), you could be in trouble for copyright infringement.</p>

internet business international data-ownership data-storage