Is web scraping legal without authentification?
<p>I am currently learning how to web scrape the web and I encountered a major problem.</p>

<p>It seems that every website has some part of its terms of use talking about web scraping being strictly prohibited.</p>

<p><strong>My question is:</strong></p>

<p>Let's say that I scrape a website like Glassdoor or Indeed which has all the public information, without logging into an account. By doing that, I technically do not break the terms of use because I never agreed to respect them in the first place, right? Is it legal? Am I right or wrong in that instance?</p>

<p>Thank you.</p>

<p>Its etiquette to abide by a websites <code>robots.txt</code> file, if that file disallows scraping of certain sites, but whether scraping breaches anything other than a boilerplate terms of use is very much undecided in law right now (you don't state a jurisdiction, so I will assume the US) in that various arguments have been tried and both succeeded and failed depending on the individual cases merits.</p>

<p>There is currently a lawsuit winding its way through the US courts specifically around scraping of public information - this <a href="https://medium.com/@chris_70736/hiq-v-linkedin-and-the-legality-of-web-scraping-e80b9ab06f1d" rel="nofollow noreferrer">particular lawsuit is between a company called HiQ Labs and LinkedIn</a>.</p>

<p>Interestingly enough however, in 2017, HiQ Labs was granted an injunction against LinkedIn to prevent them from blocking scraping by HiQ Labs while the case proceeded.  </p>

<p>So, unfortunately, the answer is very much "watch this space".</p>

terms-of-service legal-terms website