getting Forbidden by robots.txt: scrapy
<p>while crawling website like <a href="https://www.netflix.com">https://www.netflix.com</a>, getting Forbidden by robots.txt: https://www.netflix.com/></p>

<p>ERROR: No response downloaded for: <a href="https://www.netflix.com/">https://www.netflix.com/</a></p>