{"action":"created","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/168","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/168/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/168/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/168/events","html_url":"https://github.com/Norconex/collector-http/issues/168","id":111579498,"number":168,"title":"referenceFilters and robotsTxt: robots.txt being checked for \"external\" sites","user":{"login":"jetnet","id":15138506,"avatar_url":"https://avatars.githubusercontent.com/u/15138506?v=3","gravatar_id":"","url":"https://api.github.com/users/jetnet","html_url":"https://github.com/jetnet","followers_url":"https://api.github.com/users/jetnet/followers","following_url":"https://api.github.com/users/jetnet/following{/other_user}","gists_url":"https://api.github.com/users/jetnet/gists{/gist_id}","starred_url":"https://api.github.com/users/jetnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jetnet/subscriptions","organizations_url":"https://api.github.com/users/jetnet/orgs","repos_url":"https://api.github.com/users/jetnet/repos","events_url":"https://api.github.com/users/jetnet/events{/privacy}","received_events_url":"https://api.github.com/users/jetnet/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-15T09:35:02Z","updated_at":"2015-10-20T09:14:44Z","closed_at":"2015-10-19T13:13:27Z","body":"Dear norconex team,\r\n\r\ncould you please clarify, why the http collector does check the robots.txt of the \"remote\" (or \"external\") sites, although it is configured to \"stay-on-site\", e.g.\r\n```\r\n<startURLs stayOnDomain=\"true\" stayOnPort=\"true\" stayOnProtocol=\"true\">\r\n        <url>https://www.intranet.site.com/index.htm</url>\r\n</startURLs>\r\n```\r\n```\r\n<referenceFilters>\r\n        <filter \r\n            class=\"com.norconex.collector.core.filter.impl.ExtensionReferenceFilter\"\r\n            onMatch=\"exclude\" >jpg,gif,png,ico,css,js</filter>\r\n</referenceFilters>\r\n```\r\n```\r\n<robotsTxt ignore=\"false\" class=\"com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider\"/>\r\n```\r\n\r\nWhen the collector finds an external link (link to another site/domain), it tries to download the robots.txt from that site. Basically, it not that bad for the intranet sites, but if a link is pointed to an Internet site, the crawler gets run into a connection timeout.\r\n\r\nI tried to add an \"include\" regex (with the domain name) into `<referenceFilters>`, but it did not help.\r\nThe question, is it possible to disable the checking of \"robots.txt\" not belonging to the sites configured inside `<startURLs>`\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/comments/149488160","html_url":"https://github.com/Norconex/collector-http/issues/168#issuecomment-149488160","issue_url":"https://api.github.com/repos/Norconex/collector-http/issues/168","id":149488160,"user":{"login":"comschmid","id":15110299,"avatar_url":"https://avatars.githubusercontent.com/u/15110299?v=3","gravatar_id":"","url":"https://api.github.com/users/comschmid","html_url":"https://github.com/comschmid","followers_url":"https://api.github.com/users/comschmid/followers","following_url":"https://api.github.com/users/comschmid/following{/other_user}","gists_url":"https://api.github.com/users/comschmid/gists{/gist_id}","starred_url":"https://api.github.com/users/comschmid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/comschmid/subscriptions","organizations_url":"https://api.github.com/users/comschmid/orgs","repos_url":"https://api.github.com/users/comschmid/repos","events_url":"https://api.github.com/users/comschmid/events{/privacy}","received_events_url":"https://api.github.com/users/comschmid/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T09:14:44Z","updated_at":"2015-10-20T09:14:44Z","body":"Sorry to drop in, but even with the latest snapshot, I experience the same issue. \r\nI realized this behavior when I saw sporadic exceptions in the log (e.g. SSLHandshakeException) during a crawl with the message \"Not able to obtain robots.txt at: ...\" for external references that should not be crawled but only listed by the crawler event \"URLS_EXTRACTED\". \r\n\r\nThe setting of the `stayOnDomain` attribute doesn't change the behaviour and I also use a RegexReferenceFilter that is applied in code.\r\n\r\nUnfortunately, I can't see the LinkExtractorStage or URLCrawlScopeStrategy in the log for the fetched robot.txt (after changing the collectors to DEBUG), but I see it in the stack trace, e.g.:\r\n\r\n```\r\nWARN  [StandardRobotsTxtProvider] Not able to obtain robots.txt at: https://www.openstreetmap.ch/robots.txt\r\njavax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\r\n\tat sun.security.ssl.Alerts.getSSLException(Alerts.java:192)\r\n\tat sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)\r\n\tat sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)\r\n\tat sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273)\r\n\tat sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1446)\r\n\tat sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:209)\r\n\tat sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)\r\n\tat sun.security.ssl.Handshaker.process_record(Handshaker.java:837)\r\n\tat sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)\r\n\tat sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)\r\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)\r\n\tat sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)\r\n\tat org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)\r\n\tat org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)\r\n\tat org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)\r\n\tat org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)\r\n\tat org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)\r\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\r\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)\r\n\tat org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)\r\n\tat org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\r\n\tat org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)\r\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)\r\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)\r\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)\r\n\tat com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider.getRobotsTxt(StandardRobotsTxtProvider.java:80)\r\n\tat com.norconex.collector.http.pipeline.queue.HttpQueuePipelineContext.<init>(HttpQueuePipelineContext.java:41)\r\n\tat com.norconex.collector.http.pipeline.importer.LinkExtractorStage.queueURL(LinkExtractorStage.java:131)\r\n\tat com.norconex.collector.http.pipeline.importer.LinkExtractorStage.executeStage(LinkExtractorStage.java:89)\r\n\tat com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:31)\r\n\tat com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:24)\r\n\tat com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:90)\r\n\tat com.norconex.collector.http.crawler.HttpCrawler.executeImporterPipeline(HttpCrawler.java:285)\r\n\tat com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:475)\r\n\tat com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:376)\r\n\tat com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:703)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nMy config is located at [#163](https://github.com/Norconex/collector-http/issues/163#issuecomment-147985321). What am I missing?\r\nThanks for your help!"}}