{"action":"created","issue":{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/152","labels_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/152/labels{/name}","comments_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/152/comments","events_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/152/events","html_url":"https://github.com/DigitalPebble/storm-crawler/pull/152","id":91866119,"number":152,"title":"Adding support for [SOLR] Write Spout, StatusUpdaterBolt and Indexer","user":{"login":"jorgelbg","id":1291846,"avatar_url":"https://avatars.githubusercontent.com/u/1291846?v=3","gravatar_id":"","url":"https://api.github.com/users/jorgelbg","html_url":"https://github.com/jorgelbg","followers_url":"https://api.github.com/users/jorgelbg/followers","following_url":"https://api.github.com/users/jorgelbg/following{/other_user}","gists_url":"https://api.github.com/users/jorgelbg/gists{/gist_id}","starred_url":"https://api.github.com/users/jorgelbg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorgelbg/subscriptions","organizations_url":"https://api.github.com/users/jorgelbg/orgs","repos_url":"https://api.github.com/users/jorgelbg/repos","events_url":"https://api.github.com/users/jorgelbg/events{/privacy}","received_events_url":"https://api.github.com/users/jorgelbg/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-06-29T17:48:55Z","updated_at":"2015-07-02T17:41:33Z","closed_at":"2015-07-01T13:15:27Z","pull_request":{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/pulls/152","html_url":"https://github.com/DigitalPebble/storm-crawler/pull/152","diff_url":"https://github.com/DigitalPebble/storm-crawler/pull/152.diff","patch_url":"https://github.com/DigitalPebble/storm-crawler/pull/152.patch"},"body":"Solr resources for StormCrawler, this add support for:\r\n* IndexerBolt\r\n* MetricsConsumer\r\n* SolrSpout/StatusUpdaterBolt\r\n* SolrConnection\r\n* SolrCrawlTopology\r\n\r\nThe `SolrConnection` class have support for both the SolrCloud and a single Solr server by the ConcurrentUpdateSolrClient (CUSC). For using SolrCloud a `solr.TYPE.zkhost` needs to be specified instead of the `solr.TYPE.url`, also an additional parameter `solr.TYPE.collection` needs to be provided for the SolrCloud. I've detected a bug in the solr-solrj library regarding the use of the collection parameter in some methods (https://issues.apache.org/jira/browse/SOLR-7729), once this is fixed should simplify the code for the CUSC case, for now I recommend using the `url` parameter for the CUSC case to point directly to the solr collection that wants to be used(`external-conf.yaml`).\r\n\r\nThe `solr.metrics.commit.size` config option is basically a batch size to issue a hard commit against the server, this is not required (as is initialized in 250 by default) and could be leveraged by configuring autocommit in the server. \r\n\r\nThe `MetricsConsumer` has support for automatic Document Expiration (TTL), a couple of additional parameters are provided in this class: `solr.metrics.ttl` and `solr.metrics.ttl.field` for configuring the Date language expression in the first case and the target field in the later. Of course this also needs some configuration in the Solr side. The date expression used in `solr.metrics.ttl` is not validated in the code. \r\n\r\nFor `SolrCrawlTopology` to work at least an URL in the status collection is required to be present. "},"comment":{"url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/comments/118106449","html_url":"https://github.com/DigitalPebble/storm-crawler/pull/152#issuecomment-118106449","issue_url":"https://api.github.com/repos/DigitalPebble/storm-crawler/issues/152","id":118106449,"user":{"login":"jorgelbg","id":1291846,"avatar_url":"https://avatars.githubusercontent.com/u/1291846?v=3","gravatar_id":"","url":"https://api.github.com/users/jorgelbg","html_url":"https://github.com/jorgelbg","followers_url":"https://api.github.com/users/jorgelbg/followers","following_url":"https://api.github.com/users/jorgelbg/following{/other_user}","gists_url":"https://api.github.com/users/jorgelbg/gists{/gist_id}","starred_url":"https://api.github.com/users/jorgelbg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorgelbg/subscriptions","organizations_url":"https://api.github.com/users/jorgelbg/orgs","repos_url":"https://api.github.com/users/jorgelbg/repos","events_url":"https://api.github.com/users/jorgelbg/events{/privacy}","received_events_url":"https://api.github.com/users/jorgelbg/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T17:41:33Z","updated_at":"2015-07-02T17:41:33Z","body":":+1:  on the separated topology its more useful this way. I was thinking of adding a script that add some URLs, perhaps the same URLs as the RandomURLSpout for testing, but this is indeed more practical. \r\n\r\nDone, modifying the pom file :)\r\n\r\nI'm going to add the README.md file with the explanation. \r\n\r\nI think that using  _field collapsing_ can be a nice add, in this way we kind of guarantee _diversity_ in the seed links, in this case we should add a parameter for configuring the field and the amount of URLs per bucket too."}}