{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/filebeat/pulls/comments/39602809","id":39602809,"diff_hunk":"@@ -0,0 +1,100 @@\n+package crawler\n+\n+import (\n+\tcfg \"github.com/elastic/filebeat/config\"\n+\t\"github.com/elastic/filebeat/input\"\n+\t\"github.com/elastic/libbeat/logp\"\n+\t\"os\"\n+)\n+\n+/*\n+ The hierarchy for the crawler objects is explained as following\n+\n+ Crawler: Filebeat has one crawler. The crawler is the single point of control and stores the state. The state is written through the registrar\n+ Prospector: For every FileConfig the crawler starts a prospector\n+ Harvestor: For every file found inside the FileConfig, the Prospector starts a Harvestor\n+ \t\tThe harvester send their events to the spooler\n+ \t\tThe spooler sends the event to the publisher\n+ \t\tThe publisher writes the state down with the registrar\n+*/\n+\n+// Last reading state of the prospector\n+type Crawler struct {\n+\t// List of all files which were crawled with the state\n+\tFiles   map[string]*input.FileState\n+\tPersist chan *input.FileState\n+}\n+\n+func (crawler *Crawler) Start(files []cfg.FileConfig, persist map[string]*input.FileState, eventChan chan *input.FileEvent) {\n+\tpendingProspectorCnt := 0\n+\n+\t// Prospect the globs/paths given on the command line and launch harvesters\n+\tfor _, fileconfig := range files {\n+\n+\t\tlogp.Debug(\"prospector\", \"File Config:\", fileconfig)\n+\n+\t\tprospector := &Prospector{\n+\t\t\tFileConfig: fileconfig,\n+\t\t\tcrawler:    crawler,\n+\t\t}\n+\n+\t\tprospector.Init()\n+\n+\t\tgo prospector.Start(eventChan)\n+\t\tpendingProspectorCnt++\n+\t}\n+\n+\t// Now determine which states we need to persist by pulling the events from the prospectors\n+\t// When we hit a nil source a prospector had finished so we decrease the expected events\n+\tlogp.Debug(\"prospector\", \"Waiting for %d prospectors to initialise\", pendingProspectorCnt)\n+\n+\tfor event := range crawler.Persist {\n+\t\tif event.Source == nil {\n+\t\t\tpendingProspectorCnt--\n+\t\t\tif pendingProspectorCnt == 0 {\n+\t\t\t\tbreak\n+\t\t\t}\n+\t\t\tcontinue\n+\t\t}\n+\t\tpersist[*event.Source] = event\n+\t\tlogp.Debug(\"prospector\", \"Registrar will re-save state for %s\", *event.Source)\n+\t}\n+\n+\tlogp.Info(\"All prospectors initialised with %d states to persist\", len(persist))","path":"crawler/crawler.go","position":63,"original_position":63,"commit_id":"c6d4a6460b9035b17a84af9e978bda5133d0d954","original_commit_id":"17a10b4bc8a7883206939f8a6493232b64f232a0","user":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"body":"To be honest I still need to analyse the registrar part in detail. This \"docs\" are taken over from logstash-forwarder.","created_at":"2015-09-16T08:05:18Z","updated_at":"2015-09-16T08:05:18Z","html_url":"https://github.com/elastic/filebeat/pull/15#discussion_r39602809","pull_request_url":"https://api.github.com/repos/elastic/filebeat/pulls/15","_links":{"self":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/comments/39602809"},"html":{"href":"https://github.com/elastic/filebeat/pull/15#discussion_r39602809"},"pull_request":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/15"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/filebeat/pulls/15","id":44972803,"html_url":"https://github.com/elastic/filebeat/pull/15","diff_url":"https://github.com/elastic/filebeat/pull/15.diff","patch_url":"https://github.com/elastic/filebeat/pull/15.patch","issue_url":"https://api.github.com/repos/elastic/filebeat/issues/15","number":15,"state":"open","locked":false,"title":"Refactor Crawler","user":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"body":"The current implementation of Crawler, Prospector, Harvester passes around lots of state and is hard to change and test as it is tightly coupled. The goal of the refactoring is to take these blocks apart and split up in more independent blocks.\r\n\r\nThe changes made are mainly removing duplicated code and moving existing code around to simplify it.","created_at":"2015-09-14T19:53:37Z","updated_at":"2015-09-16T08:05:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"916c2ddd562d6f1a4fc5d54e353e3ff8b0a5c33c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/elastic/filebeat/pulls/15/commits","review_comments_url":"https://api.github.com/repos/elastic/filebeat/pulls/15/comments","review_comment_url":"https://api.github.com/repos/elastic/filebeat/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/filebeat/issues/15/comments","statuses_url":"https://api.github.com/repos/elastic/filebeat/statuses/c6d4a6460b9035b17a84af9e978bda5133d0d954","head":{"label":"ruflin:prospector-refactoring","ref":"prospector-refactoring","sha":"c6d4a6460b9035b17a84af9e978bda5133d0d954","user":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"repo":{"id":42054812,"name":"filebeat","full_name":"ruflin/filebeat","owner":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ruflin/filebeat","description":"Open Source File Harvester","fork":true,"url":"https://api.github.com/repos/ruflin/filebeat","forks_url":"https://api.github.com/repos/ruflin/filebeat/forks","keys_url":"https://api.github.com/repos/ruflin/filebeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ruflin/filebeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ruflin/filebeat/teams","hooks_url":"https://api.github.com/repos/ruflin/filebeat/hooks","issue_events_url":"https://api.github.com/repos/ruflin/filebeat/issues/events{/number}","events_url":"https://api.github.com/repos/ruflin/filebeat/events","assignees_url":"https://api.github.com/repos/ruflin/filebeat/assignees{/user}","branches_url":"https://api.github.com/repos/ruflin/filebeat/branches{/branch}","tags_url":"https://api.github.com/repos/ruflin/filebeat/tags","blobs_url":"https://api.github.com/repos/ruflin/filebeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ruflin/filebeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ruflin/filebeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/ruflin/filebeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ruflin/filebeat/statuses/{sha}","languages_url":"https://api.github.com/repos/ruflin/filebeat/languages","stargazers_url":"https://api.github.com/repos/ruflin/filebeat/stargazers","contributors_url":"https://api.github.com/repos/ruflin/filebeat/contributors","subscribers_url":"https://api.github.com/repos/ruflin/filebeat/subscribers","subscription_url":"https://api.github.com/repos/ruflin/filebeat/subscription","commits_url":"https://api.github.com/repos/ruflin/filebeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/ruflin/filebeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/ruflin/filebeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/ruflin/filebeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/ruflin/filebeat/contents/{+path}","compare_url":"https://api.github.com/repos/ruflin/filebeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ruflin/filebeat/merges","archive_url":"https://api.github.com/repos/ruflin/filebeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ruflin/filebeat/downloads","issues_url":"https://api.github.com/repos/ruflin/filebeat/issues{/number}","pulls_url":"https://api.github.com/repos/ruflin/filebeat/pulls{/number}","milestones_url":"https://api.github.com/repos/ruflin/filebeat/milestones{/number}","notifications_url":"https://api.github.com/repos/ruflin/filebeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ruflin/filebeat/labels{/name}","releases_url":"https://api.github.com/repos/ruflin/filebeat/releases{/id}","created_at":"2015-09-07T13:36:26Z","updated_at":"2015-09-11T18:59:43Z","pushed_at":"2015-09-15T21:01:30Z","git_url":"git://github.com/ruflin/filebeat.git","ssh_url":"git@github.com:ruflin/filebeat.git","clone_url":"https://github.com/ruflin/filebeat.git","svn_url":"https://github.com/ruflin/filebeat","homepage":null,"size":527,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"40d2b258c6ee847b7409503f38342189fa85ec94","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":42054649,"name":"filebeat","full_name":"elastic/filebeat","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/filebeat","description":"Open Source File Harvester","fork":false,"url":"https://api.github.com/repos/elastic/filebeat","forks_url":"https://api.github.com/repos/elastic/filebeat/forks","keys_url":"https://api.github.com/repos/elastic/filebeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/filebeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/filebeat/teams","hooks_url":"https://api.github.com/repos/elastic/filebeat/hooks","issue_events_url":"https://api.github.com/repos/elastic/filebeat/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/filebeat/events","assignees_url":"https://api.github.com/repos/elastic/filebeat/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/filebeat/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/filebeat/tags","blobs_url":"https://api.github.com/repos/elastic/filebeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/filebeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/filebeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/filebeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/filebeat/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/filebeat/languages","stargazers_url":"https://api.github.com/repos/elastic/filebeat/stargazers","contributors_url":"https://api.github.com/repos/elastic/filebeat/contributors","subscribers_url":"https://api.github.com/repos/elastic/filebeat/subscribers","subscription_url":"https://api.github.com/repos/elastic/filebeat/subscription","commits_url":"https://api.github.com/repos/elastic/filebeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/filebeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/filebeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/filebeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/filebeat/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/filebeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/filebeat/merges","archive_url":"https://api.github.com/repos/elastic/filebeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/filebeat/downloads","issues_url":"https://api.github.com/repos/elastic/filebeat/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/filebeat/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/filebeat/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/filebeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/filebeat/labels{/name}","releases_url":"https://api.github.com/repos/elastic/filebeat/releases{/id}","created_at":"2015-09-07T13:32:44Z","updated_at":"2015-09-14T17:01:06Z","pushed_at":"2015-09-15T21:01:30Z","git_url":"git://github.com/elastic/filebeat.git","ssh_url":"git@github.com:elastic/filebeat.git","clone_url":"https://github.com/elastic/filebeat.git","svn_url":"https://github.com/elastic/filebeat","homepage":null,"size":706,"stargazers_count":6,"watchers_count":6,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":5,"forks":8,"open_issues":5,"watchers":6,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/15"},"html":{"href":"https://github.com/elastic/filebeat/pull/15"},"issue":{"href":"https://api.github.com/repos/elastic/filebeat/issues/15"},"comments":{"href":"https://api.github.com/repos/elastic/filebeat/issues/15/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/15/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/filebeat/pulls/15/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/filebeat/statuses/c6d4a6460b9035b17a84af9e978bda5133d0d954"}}}}