{"action":"created","comment":{"url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/comments/38617601","id":38617601,"diff_hunk":"@@ -42,35 +42,38 @@ def self.generate_columns(metadata)\n         end\n \n         def run\n-          count = 0\n           from_datetime = task[:from_datetime]\n-          to_datetime = task[:to_datetime]\n+          to_datetime = task[:to_datetime] || Time.now\n+          ranges = task_target(from_datetime, to_datetime, task[:workers], index)","path":"lib/embulk/input/marketo/lead.rb","position":8,"original_position":8,"commit_id":"7d031a47abc75695f55fcfcc096b76e67d3299fd","original_commit_id":"7d031a47abc75695f55fcfcc096b76e67d3299fd","user":{"login":"muga","id":127959,"avatar_url":"https://avatars.githubusercontent.com/u/127959?v=3","gravatar_id":"","url":"https://api.github.com/users/muga","html_url":"https://github.com/muga","followers_url":"https://api.github.com/users/muga/followers","following_url":"https://api.github.com/users/muga/following{/other_user}","gists_url":"https://api.github.com/users/muga/gists{/gist_id}","starred_url":"https://api.github.com/users/muga/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/muga/subscriptions","organizations_url":"https://api.github.com/users/muga/orgs","repos_url":"https://api.github.com/users/muga/repos","events_url":"https://api.github.com/users/muga/events{/privacy}","received_events_url":"https://api.github.com/users/muga/received_events","type":"User","site_admin":false},"body":"In this case, same time ranges can be generated by run method, you know. But run method is executed on each worker thread. So, run method should not control (or divide) a whole of task. It should execute tasks.","created_at":"2015-09-03T06:57:12Z","updated_at":"2015-09-03T06:57:12Z","html_url":"https://github.com/treasure-data/embulk-input-marketo/pull/31#discussion_r38617601","pull_request_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31","_links":{"self":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/comments/38617601"},"html":{"href":"https://github.com/treasure-data/embulk-input-marketo/pull/31#discussion_r38617601"},"pull_request":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31"}}},"pull_request":{"url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31","id":43844775,"html_url":"https://github.com/treasure-data/embulk-input-marketo/pull/31","diff_url":"https://github.com/treasure-data/embulk-input-marketo/pull/31.diff","patch_url":"https://github.com/treasure-data/embulk-input-marketo/pull/31.patch","issue_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/31","number":31,"state":"open","locked":false,"title":"Concurrent worker","user":{"login":"uu59","id":499680,"avatar_url":"https://avatars.githubusercontent.com/u/499680?v=3","gravatar_id":"","url":"https://api.github.com/users/uu59","html_url":"https://github.com/uu59","followers_url":"https://api.github.com/users/uu59/followers","following_url":"https://api.github.com/users/uu59/following{/other_user}","gists_url":"https://api.github.com/users/uu59/gists{/gist_id}","starred_url":"https://api.github.com/users/uu59/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uu59/subscriptions","organizations_url":"https://api.github.com/users/uu59/orgs","repos_url":"https://api.github.com/users/uu59/repos","events_url":"https://api.github.com/users/uu59/events{/privacy}","received_events_url":"https://api.github.com/users/uu59/received_events","type":"User","site_admin":false},"body":"Supports `workers` option that allows you to access to API concurrently.\r\n\r\nTest result on my machine is here (from - to is just 24 hours):\r\n\r\n```text\r\n$ /usr/bin/time bundle exec embulk run single_worker.yml\r\n      659.00 real       209.30 user         3.55 sys\r\n$ /usr/bin/time bundle exec embulk run 20_worker.yml\r\n      194.76 real       221.03 user         3.77 sys\r\n```\r\n\r\nAs you can see, multiple(20 workers) run is almost 3 times faster than single worker used.","created_at":"2015-09-01T07:23:40Z","updated_at":"2015-09-03T06:57:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6f06b9c454d56cd6255f5d09dbec081343e141d6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31/commits","review_comments_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31/comments","review_comment_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/comments{/number}","comments_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/31/comments","statuses_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/statuses/7d031a47abc75695f55fcfcc096b76e67d3299fd","head":{"label":"treasure-data:concurrent_worker","ref":"concurrent_worker","sha":"7d031a47abc75695f55fcfcc096b76e67d3299fd","user":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"repo":{"id":37900806,"name":"embulk-input-marketo","full_name":"treasure-data/embulk-input-marketo","owner":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/treasure-data/embulk-input-marketo","description":"Embulk Input Plugin for Marketo","fork":false,"url":"https://api.github.com/repos/treasure-data/embulk-input-marketo","forks_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/forks","keys_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/keys{/key_id}","collaborators_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/teams","hooks_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/hooks","issue_events_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/events{/number}","events_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/events","assignees_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/assignees{/user}","branches_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/branches{/branch}","tags_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/tags","blobs_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/refs{/sha}","trees_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/trees{/sha}","statuses_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/statuses/{sha}","languages_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/languages","stargazers_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/stargazers","contributors_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/contributors","subscribers_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/subscribers","subscription_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/subscription","commits_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/commits{/sha}","git_commits_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/commits{/sha}","comments_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/comments{/number}","issue_comment_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/comments{/number}","contents_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/contents/{+path}","compare_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/compare/{base}...{head}","merges_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/merges","archive_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/downloads","issues_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues{/number}","pulls_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls{/number}","milestones_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/milestones{/number}","notifications_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/labels{/name}","releases_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/releases{/id}","created_at":"2015-06-23T06:14:16Z","updated_at":"2015-06-23T06:33:22Z","pushed_at":"2015-09-02T08:38:22Z","git_url":"git://github.com/treasure-data/embulk-input-marketo.git","ssh_url":"git@github.com:treasure-data/embulk-input-marketo.git","clone_url":"https://github.com/treasure-data/embulk-input-marketo.git","svn_url":"https://github.com/treasure-data/embulk-input-marketo","homepage":"","size":572,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"treasure-data:master","ref":"master","sha":"442f1a58fdfb9d8f932d856bbbc645ccbe0f10ec","user":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"repo":{"id":37900806,"name":"embulk-input-marketo","full_name":"treasure-data/embulk-input-marketo","owner":{"login":"treasure-data","id":747746,"avatar_url":"https://avatars.githubusercontent.com/u/747746?v=3","gravatar_id":"","url":"https://api.github.com/users/treasure-data","html_url":"https://github.com/treasure-data","followers_url":"https://api.github.com/users/treasure-data/followers","following_url":"https://api.github.com/users/treasure-data/following{/other_user}","gists_url":"https://api.github.com/users/treasure-data/gists{/gist_id}","starred_url":"https://api.github.com/users/treasure-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treasure-data/subscriptions","organizations_url":"https://api.github.com/users/treasure-data/orgs","repos_url":"https://api.github.com/users/treasure-data/repos","events_url":"https://api.github.com/users/treasure-data/events{/privacy}","received_events_url":"https://api.github.com/users/treasure-data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/treasure-data/embulk-input-marketo","description":"Embulk Input Plugin for Marketo","fork":false,"url":"https://api.github.com/repos/treasure-data/embulk-input-marketo","forks_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/forks","keys_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/keys{/key_id}","collaborators_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/teams","hooks_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/hooks","issue_events_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/events{/number}","events_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/events","assignees_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/assignees{/user}","branches_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/branches{/branch}","tags_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/tags","blobs_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/refs{/sha}","trees_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/trees{/sha}","statuses_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/statuses/{sha}","languages_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/languages","stargazers_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/stargazers","contributors_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/contributors","subscribers_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/subscribers","subscription_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/subscription","commits_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/commits{/sha}","git_commits_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/git/commits{/sha}","comments_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/comments{/number}","issue_comment_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/comments{/number}","contents_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/contents/{+path}","compare_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/compare/{base}...{head}","merges_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/merges","archive_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/downloads","issues_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues{/number}","pulls_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls{/number}","milestones_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/milestones{/number}","notifications_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/labels{/name}","releases_url":"https://api.github.com/repos/treasure-data/embulk-input-marketo/releases{/id}","created_at":"2015-06-23T06:14:16Z","updated_at":"2015-06-23T06:33:22Z","pushed_at":"2015-09-02T08:38:22Z","git_url":"git://github.com/treasure-data/embulk-input-marketo.git","ssh_url":"git@github.com:treasure-data/embulk-input-marketo.git","clone_url":"https://github.com/treasure-data/embulk-input-marketo.git","svn_url":"https://github.com/treasure-data/embulk-input-marketo","homepage":"","size":572,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31"},"html":{"href":"https://github.com/treasure-data/embulk-input-marketo/pull/31"},"issue":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/31"},"comments":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/issues/31/comments"},"review_comments":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31/comments"},"review_comment":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/pulls/31/commits"},"statuses":{"href":"https://api.github.com/repos/treasure-data/embulk-input-marketo/statuses/7d031a47abc75695f55fcfcc096b76e67d3299fd"}}}}