{"action":"created","comment":{"url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/comments/31801606","id":31801606,"diff_hunk":"@@ -5,32 +5,45 @@\n The user can select the batch size, the maximum number of batches and the starting batch for yielding elements\n Other function will be added to randomly sample elements and to retrieve the last batch  \n \"\"\"\n-def run_in_batches(iterable, end_batch_callback=None, batch_size=2, batch_start=0, max_batches=None):\t\n+def run_in_batches(iterable, end_batch_callback=None, batch_size=2, batch_start=0, max_batches=None, seed=None, percentage=None):\t\n     \"\"\" Create batches from an iterable\n    \n     This function is a generator that creates batches of elements from an iterable \n     The callback function is called when a) maximum batches are reached or b) iterable is exhausted \n     All batches (complete and incomplete) are ended by a callback function\n+\n+    If random options are not None, gets a percentage of the total data\n+    Pseudo-samples randomly the elements in the selected percentage (approximate)\n+    The main script logs the seed to make it repeatable\n     \"\"\" \t\n+    import random\n     c = 0\n     ended = False\n+    if percentage is not None:\t\t\t\t\t\t\t\t\t\n+        random.seed(seed)\t\t\t\t\t\t\t\t\t# seed can be passed as kw to be repeatable\n+        number = int(((100-percentage)*batch_size)/100)\t\t\t\t\t\t# e.g. to yield the 30% of batch i mask the 70%\n+        mask = set(random.sample(range(batch_size), number))\t\t\t\t\t\t\t\t\t\t\n+    else:\n+        mask = []","path":"mp/importer/batcher.py","position":24,"original_position":24,"commit_id":"188b8b0cb2ad727d209185f378339c11960ca186","original_commit_id":"188b8b0cb2ad727d209185f378339c11960ca186","user":{"login":"jinty","id":553369,"avatar_url":"https://avatars.githubusercontent.com/u/553369?v=3","gravatar_id":"","url":"https://api.github.com/users/jinty","html_url":"https://github.com/jinty","followers_url":"https://api.github.com/users/jinty/followers","following_url":"https://api.github.com/users/jinty/following{/other_user}","gists_url":"https://api.github.com/users/jinty/gists{/gist_id}","starred_url":"https://api.github.com/users/jinty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinty/subscriptions","organizations_url":"https://api.github.com/users/jinty/orgs","repos_url":"https://api.github.com/users/jinty/repos","events_url":"https://api.github.com/users/jinty/events{/privacy}","received_events_url":"https://api.github.com/users/jinty/received_events","type":"User","site_admin":false},"body":"On Fri, Jun 05, 2015 at 02:00:33AM -0700, kiarasky wrote:\n> >      c = 0\n> >      ended = False\n> > +    if percentage is not None:\t\t\t\t\t\t\t\t\t\n> > +        random.seed(seed)\t\t\t\t\t\t\t\t\t# seed can be passed as kw to be repeatable\n> > +        number = int(((100-percentage)*batch_size)/100)\t\t\t\t\t\t# e.g. to yield the 30% of batch i mask the 70%\n> > +        mask = set(random.sample(range(batch_size), number))\t\t\t\t\t\t\t\t\t\t\n> > +    else:\n> > +        mask = []\n> \n> ok, fixed. I'd like to make the tests work for py3, but i need to add a loop in the assert because the sampling methods give different results.         \n>         if _PY2:\n>              self.assertEqual(log_list, 'opqtvX')\t  \n>         if _PY3:\n>             self.assertEqual(log_list, 'qrstuX')\n> Also, I noticed that py3 is failing 2 tests, i'm not sure why but i'll check that\n\nI think you want to pass the seed as an integer, then python 2 and 3\nshould do the same thing\n\n-- \nBrian Sutherland\n","created_at":"2015-06-05T10:13:26Z","updated_at":"2015-06-05T10:13:26Z","html_url":"https://github.com/vanguardistas/mp.importer/pull/3#discussion_r31801606","pull_request_url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3","_links":{"self":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/comments/31801606"},"html":{"href":"https://github.com/vanguardistas/mp.importer/pull/3#discussion_r31801606"},"pull_request":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3"}}},"pull_request":{"url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3","id":36922389,"html_url":"https://github.com/vanguardistas/mp.importer/pull/3","diff_url":"https://github.com/vanguardistas/mp.importer/pull/3.diff","patch_url":"https://github.com/vanguardistas/mp.importer/pull/3.patch","issue_url":"https://api.github.com/repos/vanguardistas/mp.importer/issues/3","number":3,"state":"open","locked":false,"title":"#3 Random sampler to create random batches of elements","user":{"login":"kiarasky","id":1202476,"avatar_url":"https://avatars.githubusercontent.com/u/1202476?v=3","gravatar_id":"","url":"https://api.github.com/users/kiarasky","html_url":"https://github.com/kiarasky","followers_url":"https://api.github.com/users/kiarasky/followers","following_url":"https://api.github.com/users/kiarasky/following{/other_user}","gists_url":"https://api.github.com/users/kiarasky/gists{/gist_id}","starred_url":"https://api.github.com/users/kiarasky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiarasky/subscriptions","organizations_url":"https://api.github.com/users/kiarasky/orgs","repos_url":"https://api.github.com/users/kiarasky/repos","events_url":"https://api.github.com/users/kiarasky/events{/privacy}","received_events_url":"https://api.github.com/users/kiarasky/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-06-04T10:43:59Z","updated_at":"2015-06-05T10:13:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7caceb5c0a65645fbf42c50adda87c8c139c87f4","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3/commits","review_comments_url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3/comments","review_comment_url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/comments{/number}","comments_url":"https://api.github.com/repos/vanguardistas/mp.importer/issues/3/comments","statuses_url":"https://api.github.com/repos/vanguardistas/mp.importer/statuses/188b8b0cb2ad727d209185f378339c11960ca186","head":{"label":"kiarasky:master","ref":"master","sha":"188b8b0cb2ad727d209185f378339c11960ca186","user":{"login":"kiarasky","id":1202476,"avatar_url":"https://avatars.githubusercontent.com/u/1202476?v=3","gravatar_id":"","url":"https://api.github.com/users/kiarasky","html_url":"https://github.com/kiarasky","followers_url":"https://api.github.com/users/kiarasky/followers","following_url":"https://api.github.com/users/kiarasky/following{/other_user}","gists_url":"https://api.github.com/users/kiarasky/gists{/gist_id}","starred_url":"https://api.github.com/users/kiarasky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiarasky/subscriptions","organizations_url":"https://api.github.com/users/kiarasky/orgs","repos_url":"https://api.github.com/users/kiarasky/repos","events_url":"https://api.github.com/users/kiarasky/events{/privacy}","received_events_url":"https://api.github.com/users/kiarasky/received_events","type":"User","site_admin":false},"repo":{"id":32149353,"name":"mp.importer","full_name":"kiarasky/mp.importer","owner":{"login":"kiarasky","id":1202476,"avatar_url":"https://avatars.githubusercontent.com/u/1202476?v=3","gravatar_id":"","url":"https://api.github.com/users/kiarasky","html_url":"https://github.com/kiarasky","followers_url":"https://api.github.com/users/kiarasky/followers","following_url":"https://api.github.com/users/kiarasky/following{/other_user}","gists_url":"https://api.github.com/users/kiarasky/gists{/gist_id}","starred_url":"https://api.github.com/users/kiarasky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiarasky/subscriptions","organizations_url":"https://api.github.com/users/kiarasky/orgs","repos_url":"https://api.github.com/users/kiarasky/repos","events_url":"https://api.github.com/users/kiarasky/events{/privacy}","received_events_url":"https://api.github.com/users/kiarasky/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kiarasky/mp.importer","description":"Utilities to ease imports of content to MetroPublisher.","fork":true,"url":"https://api.github.com/repos/kiarasky/mp.importer","forks_url":"https://api.github.com/repos/kiarasky/mp.importer/forks","keys_url":"https://api.github.com/repos/kiarasky/mp.importer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kiarasky/mp.importer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kiarasky/mp.importer/teams","hooks_url":"https://api.github.com/repos/kiarasky/mp.importer/hooks","issue_events_url":"https://api.github.com/repos/kiarasky/mp.importer/issues/events{/number}","events_url":"https://api.github.com/repos/kiarasky/mp.importer/events","assignees_url":"https://api.github.com/repos/kiarasky/mp.importer/assignees{/user}","branches_url":"https://api.github.com/repos/kiarasky/mp.importer/branches{/branch}","tags_url":"https://api.github.com/repos/kiarasky/mp.importer/tags","blobs_url":"https://api.github.com/repos/kiarasky/mp.importer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kiarasky/mp.importer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kiarasky/mp.importer/git/refs{/sha}","trees_url":"https://api.github.com/repos/kiarasky/mp.importer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kiarasky/mp.importer/statuses/{sha}","languages_url":"https://api.github.com/repos/kiarasky/mp.importer/languages","stargazers_url":"https://api.github.com/repos/kiarasky/mp.importer/stargazers","contributors_url":"https://api.github.com/repos/kiarasky/mp.importer/contributors","subscribers_url":"https://api.github.com/repos/kiarasky/mp.importer/subscribers","subscription_url":"https://api.github.com/repos/kiarasky/mp.importer/subscription","commits_url":"https://api.github.com/repos/kiarasky/mp.importer/commits{/sha}","git_commits_url":"https://api.github.com/repos/kiarasky/mp.importer/git/commits{/sha}","comments_url":"https://api.github.com/repos/kiarasky/mp.importer/comments{/number}","issue_comment_url":"https://api.github.com/repos/kiarasky/mp.importer/issues/comments{/number}","contents_url":"https://api.github.com/repos/kiarasky/mp.importer/contents/{+path}","compare_url":"https://api.github.com/repos/kiarasky/mp.importer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kiarasky/mp.importer/merges","archive_url":"https://api.github.com/repos/kiarasky/mp.importer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kiarasky/mp.importer/downloads","issues_url":"https://api.github.com/repos/kiarasky/mp.importer/issues{/number}","pulls_url":"https://api.github.com/repos/kiarasky/mp.importer/pulls{/number}","milestones_url":"https://api.github.com/repos/kiarasky/mp.importer/milestones{/number}","notifications_url":"https://api.github.com/repos/kiarasky/mp.importer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kiarasky/mp.importer/labels{/name}","releases_url":"https://api.github.com/repos/kiarasky/mp.importer/releases{/id}","created_at":"2015-03-13T10:16:41Z","updated_at":"2015-04-17T15:15:16Z","pushed_at":"2015-06-04T16:13:28Z","git_url":"git://github.com/kiarasky/mp.importer.git","ssh_url":"git@github.com:kiarasky/mp.importer.git","clone_url":"https://github.com/kiarasky/mp.importer.git","svn_url":"https://github.com/kiarasky/mp.importer","homepage":null,"size":265,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"vanguardistas:master","ref":"master","sha":"bc20d70e7fdf4bccebd07eab7067c583a851e851","user":{"login":"vanguardistas","id":1736830,"avatar_url":"https://avatars.githubusercontent.com/u/1736830?v=3","gravatar_id":"","url":"https://api.github.com/users/vanguardistas","html_url":"https://github.com/vanguardistas","followers_url":"https://api.github.com/users/vanguardistas/followers","following_url":"https://api.github.com/users/vanguardistas/following{/other_user}","gists_url":"https://api.github.com/users/vanguardistas/gists{/gist_id}","starred_url":"https://api.github.com/users/vanguardistas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanguardistas/subscriptions","organizations_url":"https://api.github.com/users/vanguardistas/orgs","repos_url":"https://api.github.com/users/vanguardistas/repos","events_url":"https://api.github.com/users/vanguardistas/events{/privacy}","received_events_url":"https://api.github.com/users/vanguardistas/received_events","type":"Organization","site_admin":false},"repo":{"id":20488156,"name":"mp.importer","full_name":"vanguardistas/mp.importer","owner":{"login":"vanguardistas","id":1736830,"avatar_url":"https://avatars.githubusercontent.com/u/1736830?v=3","gravatar_id":"","url":"https://api.github.com/users/vanguardistas","html_url":"https://github.com/vanguardistas","followers_url":"https://api.github.com/users/vanguardistas/followers","following_url":"https://api.github.com/users/vanguardistas/following{/other_user}","gists_url":"https://api.github.com/users/vanguardistas/gists{/gist_id}","starred_url":"https://api.github.com/users/vanguardistas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanguardistas/subscriptions","organizations_url":"https://api.github.com/users/vanguardistas/orgs","repos_url":"https://api.github.com/users/vanguardistas/repos","events_url":"https://api.github.com/users/vanguardistas/events{/privacy}","received_events_url":"https://api.github.com/users/vanguardistas/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/vanguardistas/mp.importer","description":"Utilities to ease imports of content to MetroPublisher.","fork":false,"url":"https://api.github.com/repos/vanguardistas/mp.importer","forks_url":"https://api.github.com/repos/vanguardistas/mp.importer/forks","keys_url":"https://api.github.com/repos/vanguardistas/mp.importer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vanguardistas/mp.importer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vanguardistas/mp.importer/teams","hooks_url":"https://api.github.com/repos/vanguardistas/mp.importer/hooks","issue_events_url":"https://api.github.com/repos/vanguardistas/mp.importer/issues/events{/number}","events_url":"https://api.github.com/repos/vanguardistas/mp.importer/events","assignees_url":"https://api.github.com/repos/vanguardistas/mp.importer/assignees{/user}","branches_url":"https://api.github.com/repos/vanguardistas/mp.importer/branches{/branch}","tags_url":"https://api.github.com/repos/vanguardistas/mp.importer/tags","blobs_url":"https://api.github.com/repos/vanguardistas/mp.importer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vanguardistas/mp.importer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vanguardistas/mp.importer/git/refs{/sha}","trees_url":"https://api.github.com/repos/vanguardistas/mp.importer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vanguardistas/mp.importer/statuses/{sha}","languages_url":"https://api.github.com/repos/vanguardistas/mp.importer/languages","stargazers_url":"https://api.github.com/repos/vanguardistas/mp.importer/stargazers","contributors_url":"https://api.github.com/repos/vanguardistas/mp.importer/contributors","subscribers_url":"https://api.github.com/repos/vanguardistas/mp.importer/subscribers","subscription_url":"https://api.github.com/repos/vanguardistas/mp.importer/subscription","commits_url":"https://api.github.com/repos/vanguardistas/mp.importer/commits{/sha}","git_commits_url":"https://api.github.com/repos/vanguardistas/mp.importer/git/commits{/sha}","comments_url":"https://api.github.com/repos/vanguardistas/mp.importer/comments{/number}","issue_comment_url":"https://api.github.com/repos/vanguardistas/mp.importer/issues/comments{/number}","contents_url":"https://api.github.com/repos/vanguardistas/mp.importer/contents/{+path}","compare_url":"https://api.github.com/repos/vanguardistas/mp.importer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vanguardistas/mp.importer/merges","archive_url":"https://api.github.com/repos/vanguardistas/mp.importer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vanguardistas/mp.importer/downloads","issues_url":"https://api.github.com/repos/vanguardistas/mp.importer/issues{/number}","pulls_url":"https://api.github.com/repos/vanguardistas/mp.importer/pulls{/number}","milestones_url":"https://api.github.com/repos/vanguardistas/mp.importer/milestones{/number}","notifications_url":"https://api.github.com/repos/vanguardistas/mp.importer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vanguardistas/mp.importer/labels{/name}","releases_url":"https://api.github.com/repos/vanguardistas/mp.importer/releases{/id}","created_at":"2014-06-04T15:03:29Z","updated_at":"2015-05-28T13:06:35Z","pushed_at":"2015-06-04T16:13:29Z","git_url":"git://github.com/vanguardistas/mp.importer.git","ssh_url":"git@github.com:vanguardistas/mp.importer.git","clone_url":"https://github.com/vanguardistas/mp.importer.git","svn_url":"https://github.com/vanguardistas/mp.importer","homepage":null,"size":715,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3"},"html":{"href":"https://github.com/vanguardistas/mp.importer/pull/3"},"issue":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/issues/3"},"comments":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/issues/3/comments"},"review_comments":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3/comments"},"review_comment":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/pulls/3/commits"},"statuses":{"href":"https://api.github.com/repos/vanguardistas/mp.importer/statuses/188b8b0cb2ad727d209185f378339c11960ca186"}}}}