{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/5791","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/5791/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/5791/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/5791/events","html_url":"https://github.com/dmwm/WMCore/issues/5791","id":68952981,"number":5791,"title":"Implement shuffling of PU files","user":{"login":"amaltaro","id":1798520,"avatar_url":"https://avatars.githubusercontent.com/u/1798520?v=3","gravatar_id":"","url":"https://api.github.com/users/amaltaro","html_url":"https://github.com/amaltaro","followers_url":"https://api.github.com/users/amaltaro/followers","following_url":"https://api.github.com/users/amaltaro/following{/other_user}","gists_url":"https://api.github.com/users/amaltaro/gists{/gist_id}","starred_url":"https://api.github.com/users/amaltaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaltaro/subscriptions","organizations_url":"https://api.github.com/users/amaltaro/orgs","repos_url":"https://api.github.com/users/amaltaro/repos","events_url":"https://api.github.com/users/amaltaro/events{/privacy}","received_events_url":"https://api.github.com/users/amaltaro/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"ericvaandering","id":1672366,"avatar_url":"https://avatars.githubusercontent.com/u/1672366?v=3","gravatar_id":"","url":"https://api.github.com/users/ericvaandering","html_url":"https://github.com/ericvaandering","followers_url":"https://api.github.com/users/ericvaandering/followers","following_url":"https://api.github.com/users/ericvaandering/following{/other_user}","gists_url":"https://api.github.com/users/ericvaandering/gists{/gist_id}","starred_url":"https://api.github.com/users/ericvaandering/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericvaandering/subscriptions","organizations_url":"https://api.github.com/users/ericvaandering/orgs","repos_url":"https://api.github.com/users/ericvaandering/repos","events_url":"https://api.github.com/users/ericvaandering/events{/privacy}","received_events_url":"https://api.github.com/users/ericvaandering/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-04-16T15:49:16Z","updated_at":"2015-04-20T16:07:22Z","closed_at":"2015-04-20T02:53:31Z","body":"As a follow up of discussions all around and this summary from @chwissing \r\n\"in the last few weeks we have constantly trouble with processing\r\nworkflows, because each and every job opens the _same_ pileup file for\r\ndiscovering the data products. That causes some issues at some sites.\"\r\n\r\nwe should shuffle the pileup files to avoid opening always the same one by each job.\r\n\r\nHere is my understand of the code and how it works in WMAgent. WMAgent will create a list of PU files available at the site whitelisted (based only on *completed* blocks available there) and this list is then shipped together with the job (in a json file), which is then fed into the PSet.py configuration during runtime in the worker node (such to avoid DBS calls from the WN).\r\nI guess @ericvaandering is taking over from here ;-)"},"comment":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/comments/94494878","html_url":"https://github.com/dmwm/WMCore/issues/5791#issuecomment-94494878","issue_url":"https://api.github.com/repos/dmwm/WMCore/issues/5791","id":94494878,"user":{"login":"Dr15Jones","id":3757616,"avatar_url":"https://avatars.githubusercontent.com/u/3757616?v=3","gravatar_id":"","url":"https://api.github.com/users/Dr15Jones","html_url":"https://github.com/Dr15Jones","followers_url":"https://api.github.com/users/Dr15Jones/followers","following_url":"https://api.github.com/users/Dr15Jones/following{/other_user}","gists_url":"https://api.github.com/users/Dr15Jones/gists{/gist_id}","starred_url":"https://api.github.com/users/Dr15Jones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dr15Jones/subscriptions","organizations_url":"https://api.github.com/users/Dr15Jones/orgs","repos_url":"https://api.github.com/users/Dr15Jones/repos","events_url":"https://api.github.com/users/Dr15Jones/events{/privacy}","received_events_url":"https://api.github.com/users/Dr15Jones/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T16:07:22Z","updated_at":"2015-04-20T16:07:22Z","body":"The framework right now just opens the first file in the list in order to find out what data is available and then it closes that file. However, when it comes time to actually get events for pileup, we randomly choose which file to start from and then randomly choose which event to start from within that file."}}