{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/22029","labels_url":"https://api.github.com/repos/saltstack/salt/issues/22029/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/22029/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/22029/events","html_url":"https://github.com/saltstack/salt/issues/22029","id":64469681,"number":22029,"title":"schedule does not honor large splay values","user":{"login":"felskrone","id":3438840,"avatar_url":"https://avatars.githubusercontent.com/u/3438840?v=3","gravatar_id":"","url":"https://api.github.com/users/felskrone","html_url":"https://github.com/felskrone","followers_url":"https://api.github.com/users/felskrone/followers","following_url":"https://api.github.com/users/felskrone/following{/other_user}","gists_url":"https://api.github.com/users/felskrone/gists{/gist_id}","starred_url":"https://api.github.com/users/felskrone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felskrone/subscriptions","organizations_url":"https://api.github.com/users/felskrone/orgs","repos_url":"https://api.github.com/users/felskrone/repos","events_url":"https://api.github.com/users/felskrone/events{/privacy}","received_events_url":"https://api.github.com/users/felskrone/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Core","name":"Core","color":"006b75"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Fixed%20Pending%20Verification","name":"Fixed Pending Verification","color":"d4c5f9"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P4","name":"P4","color":"031a39"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2162,"closed_issues":1941,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-05-05T14:42:21Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2015-03-26T08:34:36Z","updated_at":"2015-05-05T14:42:21Z","closed_at":"2015-05-05T14:42:21Z","body":"Im trying to setup automated downloading (not installing) of available updates. The actual downloading is triggered by a schedule from pillar which looks like this:\r\n\r\n```yaml\r\nschedule:\r\n  update_downloader:\r\n    function: apt_downloader.run\r\n    maxrunning: 1\r\n    jid_include: True\r\n    minutes: 330\r\n    range:\r\n      start: 08:30am\r\n      end: 04:00pm\r\n    splay: 23000\r\n```\r\nThe idea behind the schedule is, to have all my hosts (15.000+) download the available updates during the day with approx. 2-3 minions hitting our package mirror at a time. \r\n\r\nIt seems though, that large splay values are not honored for some reason, but sometimes they are.\r\n\r\nMy hosts killed our mirror yesterday, because 15.000 hosts started their download-job within the first hour of the schedule (i.e. between 08:30am and 09:30am), hitting the mirror with about 5 minions per second.\r\n\r\nI disabled the schedule for the majority of the hosts, ran saltutil.refresh_pillar on all hosts while leaving the schedule enabled on about 350 hosts.\r\n\r\nThis morning i checked the 350 hosts and noticed the same behaviour again:\r\n\r\n```bash\r\nserver192.mydomain.de: 2015-03-26 08:30:10,265 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver193.mydomain.de: 2015-03-26 08:30:00,149 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver194.mydomain.de: 2015-03-26 08:30:00,104 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver195.mydomain.de: 2015-03-26 08:30:00,479 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver196.mydomain.de: 2015-03-26 09:32:04,848 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver197.mydomain.de: 2015-03-26 08:30:00,175 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver198.mydomain.de: 2015-03-26 08:30:00,423 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver199.mydomain.de: 2015-03-26 08:30:00,036 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver200.mydomain.de: 2015-03-26 08:30:00,762 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver201.mydomain.de: 2015-03-26 08:30:00,693 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver202.mydomain.de: 2015-03-26 08:30:00,535 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver203.mydomain.de: 2015-03-26 08:30:00,661 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver204.mydomain.de: 2015-03-26 08:30:00,582 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver205.mydomain.de: 2015-03-26 08:30:00,190 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver206.mydomain.de: 2015-03-26 08:30:00,290 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver207.mydomain.de: 2015-03-26 08:30:00,851 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver208.mydomain.de: 2015-03-26 08:30:00,103 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver209.mydomain.de: 2015-03-26 08:30:00,166 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver210.mydomain.de: 2015-03-26 09:41:32,218 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver211.mydomain.de: 2015-03-26 08:30:00,766 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver212.mydomain.de: 2015-03-26 08:30:00,063 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver213.mydomain.de: 2015-03-26 08:30:01,008 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver214.mydomain.de: 2015-03-26 08:30:00,736 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver215.mydomain.de: 2015-03-26 08:30:00,898 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver216.mydomain.de: 2015-03-26 08:30:00,582 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver217.mydomain.de: 2015-03-26 08:30:00,859 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver218.mydomain.de: 2015-03-26 08:30:00,140 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver219.mydomain.de: 2015-03-26 08:53:15,673 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver220.mydomain.de: 2015-03-26 08:30:00,528 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver221.mydomain.de: 2015-03-26 08:30:00,172 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver222.mydomain.de: 2015-03-26 08:30:00,489 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver223.mydomain.de: 2015-03-26 08:30:00,128 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver224.mydomain.de: 2015-03-26 08:30:00,667 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver225.mydomain.de: 2015-03-26 08:30:00,764 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver226.mydomain.de: 2015-03-26 08:58:38,852 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver227.mydomain.de: 2015-03-26 08:30:00,326 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver228.mydomain.de: 2015-03-26 08:30:00,741 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver229.mydomain.de: 2015-03-26 08:30:00,926 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver230.mydomain.de: 2015-03-26 09:13:32,625 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver231.mydomain.de: 2015-03-26 08:30:00,737 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver232.mydomain.de: 2015-03-26 08:30:00,224 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver233.mydomain.de: 2015-03-26 08:30:00,801 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver234.mydomain.de: 2015-03-26 08:30:00,895 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver235.mydomain.de: 2015-03-26 08:30:00,244 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver236.mydomain.de: 2015-03-26 08:30:00,958 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver237.mydomain.de: 2015-03-26 08:30:00,860 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver238.mydomain.de: 2015-03-26 08:30:00,159 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver239.mydomain.de: 2015-03-26 08:30:00,136 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver240.mydomain.de: 2015-03-26 08:30:00,129 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver241.mydomain.de: 2015-03-26 08:30:00,491 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver242.mydomain.de: 2015-03-26 08:30:00,425 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver243.mydomain.de: 2015-03-26 08:30:00,854 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver244.mydomain.de: 2015-03-26 08:30:00,778 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver245.mydomain.de: 2015-03-26 08:30:00,211 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver246.mydomain.de: 2015-03-26 08:30:00,484 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver247.mydomain.de: 2015-03-26 08:30:00,493 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver248.mydomain.de: 2015-03-26 08:30:00,506 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver249.mydomain.de: 2015-03-26 08:30:00,825 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver250.mydomain.de: 2015-03-26 08:30:00,455 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver251.mydomain.de: 2015-03-26 08:30:00,847 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver252.mydomain.de: 2015-03-26 08:30:00,096 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver253.mydomain.de: 2015-03-26 08:30:00,644 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver254.mydomain.de: 2015-03-26 08:30:00,490 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver256.mydomain.de: 2015-03-26 08:30:00,475 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver257.mydomain.de: 2015-03-26 08:30:00,681 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver258.mydomain.de: 2015-03-26 08:30:00,384 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver259.mydomain.de: 2015-03-26 08:30:00,831 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver260.mydomain.de: 2015-03-26 08:30:00,440 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver261.mydomain.de: 2015-03-26 08:30:00,281 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver262.mydomain.de: 2015-03-26 08:30:00,947 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver263.mydomain.de: 2015-03-26 08:30:00,778 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver264.mydomain.de: 2015-03-26 08:30:00,635 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\n...\r\n```\r\n\r\nSome of the hosts from above, honored the splay value:\r\n\r\n```bash\r\nserver044.mydomain.de: 2015-03-26 09:37:14,906 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver049.mydomain.de: 2015-03-26 08:41:44,410 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver077.mydomain.de: 2015-03-26 08:45:05,882 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver083.mydomain.de: 2015-03-26 09:04:07,955 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver138.mydomain.de: 2015-03-26 08:39:37,430 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver196.mydomain.de: 2015-03-26 09:32:04,848 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver210.mydomain.de: 2015-03-26 09:41:32,218 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver219.mydomain.de: 2015-03-26 08:53:15,673 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver226.mydomain.de: 2015-03-26 08:58:38,852 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver230.mydomain.de: 2015-03-26 09:13:32,625 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver328.mydomain.de: 2015-03-26 08:51:32,193 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\nserver331.mydomain.de: 2015-03-26 08:44:05,440 [salt.utils.schedule ][INFO ] Running scheduled job: update_downloader\r\n```\r\n\r\nLooking at the schedule of several of the hosts, they all look alike:\r\n\r\n```yaml\r\n    schedule:\r\n      presence:\r\n        function: presence.fire\r\n        jid_include: false\r\n        maxrunning: 1\r\n        minutes: 5\r\n        splay: 120\r\n      refresh:\r\n        function: saltutil.sync_all\r\n        jid_include: false\r\n        maxrunning: 1\r\n        minutes: 90\r\n        splay: 4800\r\n      update_downloader:\r\n        function: apt_downloader.run\r\n        jid_include: true\r\n        maxrunning: 1\r\n        minutes: 330\r\n        range:\r\n          end: 04:00pm\r\n          start: 08:30am\r\n        splay: 23000\r\n```\r\nThe interesting thing is, that the splay values for the other jobs work fine. Here is an example of the refresh-job from the log of a minion with the above schedule with matches 90 minutes to the second.\r\n\r\n```bash\r\n2015-03-25 02:54:58,297 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 04:24:58,397 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 05:54:58,897 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 07:24:58,687 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 08:54:58,467 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 10:24:58,167 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 11:54:58,527 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 13:24:58,307 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 14:54:58,398 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n2015-03-25 16:24:58,647 [salt.utils.schedule                          ][INFO    ] Running scheduled job: refresh\r\n```\r\n\r\n\r\n\r\n```bash\r\n           Salt: 2014.7.1\r\n         Python: 2.6.6 (r266:84292, Aug 12 2014, 07:57:00)\r\n         Jinja2: 2.5.5\r\n       M2Crypto: 0.20.1\r\n msgpack-python: 0.1.10\r\n   msgpack-pure: Not Installed\r\n       pycrypto: 2.1.0\r\n        libnacl: Not Installed\r\n         PyYAML: 3.09\r\n          ioflo: Not Installed\r\n          PyZMQ: 13.1.0\r\n           RAET: Not Installed\r\n            ZMQ: 3.2.3\r\n           Mako: Not Installed\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/99100173","html_url":"https://github.com/saltstack/salt/issues/22029#issuecomment-99100173","issue_url":"https://api.github.com/repos/saltstack/salt/issues/22029","id":99100173,"user":{"login":"cachedout","id":111616,"avatar_url":"https://avatars.githubusercontent.com/u/111616?v=3","gravatar_id":"","url":"https://api.github.com/users/cachedout","html_url":"https://github.com/cachedout","followers_url":"https://api.github.com/users/cachedout/followers","following_url":"https://api.github.com/users/cachedout/following{/other_user}","gists_url":"https://api.github.com/users/cachedout/gists{/gist_id}","starred_url":"https://api.github.com/users/cachedout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cachedout/subscriptions","organizations_url":"https://api.github.com/users/cachedout/orgs","repos_url":"https://api.github.com/users/cachedout/repos","events_url":"https://api.github.com/users/cachedout/events{/privacy}","received_events_url":"https://api.github.com/users/cachedout/received_events","type":"User","site_admin":false},"created_at":"2015-05-05T14:42:21Z","updated_at":"2015-05-05T14:42:21Z","body":"Since we haven't received a reply here and it sounds like this is fixed, I will go ahead and close this issue. Thanks, all."}}