{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/23110","labels_url":"https://api.github.com/repos/saltstack/salt/issues/23110/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/23110/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/23110/events","html_url":"https://github.com/saltstack/salt/issues/23110","id":71413934,"number":23110,"title":"Copying files from gitfs in file.recurse state fails","user":{"login":"martinhoefling","id":1011603,"avatar_url":"https://avatars.githubusercontent.com/u/1011603?v=3","gravatar_id":"","url":"https://api.github.com/users/martinhoefling","html_url":"https://github.com/martinhoefling","followers_url":"https://api.github.com/users/martinhoefling/followers","following_url":"https://api.github.com/users/martinhoefling/following{/other_user}","gists_url":"https://api.github.com/users/martinhoefling/gists{/gist_id}","starred_url":"https://api.github.com/users/martinhoefling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinhoefling/subscriptions","organizations_url":"https://api.github.com/users/martinhoefling/orgs","repos_url":"https://api.github.com/users/martinhoefling/repos","events_url":"https://api.github.com/users/martinhoefling/events{/privacy}","received_events_url":"https://api.github.com/users/martinhoefling/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/File%20Servers","name":"File Servers","color":"006b75"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P3","name":"P3","color":"0a3d77"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Platform","name":"Platform","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2518,"closed_issues":2581,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2015-09-09T21:28:24Z","due_on":null,"closed_at":null},"comments":19,"created_at":"2015-04-27T21:56:38Z","updated_at":"2015-09-09T22:01:46Z","closed_at":null,"body":"# What's the problem?\r\n\r\nWhen highstating, copying a 'large file' tree (~250 files) from a git repository fails *sometimes*. The traceback looks like if a lock file is cleaned up too early in some cases. The fix in #18839 does not resolve the issue.\r\n\r\n# How are we using gitfs\r\n\r\nWe have several minions (which are masters themselves) and each one obtains the correct version of the (state-)tree by matching the branch in git. We also use an 'intermediate' top file to workaround issue #12483, sth. like:\r\n```\r\nsalt-beta:\r\n  'beta-instancemaster':\r\n    - instance-beta\r\n```\r\nThe matching branch here is `salt-beta`, the environment `salt-beta` exclusively targets `beta-instancemaster` and the `top.sls` replacement is `instance-beta.sls`.\r\n\r\n# Traceback on Server\r\n```\r\n[DEBUG   ] Updating fileserver for git_pillar module\r\n[ERROR   ] Error in function _file_hash:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/salt/master.py\", line 1348, in run_func\r\n    ret = getattr(self, func)(load)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py\", line 512, in file_hash\r\n    fnd = self.find_file(load['path'], load['saltenv'])\r\n  File \"/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py\", line 466, in find_file\r\n    fnd = self.servers[fstr](path, saltenv, **kwargs)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/fileserver/gitfs.py\", line 1509, in find_file\r\n    with salt.utils.fopen(lk_fn, 'w+') as fp_:\r\n  File \"/usr/lib/python2.7/dist-packages/salt/utils/__init__.py\", line 1086, in fopen\r\n    fhandle = open(*args, **kwargs)\r\nIOError: [Errno 2] No such file or directory: '/var/cache/salt/master/gitfs/hash/salt-staging/top.sls.lk'\r\n```\r\n\r\n# Minion Output\r\n```\r\n----------\r\n          ID: /srv/salt-common\r\n    Function: file.recurse\r\n      Result: False\r\n     Comment: #### /srv/salt-common/redis-server/init.sls ####\r\n              Source file salt://|common/states/redis-server/init.sls not found\r\n     Started: 01:53:45.613956\r\n    Duration: 52353.076 ms\r\n     Changes:  \r\n```\r\n\r\n# Salt Version\r\n```\r\nroot@77ff44f4d585:/# salt --versions-report\r\n                  Salt: 2014.7.4\r\n                Python: 2.7.6 (default, Mar 22 2014, 22:59:56)\r\n                Jinja2: 2.7.2\r\n              M2Crypto: 0.21.1\r\n        msgpack-python: 0.3.0\r\n          msgpack-pure: Not Installed\r\n              pycrypto: 2.6.1\r\n               libnacl: Not Installed\r\n                PyYAML: 3.10\r\n                 ioflo: Not Installed\r\n                 PyZMQ: 14.0.1\r\n                  RAET: Not Installed\r\n                   ZMQ: 4.0.4\r\n                  Mako: 0.9.1\r\n Debian source package: 2014.7.4+ds-1trusty1\r\n```\r\n\r\n# Master Config\r\n```\r\nfileserver_backend:\r\n  - git\r\n  - roots\r\n\r\ngitfs_provider: gitpython\r\n\r\ngitfs_env_whitelist:\r\n  - salt-*\r\n\r\ngitfs_remotes:\r\n\r\n# State tree used by the overlord itself:\r\n  - file:///somerepo.git:\r\n    - root: deployment/salt/common/states\r\n  - file:///somerepo.git:\r\n    - root: deployment/salt/overlord/states\r\n\r\n# State trees and Pillar trees for export to minions:\r\n  - file:///somerepo.git:\r\n    - root: deployment/salt/common\r\n    - mountpoint: salt://common\r\n  - file:///somerepo.git:\r\n    - root: deployment/salt/instance-masters\r\n    - mountpoint: salt://instance-masters\r\n\r\nfile_roots:\r\n  base:\r\n    - /packages\r\n\r\nfile_recv: True\r\n\r\next_pillar:\r\n  - git: salt-daily file:///somerepo.git root=deployment/salt/overlord/pillar/daily\r\n  - git: salt-beta file:///somerepo.git root=deployment/salt/overlord/pillar/beta\r\n  - git: salt-staging file:///somerepo.git root=deployment/salt/overlord/pillar/staging\r\n  - git: salt-www file:///somerepo.git root=deployment/salt/overlord/pillar/www\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/139058332","html_url":"https://github.com/saltstack/salt/issues/23110#issuecomment-139058332","issue_url":"https://api.github.com/repos/saltstack/salt/issues/23110","id":139058332,"user":{"login":"martinhoefling","id":1011603,"avatar_url":"https://avatars.githubusercontent.com/u/1011603?v=3","gravatar_id":"","url":"https://api.github.com/users/martinhoefling","html_url":"https://github.com/martinhoefling","followers_url":"https://api.github.com/users/martinhoefling/followers","following_url":"https://api.github.com/users/martinhoefling/following{/other_user}","gists_url":"https://api.github.com/users/martinhoefling/gists{/gist_id}","starred_url":"https://api.github.com/users/martinhoefling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinhoefling/subscriptions","organizations_url":"https://api.github.com/users/martinhoefling/orgs","repos_url":"https://api.github.com/users/martinhoefling/repos","events_url":"https://api.github.com/users/martinhoefling/events{/privacy}","received_events_url":"https://api.github.com/users/martinhoefling/received_events","type":"User","site_admin":false},"created_at":"2015-09-09T22:01:46Z","updated_at":"2015-09-09T22:01:46Z","body":"@jfindlay We did not observer this problem anymore, shall we close the issue?"}}