{"action":"created","issue":{"url":"https://api.github.com/repos/fedora-infra/bodhi/issues/328","labels_url":"https://api.github.com/repos/fedora-infra/bodhi/issues/328/labels{/name}","comments_url":"https://api.github.com/repos/fedora-infra/bodhi/issues/328/comments","events_url":"https://api.github.com/repos/fedora-infra/bodhi/issues/328/events","html_url":"https://github.com/fedora-infra/bodhi/issues/328","id":102564413,"number":328,"title":"Bug in caching repodata for EL5 & EL6","user":{"login":"lmacken","id":9980,"avatar_url":"https://avatars.githubusercontent.com/u/9980?v=3","gravatar_id":"","url":"https://api.github.com/users/lmacken","html_url":"https://github.com/lmacken","followers_url":"https://api.github.com/users/lmacken/followers","following_url":"https://api.github.com/users/lmacken/following{/other_user}","gists_url":"https://api.github.com/users/lmacken/gists{/gist_id}","starred_url":"https://api.github.com/users/lmacken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmacken/subscriptions","organizations_url":"https://api.github.com/users/lmacken/orgs","repos_url":"https://api.github.com/users/lmacken/repos","events_url":"https://api.github.com/users/lmacken/events{/privacy}","received_events_url":"https://api.github.com/users/lmacken/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/fedora-infra/bodhi/labels/Masher","name":"Masher","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-22T20:07:41Z","updated_at":"2015-09-14T18:59:20Z","closed_at":null,"body":"We're hitting this traceback in production when trying to cache the repodata of EL6 & EL5 (not EL7 or Fedora). The contents of that directory are `/mnt/koji/mash/updates/dist-6E-epel-150822.1734/../dist-6E-epel.repocache/repodata/.nfs0000000007d203a900000196`. We think it might be related to the fact that we're still using bzip2 compression for that repodata. Could be a bug in createrepo_c or bzip2? \r\n\r\n```\r\nTraceback (most recent call last):\r\nFile \"/usr/lib/python2.7/site-packages/bodhi/masher.py\", line 252, in run\r\nself.work()\r\nFile \"/usr/lib/python2.7/site-packages/bodhi/masher.py\", line 327, in work\r\nuinfo.cache_repodata()\r\nFile \"/usr/lib/python2.7/site-packages/bodhi/metadata.py\", line 323, in cache_repodata\r\nshutil.rmtree(cache)\r\nFile \"/usr/lib64/python2.7/shutil.py\", line 256, in rmtree\r\nonerror(os.rmdir, path, sys.exc_info())\r\nFile \"/usr/lib64/python2.7/shutil.py\", line 254, in rmtree\r\nos.rmdir(path)\r\nOSError: [Errno 39] Directory not empty: '/mnt/koji/mash/updates/dist-6E-epel-testing-150822.1930/../dist-6E-epel-testing.repocache/repodata/'\r\n```"},"comment":{"url":"https://api.github.com/repos/fedora-infra/bodhi/issues/comments/140176294","html_url":"https://github.com/fedora-infra/bodhi/issues/328#issuecomment-140176294","issue_url":"https://api.github.com/repos/fedora-infra/bodhi/issues/328","id":140176294,"user":{"login":"nirik","id":1779112,"avatar_url":"https://avatars.githubusercontent.com/u/1779112?v=3","gravatar_id":"","url":"https://api.github.com/users/nirik","html_url":"https://github.com/nirik","followers_url":"https://api.github.com/users/nirik/followers","following_url":"https://api.github.com/users/nirik/following{/other_user}","gists_url":"https://api.github.com/users/nirik/gists{/gist_id}","starred_url":"https://api.github.com/users/nirik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nirik/subscriptions","organizations_url":"https://api.github.com/users/nirik/orgs","repos_url":"https://api.github.com/users/nirik/repos","events_url":"https://api.github.com/users/nirik/events{/privacy}","received_events_url":"https://api.github.com/users/nirik/received_events","type":"User","site_admin":false},"created_at":"2015-09-14T18:59:20Z","updated_at":"2015-09-14T18:59:20Z","body":"The file seems to be the updateinfo.xml.bz2 fiile. It's being held open by the fedmsg-hub process. (so, restarting fedmsg-hub makes the file go away). \r\n\r\nLooking at the fedmsg-hub process: \r\n\r\n# ls -la /proc/29522/fd | grep epel7\r\nl-wx------. 1 masher masher 64 Sep 14 18:57 437 -> /mnt/fedora_koji/koji/mash/updates/epel7-150914.1814/epel7/x86_64/repodata/6822870af83ae6bcdbfeded540fb33c3c5f8a5dcd3c16d283f3d7dc4ad351b32-updateinfo.xml.bz2\r\nlr-x------. 1 masher masher 64 Sep 14 18:57 67 -> /mnt/fedora_koji/koji/mash/updates/epel7.repocache/repodata/.nfs00000000057b4bfe00000406\r\nl-wx------. 1 masher masher 64 Sep 14 18:57 93 -> /mnt/fedora_koji/koji/mash/updates/epel7-150914.1814/epel7/SRPMS/repodata/6822870af83ae6bcdbfeded540fb33c3c5f8a5dcd3c16d283f3d7dc4ad351b32-updateinfo.xml.bz2\r\nl-wx------. 1 masher masher 64 Sep 14 18:57 97 -> /mnt/fedora_koji/koji/mash/updates/epel7-150914.1814/epel7/ppc64/repodata/6822870af83ae6bcdbfeded540fb33c3c5f8a5dcd3c16d283f3d7dc4ad351b32-updateinfo.xml.bz2\r\n\r\nIt seems like a race condition where it sometimes tries to remove the file before it's done writing it?"}}