{"action":"created","issue":{"url":"https://api.github.com/repos/MSOpenTech/redis/issues/235","labels_url":"https://api.github.com/repos/MSOpenTech/redis/issues/235/labels{/name}","comments_url":"https://api.github.com/repos/MSOpenTech/redis/issues/235/comments","events_url":"https://api.github.com/repos/MSOpenTech/redis/issues/235/events","html_url":"https://github.com/MSOpenTech/redis/issues/235","id":71908528,"number":235,"title":"Redis 2.8.19 BGREWRITEAOF - Failure Permission Denied","user":{"login":"chrisd88","id":12172430,"avatar_url":"https://avatars.githubusercontent.com/u/12172430?v=3","gravatar_id":"","url":"https://api.github.com/users/chrisd88","html_url":"https://github.com/chrisd88","followers_url":"https://api.github.com/users/chrisd88/followers","following_url":"https://api.github.com/users/chrisd88/following{/other_user}","gists_url":"https://api.github.com/users/chrisd88/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisd88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisd88/subscriptions","organizations_url":"https://api.github.com/users/chrisd88/orgs","repos_url":"https://api.github.com/users/chrisd88/repos","events_url":"https://api.github.com/users/chrisd88/events{/privacy}","received_events_url":"https://api.github.com/users/chrisd88/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/MSOpenTech/redis/labels/P1","name":"P1","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-29T15:30:11Z","updated_at":"2015-05-11T17:06:35Z","closed_at":null,"body":"Hi,\r\n\r\nWe currently have a Redis 2.8.19 on Win 2012 STD.\r\n\r\nThe box has 256GB Mem and the DB is eating around 187GB\r\n\r\nDue to the DB using so much memory we have opted to use AOF to store the data rather than the rdb file due to the memory requirements of doing BGSAVE on the Windows version of Redis.\r\n\r\nEach night we run 'BGREWRITEAOF' to regenerate the aof file so its more efficient if it has to reload the dataset.\r\n\r\nThis morning the process failed in the error logs we see\r\n\"EndForkOperation: 0x00000005 - EndForkOperation: Killing forked process failed.: permission denied\"\r\n\r\nThis is a bit of a mystery the new aof file has been generated but still showing the temp file name, the original aof file is still in place. Now the new aof file looks the correct size (~43GB) like its been fully generated.\r\n\r\nI'm under the assumption that when the aof file is generated, redis tries to delete the old aof file then renames the new aof file to the correct file name. Again I'm assuming the delete option failed but I could be wrong.\r\n\r\nI thought this might have been related to the AV software however the entire drive holding the redis data is excluded from both real-time and manual scanning (the drive holds nothing other than redis)\r\n\r\nWhat doesn't make sense is why the failure killed the entire redis-server process. The idea of the BGREWRITEAOF process is that if it fails the redis-server process should not die. So in our case we had to wait 2.5hrs for the server to re-load the dataset.\r\n\r\nUnfortunately the permission denied error doesn't explain what it couldn't do."},"comment":{"url":"https://api.github.com/repos/MSOpenTech/redis/issues/comments/100980296","html_url":"https://github.com/MSOpenTech/redis/issues/235#issuecomment-100980296","issue_url":"https://api.github.com/repos/MSOpenTech/redis/issues/235","id":100980296,"user":{"login":"chrisd88","id":12172430,"avatar_url":"https://avatars.githubusercontent.com/u/12172430?v=3","gravatar_id":"","url":"https://api.github.com/users/chrisd88","html_url":"https://github.com/chrisd88","followers_url":"https://api.github.com/users/chrisd88/followers","following_url":"https://api.github.com/users/chrisd88/following{/other_user}","gists_url":"https://api.github.com/users/chrisd88/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisd88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisd88/subscriptions","organizations_url":"https://api.github.com/users/chrisd88/orgs","repos_url":"https://api.github.com/users/chrisd88/repos","events_url":"https://api.github.com/users/chrisd88/events{/privacy}","received_events_url":"https://api.github.com/users/chrisd88/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T17:06:35Z","updated_at":"2015-05-11T17:06:35Z","body":"Hi,\r\n\r\nThe redis event log can be grabbed from http://chrisdaley.net/redislog.zip I don't think its going to be much help to be honest\r\n\r\nNothing in the Application or System logs, all we get is \"The Redis service terminated unexpectedly.\"\r\n\r\nWe had the service set to automatic recovery, 5min delay between restart however this has highlighted another weird problem. Sometimes when it loads the data into memory the logical drive where the AOF and Heap files are stored goes into meltdown essentially file browsing access in explorer is next to impossible, redis slows to a crawl when loading the data etc. Only way to recover is to manually kill the process in task manager (doesn't always kill it though). Nothing wrong with the drives, new batch of 8x Intel DC S3700 SSD's, Raid 6 array on a Dell PERC H720P Controller. \r\n\r\nFrom a normal boot you get 1-2 days where Redis works and the 'BGREWRITEAOF' process functions correctly. Then on the next day it will fail and usually loading the dataset will go badly (as described above). Reboot the box and it will function correctly for another 1-2 days.\r\n\r\nTotally removed the AV software however the problem persists so I don't believe that is the cause.\r\n"}}