{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/3969","labels_url":"https://api.github.com/repos/deis/deis/issues/3969/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/3969/comments","events_url":"https://api.github.com/repos/deis/deis/issues/3969/events","html_url":"https://github.com/deis/deis/issues/3969","id":93322731,"number":3969,"title":"Errors in logs, possibly from store.","user":{"login":"ngerakines","id":12125,"avatar_url":"https://avatars.githubusercontent.com/u/12125?v=3","gravatar_id":"","url":"https://api.github.com/users/ngerakines","html_url":"https://github.com/ngerakines","followers_url":"https://api.github.com/users/ngerakines/followers","following_url":"https://api.github.com/users/ngerakines/following{/other_user}","gists_url":"https://api.github.com/users/ngerakines/gists{/gist_id}","starred_url":"https://api.github.com/users/ngerakines/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngerakines/subscriptions","organizations_url":"https://api.github.com/users/ngerakines/orgs","repos_url":"https://api.github.com/users/ngerakines/repos","events_url":"https://api.github.com/users/ngerakines/events{/privacy}","received_events_url":"https://api.github.com/users/ngerakines/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/deis/deis/labels/store%20(Ceph)","name":"store (Ceph)","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2015-07-06T16:38:13Z","updated_at":"2015-09-01T19:56:21Z","closed_at":null,"body":"I'm seeing a huge number of errors in the logs.\r\n\r\n```\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.677722 7fd2a4088700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.698643 7fd2a708e700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.716780 7fcdc54e2700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.737903 7fa2afd7a700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.764561 7f0bfbbeb700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.790842 7fa2ad575700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.806244 7feea7fad700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.813644 7f0bfe3f0700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.841653 7fa2acd74700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.849120 7feea6fab700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.864762 7f0bfb3ea700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.865417 7feea57a8700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.884251 7fa2b057b700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.907858 7fd2a4889700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.935906 7fd2a508a700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\n```\r\n\r\nI saw #3884 but it looks like it was closed. I'm running 1.7.3 in AWS."},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/136841113","html_url":"https://github.com/deis/deis/issues/3969#issuecomment-136841113","issue_url":"https://api.github.com/repos/deis/deis/issues/3969","id":136841113,"user":{"login":"gvilarino","id":2538465,"avatar_url":"https://avatars.githubusercontent.com/u/2538465?v=3","gravatar_id":"","url":"https://api.github.com/users/gvilarino","html_url":"https://github.com/gvilarino","followers_url":"https://api.github.com/users/gvilarino/followers","following_url":"https://api.github.com/users/gvilarino/following{/other_user}","gists_url":"https://api.github.com/users/gvilarino/gists{/gist_id}","starred_url":"https://api.github.com/users/gvilarino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gvilarino/subscriptions","organizations_url":"https://api.github.com/users/gvilarino/orgs","repos_url":"https://api.github.com/users/gvilarino/repos","events_url":"https://api.github.com/users/gvilarino/events{/privacy}","received_events_url":"https://api.github.com/users/gvilarino/received_events","type":"User","site_admin":false},"created_at":"2015-09-01T19:56:21Z","updated_at":"2015-09-01T19:56:21Z","body":"I already did `deisctl restart store-daemon` and the first log I see after it complets the restart is:\r\n\r\n```\r\n<cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\n```\r\n\r\n:disappointed: "}}