{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/3969","labels_url":"https://api.github.com/repos/deis/deis/issues/3969/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/3969/comments","events_url":"https://api.github.com/repos/deis/deis/issues/3969/events","html_url":"https://github.com/deis/deis/issues/3969","id":93322731,"number":3969,"title":"Errors in logs, possibly from store.","user":{"login":"ngerakines","id":12125,"avatar_url":"https://avatars.githubusercontent.com/u/12125?v=3","gravatar_id":"","url":"https://api.github.com/users/ngerakines","html_url":"https://github.com/ngerakines","followers_url":"https://api.github.com/users/ngerakines/followers","following_url":"https://api.github.com/users/ngerakines/following{/other_user}","gists_url":"https://api.github.com/users/ngerakines/gists{/gist_id}","starred_url":"https://api.github.com/users/ngerakines/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngerakines/subscriptions","organizations_url":"https://api.github.com/users/ngerakines/orgs","repos_url":"https://api.github.com/users/ngerakines/repos","events_url":"https://api.github.com/users/ngerakines/events{/privacy}","received_events_url":"https://api.github.com/users/ngerakines/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/deis/deis/labels/store%20(Ceph)","name":"store (Ceph)","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-07-06T16:38:13Z","updated_at":"2015-09-01T16:13:38Z","closed_at":null,"body":"I'm seeing a huge number of errors in the logs.\r\n\r\n```\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.677722 7fd2a4088700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.698643 7fd2a708e700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.716780 7fcdc54e2700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.737903 7fa2afd7a700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.764561 7f0bfbbeb700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.790842 7fa2ad575700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.806244 7feea7fad700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.813644 7f0bfe3f0700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.841653 7fa2acd74700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.849120 7feea6fab700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.864762 7f0bfb3ea700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.865417 7feea57a8700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.884251 7fa2b057b700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.907858 7fd2a4889700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\nJul 06 12:36:44 deis sh:  2015-07-06 16:36:44.935906 7fd2a508a700  0 <cls> cls/rgw/cls_rgw.cc:1947: ERROR: rgw_obj_remove(): cls_cxx_remove returned -2\r\n```\r\n\r\nI saw #3884 but it looks like it was closed. I'm running 1.7.3 in AWS."},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/136777083","html_url":"https://github.com/deis/deis/issues/3969#issuecomment-136777083","issue_url":"https://api.github.com/repos/deis/deis/issues/3969","id":136777083,"user":{"login":"matiasdecarli","id":2442390,"avatar_url":"https://avatars.githubusercontent.com/u/2442390?v=3","gravatar_id":"","url":"https://api.github.com/users/matiasdecarli","html_url":"https://github.com/matiasdecarli","followers_url":"https://api.github.com/users/matiasdecarli/followers","following_url":"https://api.github.com/users/matiasdecarli/following{/other_user}","gists_url":"https://api.github.com/users/matiasdecarli/gists{/gist_id}","starred_url":"https://api.github.com/users/matiasdecarli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matiasdecarli/subscriptions","organizations_url":"https://api.github.com/users/matiasdecarli/orgs","repos_url":"https://api.github.com/users/matiasdecarli/repos","events_url":"https://api.github.com/users/matiasdecarli/events{/privacy}","received_events_url":"https://api.github.com/users/matiasdecarli/received_events","type":"User","site_admin":false},"created_at":"2015-09-01T16:13:37Z","updated_at":"2015-09-01T16:13:37Z","body":"Im running into this aswell. Running on Azure with a 5 machine cluster. The ceph looks healthy, but Im still getting the `rgw_obj_remove()`  "}}