{"action":"created","issue":{"url":"https://api.github.com/repos/ceph/ceph/issues/4955","labels_url":"https://api.github.com/repos/ceph/ceph/issues/4955/labels{/name}","comments_url":"https://api.github.com/repos/ceph/ceph/issues/4955/comments","events_url":"https://api.github.com/repos/ceph/ceph/issues/4955/events","html_url":"https://github.com/ceph/ceph/pull/4955","id":88305363,"number":4955,"title":"bug fix: librados segmentation fault, when two read ops share one Aio…","user":{"login":"cmgitdream","id":6038557,"avatar_url":"https://avatars.githubusercontent.com/u/6038557?v=3","gravatar_id":"","url":"https://api.github.com/users/cmgitdream","html_url":"https://github.com/cmgitdream","followers_url":"https://api.github.com/users/cmgitdream/followers","following_url":"https://api.github.com/users/cmgitdream/following{/other_user}","gists_url":"https://api.github.com/users/cmgitdream/gists{/gist_id}","starred_url":"https://api.github.com/users/cmgitdream/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmgitdream/subscriptions","organizations_url":"https://api.github.com/users/cmgitdream/orgs","repos_url":"https://api.github.com/users/cmgitdream/repos","events_url":"https://api.github.com/users/cmgitdream/events{/privacy}","received_events_url":"https://api.github.com/users/cmgitdream/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ceph/ceph/labels/bug%20fix","name":"bug fix","color":"e11d21"},{"url":"https://api.github.com/repos/ceph/ceph/labels/core","name":"core","color":"0052cc"},{"url":"https://api.github.com/repos/ceph/ceph/labels/needs-qa","name":"needs-qa","color":"fad8c7"},{"url":"https://api.github.com/repos/ceph/ceph/labels/wip-sam-testing","name":"wip-sam-testing","color":"f7c6c7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-06-15T03:50:03Z","updated_at":"2015-08-26T16:16:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ceph/ceph/pulls/4955","html_url":"https://github.com/ceph/ceph/pull/4955","diff_url":"https://github.com/ceph/ceph/pull/4955.diff","patch_url":"https://github.com/ceph/ceph/pull/4955.patch"},"body":"      As we know,  AioCompletionImpl is mapped to one async io request. \r\nBuf if there are two or more aio requests bind to one AioCompletionImpl, \r\nsome unexpected result will happen.\r\n      this bug fix is about two aio read opeatons bind to one AioCompletionImpl \r\ncausing segmentaion fault. \r\n\r\nI use a simple librados client to test the bug, the main logic is:\r\n\r\nrados_read_op_read(read_op, 0, len, buf, &bytes_read, &ret);\r\nrados_read_op_read(read_op2, 0, len, buf2, &bytes_read2, &ret2);\r\nret = rados_aio_read_op_operate(read_op, ioctx, read_completion, object, 0);\r\nret = rados_aio_read_op_operate(read_op2, ioctx, read_completion, object, 0);\r\nret = rados_aio_wait_for_complete(read_completion);\r\n\r\nand other misc binds are:\r\naio_write, aio_write\r\naio_read, aio_write\r\naio_stat, aio_read\r\naio_write, aio_stat\r\n......\r\n\r\nMaybe some binds not causing segmentaion fault, but we \r\nstill need to make AioCompletionImpl semantic right.\r\n"},"comment":{"url":"https://api.github.com/repos/ceph/ceph/issues/comments/135083823","html_url":"https://github.com/ceph/ceph/pull/4955#issuecomment-135083823","issue_url":"https://api.github.com/repos/ceph/ceph/issues/4955","id":135083823,"user":{"login":"athanatos","id":544782,"avatar_url":"https://avatars.githubusercontent.com/u/544782?v=3","gravatar_id":"","url":"https://api.github.com/users/athanatos","html_url":"https://github.com/athanatos","followers_url":"https://api.github.com/users/athanatos/followers","following_url":"https://api.github.com/users/athanatos/following{/other_user}","gists_url":"https://api.github.com/users/athanatos/gists{/gist_id}","starred_url":"https://api.github.com/users/athanatos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/athanatos/subscriptions","organizations_url":"https://api.github.com/users/athanatos/orgs","repos_url":"https://api.github.com/users/athanatos/repos","events_url":"https://api.github.com/users/athanatos/events{/privacy}","received_events_url":"https://api.github.com/users/athanatos/received_events","type":"User","site_admin":false},"created_at":"2015-08-26T16:16:32Z","updated_at":"2015-08-26T16:16:32Z","body":"Caused crashes in testing, removing wip-sam-testing for now:\r\n\r\n2015-08-25T21:16:26.588 INFO:teuthology.run_tasks:Running task radosbench...\r\n2015-08-25T21:16:26.589 INFO:tasks.radosbench:Beginning radosbench...\r\n2015-08-25T21:16:26.589 INFO:tasks.ceph.ceph_manager:creating pool_name unique_pool_0\r\n2015-08-25T21:16:26.589 INFO:teuthology.orchestra.run.burnupi59:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool create unique_pool_0 16'\r\n2015-08-25T21:16:27.655 INFO:teuthology.orchestra.run.burnupi59.stderr:marked out osd.0.\r\n2015-08-25T21:16:27.670 INFO:teuthology.orchestra.run.burnupi59.stderr:pool 'unique_pool_0' created\r\n2015-08-25T21:16:27.693 INFO:teuthology.orchestra.run.burnupi59:Running: \"/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 4194304 -p unique_pool_0 bench 1800 write'\"\r\n2015-08-25T21:16:27.760 DEBUG:teuthology.run_tasks:Unwinding manager radosbench\r\n2015-08-25T21:16:27.760 INFO:tasks.radosbench:joining radosbench (timing out after 9000s)\r\n2015-08-25T21:16:27.823 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:warning: line 28: 'osd_pool_default_size' in section 'global' redefined\r\n2015-08-25T21:16:27.844 INFO:tasks.radosbench.radosbench.0.burnupi59.stdout:Maintaining 16 concurrent writes of 4194304 bytes for up to 1800 seconds\r\n2015-08-25T21:16:27.844 INFO:tasks.radosbench.radosbench.0.burnupi59.stdout:Object prefix: benchmark_data_burnupi59_25963\r\n2015-08-25T21:16:27.885 INFO:tasks.radosbench.radosbench.0.burnupi59.stdout:  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat\r\n2015-08-25T21:16:27.885 INFO:tasks.radosbench.radosbench.0.burnupi59.stdout:    0       0         0         0         0         0         -         0\r\n2015-08-25T21:16:27.885 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:librados/IoCtxImpl.cc: In function 'librados::IoCtxImpl::C_aio_Ack::C_aio_Ack(librados::AioCompletionImpl*)' thread 7f911c0cf840 time 2015-08-25 21:16:27.876588\r\n2015-08-25T21:16:27.885 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:librados/IoCtxImpl.cc: 1302: FAILED assert(!c->io)\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: ceph version 9.0.3-879-g09b1303 (09b1303a19d2aaeca4e44585d4724ef3f92f4d77)\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 1: (()+0x1581fb) [0x7f91197c81fb]\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 2: (()+0xadbda) [0x7f911971dbda]\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 3: (()+0xb2159) [0x7f9119722159]\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 4: (librados::IoCtx::aio_write(std::string const&, librados::AioCompletion*, ceph::buffer::list const&, unsigned long, unsigned long)+0x44) [0x7f91196e6bf4]\r\n2015-08-25T21:16:27.894 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 5: (ObjBencher::write_bench(int, int, std::string const&)+0x53d) [0x7f911bed7d7d]\r\n2015-08-25T21:16:27.895 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 6: (ObjBencher::aio_bench(int, int, int, int, bool, std::string const&, bool)+0x20a) [0x7f911bed916a]\r\n2015-08-25T21:16:27.895 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 7: (()+0x1debc) [0x7f911bebdebc]\r\n2015-08-25T21:16:27.895 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 8: (main()+0x2ca) [0x7f911beb11fa]\r\n2015-08-25T21:16:27.896 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 9: (__libc_start_main()+0xf5) [0x7f9115da2ec5]\r\n2015-08-25T21:16:27.896 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 10: (()+0x132a7) [0x7f911beb32a7]\r\n2015-08-25T21:16:27.896 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.\r\n2015-08-25T21:16:27.896 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'\r\n2015-08-25T21:16:27.897 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:*** Caught signal (Aborted) **\r\n2015-08-25T21:16:27.897 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: in thread 7f911c0cf840\r\n2015-08-25T21:16:27.897 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: ceph version 9.0.3-879-g09b1303 (09b1303a19d2aaeca4e44585d4724ef3f92f4d77)\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 1: (()+0x407ea) [0x7f911bee07ea]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 2: (()+0x10340) [0x7f9116b7b340]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 3: (gsignal()+0x39) [0x7f9115db7f89]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 4: (abort()+0x148) [0x7f9115dbb398]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f91166c36b5]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 6: (()+0x5e836) [0x7f91166c1836]\r\n2015-08-25T21:16:27.898 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 7: (()+0x5e863) [0x7f91166c1863]\r\n2015-08-25T21:16:27.899 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 8: (()+0x5eaa2) [0x7f91166c1aa2]\r\n2015-08-25T21:16:27.899 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 9: (()+0x1583e8) [0x7f91197c83e8]\r\n2015-08-25T21:16:27.899 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 10: (()+0xadbda) [0x7f911971dbda]\r\n2015-08-25T21:16:27.899 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 11: (()+0xb2159) [0x7f9119722159]\r\n2015-08-25T21:16:27.899 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 12: (librados::IoCtx::aio_write(std::string const&, librados::AioCompletion*, ceph::buffer::list const&, unsigned long, unsigned long)+0x44) [0x7f91196e6bf4]\r\n2015-08-25T21:16:27.900 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 13: (ObjBencher::write_bench(int, int, std::string const&)+0x53d) [0x7f911bed7d7d]\r\n2015-08-25T21:16:27.900 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 14: (ObjBencher::aio_bench(int, int, int, int, bool, std::string const&, bool)+0x20a) [0x7f911bed916a]\r\n2015-08-25T21:16:27.900 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 15: (()+0x1debc) [0x7f911bebdebc]\r\n2015-08-25T21:16:27.900 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 16: (main()+0x2ca) [0x7f911beb11fa]\r\n2015-08-25T21:16:27.900 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 17: (__libc_start_main()+0xf5) [0x7f9115da2ec5]\r\n2015-08-25T21:16:27.901 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: 18: (()+0x132a7) [0x7f911beb32a7]\r\n2015-08-25T21:16:27.901 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:2015-08-25 21:16:27.889101 7f911c0cf840 -1 *** Caught signal (Aborted) **\r\n2015-08-25T21:16:27.901 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr: in thread 7f911c0cf840\r\n2015-08-25T21:16:27.901 INFO:tasks.radosbench.radosbench.0.burnupi59.stderr:\r\n"}}