{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/598","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/598/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/598/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/598/events","html_url":"https://github.com/scylladb/scylla/issues/598","id":118395057,"number":598,"title":"Scylla core dump on EC2, reporting Checksum error","user":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-23T14:41:20Z","updated_at":"2015-11-23T14:43:40Z","closed_at":null,"body":"Using Scylla AMI 0.12\r\nA stress run result in scylla service goes down\r\n\r\nLast messages on journal\r\n```\r\nd)\r\nNov 23 14:36:33 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130808.log complete, 0 replayed mutations (0 invalid, 0 skipped)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130801.log complete, 7192 replayed mutations (0 invalid, 0 skipped)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-18014398510612785.log: std::runtime_error (Checksum error in data entry)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-1130800.log: std::runtime_error (Checksum error in data entry)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-18014398510612786.log complete, 10688 replayed mutations (0 invalid, 0 skipped)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-1130798.log: std::runtime_error (Checksum error in data entry)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-18014398510612784.log: std::runtime_error (Checksum error in data entry)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130799.log complete, 6815 replayed mutations (0 invalid, 0 skipped)\r\nNov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: Exiting on unhandled exception of type 'std::runtime_error': Checksum error in data entry\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] compaction_manager - compaction task handler stopped due to shutdown\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] compaction_manager - compaction task handler stopped due to shutdown\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 1] compaction_manager - compaction task handler stopped due to shutdown\r\nNov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 1] compaction_manager - compaction task handler stopped due to shutdown\r\nNov 23 14:36:34 ip-10-16-211-188 sudo[10715]: pam_unix(sudo:session): session closed for user scylla\r\n```\r\n\r\ncore sump\r\n```\r\n      Message: Process 2927 (scylla) of user 995 dumped core.\r\n                \r\n                Stack trace of thread 2927:\r\n                #0  0x00007f7da64d19c8 raise (libc.so.6)\r\n                #1  0x00007f7da64d365a abort (libc.so.6)\r\n                #2  0x0000000000deb2d9 _ZN6futureIJEE12then_wrappedIZNS0_7finallyIZ7do_withI13basic_sstringIcjLj15EEZZN3rpc11send_helperIN3net10serializerENS8_14messaging_verbENS6_12no_\r\n                #3  0x0000000000e53080 _ZZN3rpc11send_helperIN3net10serializerENS1_14messaging_verbENS_12no_wait_typeEJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaISA_EESA_jmEEED\r\n                #4  0x0000000000e533c5 _ZZN3rpc11send_helperIN3net10serializerENS1_14messaging_verbENS_12no_wait_typeEJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaISA_EESA_jmEEED\r\n                #5  0x0000000000d94d5f _ZN3net19send_message_onewayIJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaIS6_EES6_jmEEEDaPNS_17messaging_serviceENS_14messaging_verbENS_8s\r\n                #6  0x0000000000aa84b4 _ZZZZN7service13storage_proxy22init_messaging_serviceEvENKUl15frozen_mutationSt6vectorIN3gms12inet_addressESaIS4_EES4_jmE1_clES1_S6_S4_jmENKUlRKS1\r\n                #7  0x0000000000aaa2bf apply (scylla)\r\n                #8  0x0000000000e36e64 _ZZN3rpc11recv_helperIN3net10serializerENS1_14messaging_verbESt8functionIFNS_12no_wait_typeE15frozen_mutationSt6vectorIN3gms12inet_addressESaIS9_E\r\n                #9  0x0000000000e37218 _ZNSt17_Function_handlerIFv13lw_shared_ptrIN3rpc8protocolIN3net10serializerENS3_14messaging_verbEE6server10connectionEEl16temporary_bufferIcEEZNS1\r\n                #10 0x0000000000e2d26d _ZNKSt8functionIFv13lw_shared_ptrIN3rpc8protocolIN3net10serializerENS3_14messaging_verbEE6server10connectionEEl16temporary_bufferIcEEEclES9_lSB_ (\r\n                #11 0x0000000000e2d408 _ZN8futurizeIvE5applyIZZN3rpc8protocolIN3net10serializerENS4_14messaging_verbEE6server10connection7processEvENUlvE0_clEvEUlS6_l16temporary_bufferI\r\n                #12 0x00000000004757cd _ZN7reactor9run_tasksER15circular_bufferISt10unique_ptrI4taskSt14default_deleteIS2_EESaIS5_EE (scylla)\r\n                #13 0x000000000049e9b0 _ZN7reactor3runEv (scylla)\r\n                #14 0x00000000004fd174 _ZN12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla)\r\n                #15 0x000000000041d23b main (scylla)\r\n                #16 0x00007f7da64bd700 __libc_start_main (libc.so.6)\r\n                #17 0x0000000000472749 _start (scylla)\r\n                \r\n                Stack trace of thread 2928:\r\n                #0  0x00007f7da65a0193 epoll_wait (libc.so.6)\r\n                #1  0x0000000000645c9c eal_intr_thread_main (scylla)\r\n                #2  0x00007f7da6864555 start_thread (libpthread.so.0)\r\n                #3  0x00007f7da659fb9d __clone (libc.so.6)\r\n                \r\n                Stack trace of thread 2938:\r\n                #0  0x00007f7da686c54d read (libpthread.so.0)\r\n                #1  0x0000000000475e95 _ZN11thread_pool4workEv (scylla)\r\n                #2  0x00000000004eb08e _ZNKSt8functionIFvvEEclEv (scylla)\r\n                #3  0x00007f7da6864555 start_thread (libpthread.so.0)\r\n                #4  0x00007f7da659fb9d __clone (libc.so.6)\r\n\r\n              Stack trace of thread 2937:\r\n                #0  0x00007f7da686c54d read (libpthread.so.0)\r\n                #1  0x0000000000475e95 _ZN11thread_pool4workEv (scylla)\r\n                #2  0x00000000004eb08e _ZNKSt8\r\n```"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/158956328","html_url":"https://github.com/scylladb/scylla/issues/598#issuecomment-158956328","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/598","id":158956328,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T14:43:39Z","updated_at":"2015-11-23T14:43:39Z","body":"isn't this a dup of https://github.com/scylladb/scylla/issues/593\n\n@tzach please note that you are replaying commitlogs - so it mean the\nserver was stopped/killed and then restarted - is that by intent.\n\nOn Mon, Nov 23, 2015 at 4:41 PM, Tzach Livyatan <notifications@github.com>\nwrote:\n\n> Using Scylla AMI 0.12\n> A stress run result in scylla service goes down\n>\n> Last messages on journal\n>\n> d)\n> Nov 23 14:36:33 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130808.log complete, 0 replayed mutations (0 invalid, 0 skipped)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130801.log complete, 7192 replayed mutations (0 invalid, 0 skipped)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-18014398510612785.log: std::runtime_error (Checksum error in data entry)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-1130800.log: std::runtime_error (Checksum error in data entry)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-18014398510612786.log complete, 10688 replayed mutations (0 invalid, 0 skipped)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-1130798.log: std::runtime_error (Checksum error in data entry)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Error recovering /data/commitlog/CommitLog-1-18014398510612784.log: std::runtime_error (Checksum error in data entry)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: WARNING: exceptional future ignored of type 'std::runtime_error': Checksum error in data entry\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] commitlog_replayer - Log replay of /data/commitlog/CommitLog-1-1130799.log complete, 6815 replayed mutations (0 invalid, 0 skipped)\n> Nov 23 14:36:34 ip-10-16-211-188 scylla_run[10715]: Exiting on unhandled exception of type 'std::runtime_error': Checksum error in data entry\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] compaction_manager - compaction task handler stopped due to shutdown\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 0] compaction_manager - compaction task handler stopped due to shutdown\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 1] compaction_manager - compaction task handler stopped due to shutdown\n> Nov 23 14:36:34 ip-10-16-211-188 scylla[10728]:  [shard 1] compaction_manager - compaction task handler stopped due to shutdown\n> Nov 23 14:36:34 ip-10-16-211-188 sudo[10715]: pam_unix(sudo:session): session closed for user scylla\n>\n> core sump\n>\n>       Message: Process 2927 (scylla) of user 995 dumped core.\n>\n>                 Stack trace of thread 2927:\n>                 #0  0x00007f7da64d19c8 raise (libc.so.6)\n>                 #1  0x00007f7da64d365a abort (libc.so.6)\n>                 #2  0x0000000000deb2d9 _ZN6futureIJEE12then_wrappedIZNS0_7finallyIZ7do_withI13basic_sstringIcjLj15EEZZN3rpc11send_helperIN3net10serializerENS8_14messaging_verbENS6_12no_\n>                 #3  0x0000000000e53080 _ZZN3rpc11send_helperIN3net10serializerENS1_14messaging_verbENS_12no_wait_typeEJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaISA_EESA_jmEEED\n>                 #4  0x0000000000e533c5 _ZZN3rpc11send_helperIN3net10serializerENS1_14messaging_verbENS_12no_wait_typeEJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaISA_EESA_jmEEED\n>                 #5  0x0000000000d94d5f _ZN3net19send_message_onewayIJRK15frozen_mutationSt6vectorIN3gms12inet_addressESaIS6_EES6_jmEEEDaPNS_17messaging_serviceENS_14messaging_verbENS_8s\n>                 #6  0x0000000000aa84b4 _ZZZZN7service13storage_proxy22init_messaging_serviceEvENKUl15frozen_mutationSt6vectorIN3gms12inet_addressESaIS4_EES4_jmE1_clES1_S6_S4_jmENKUlRKS1\n>                 #7  0x0000000000aaa2bf apply (scylla)\n>                 #8  0x0000000000e36e64 _ZZN3rpc11recv_helperIN3net10serializerENS1_14messaging_verbESt8functionIFNS_12no_wait_typeE15frozen_mutationSt6vectorIN3gms12inet_addressESaIS9_E\n>                 #9  0x0000000000e37218 _ZNSt17_Function_handlerIFv13lw_shared_ptrIN3rpc8protocolIN3net10serializerENS3_14messaging_verbEE6server10connectionEEl16temporary_bufferIcEEZNS1\n>                 #10 0x0000000000e2d26d _ZNKSt8functionIFv13lw_shared_ptrIN3rpc8protocolIN3net10serializerENS3_14messaging_verbEE6server10connectionEEl16temporary_bufferIcEEEclES9_lSB_ (\n>                 #11 0x0000000000e2d408 _ZN8futurizeIvE5applyIZZN3rpc8protocolIN3net10serializerENS4_14messaging_verbEE6server10connection7processEvENUlvE0_clEvEUlS6_l16temporary_bufferI\n>                 #12 0x00000000004757cd _ZN7reactor9run_tasksER15circular_bufferISt10unique_ptrI4taskSt14default_deleteIS2_EESaIS5_EE (scylla)\n>                 #13 0x000000000049e9b0 _ZN7reactor3runEv (scylla)\n>                 #14 0x00000000004fd174 _ZN12app_template14run_deprecatedEiPPcOSt8functionIFvvEE (scylla)\n>                 #15 0x000000000041d23b main (scylla)\n>                 #16 0x00007f7da64bd700 __libc_start_main (libc.so.6)\n>                 #17 0x0000000000472749 _start (scylla)\n>\n>                 Stack trace of thread 2928:\n>                 #0  0x00007f7da65a0193 epoll_wait (libc.so.6)\n>                 #1  0x0000000000645c9c eal_intr_thread_main (scylla)\n>                 #2  0x00007f7da6864555 start_thread (libpthread.so.0)\n>                 #3  0x00007f7da659fb9d __clone (libc.so.6)\n>\n>                 Stack trace of thread 2938:\n>                 #0  0x00007f7da686c54d read (libpthread.so.0)\n>                 #1  0x0000000000475e95 _ZN11thread_pool4workEv (scylla)\n>                 #2  0x00000000004eb08e _ZNKSt8functionIFvvEEclEv (scylla)\n>                 #3  0x00007f7da6864555 start_thread (libpthread.so.0)\n>                 #4  0x00007f7da659fb9d __clone (libc.so.6)\n>\n>               Stack trace of thread 2937:\n>                 #0  0x00007f7da686c54d read (libpthread.so.0)\n>                 #1  0x0000000000475e95 _ZN11thread_pool4workEv (scylla)\n>                 #2  0x00000000004eb08e _ZNKSt8\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/scylladb/scylla/issues/598>.\n>\n"}}