{"action":"opened","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/694","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/694/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/694/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/694/events","html_url":"https://github.com/scylladb/scylla/issues/694","id":123597195,"number":694,"title":"Failure starting scylla on debug builds","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-23T03:36:17Z","updated_at":"2015-12-23T03:36:17Z","closed_at":null,"body":"We have verified the following errors starting scylla on CCM with the debug build:\r\n\r\n    Error starting scylla\r\n    Waiting for start script '/home/foobar/.ccm/scylla-1/node1/bin/run.sh /home/foobar/.ccm/scylla-1/node1 --api-address 127.0.0.1 --collectd-hostname lmr-XPS-13-9350.node1 --smp 1 --memory 512M --default-log-level info --collectd 0 --cpuset 0' to end\r\n    output:\r\n    + i=0\r\n    + + cat /home/lmr/.ccm/scylla-1/node1/conf/cassandra.yaml\r\n    grep listen_address\r\n    + cut -f2 -d \r\n    + ip=127.0.0.1\r\n    + + cat /home/lmr/.ccm/scylla-1/node1/node.conf\r\n    grep jmx_port\r\n    + cut -f2 -d'\r\n    + jmx_port=7100\r\n    + echo \r\n    + export SCYLLA_HOME=/home/lmr/.ccm/scylla-1/node1\r\n    + shift\r\n    + sleep 2\r\n    + + tee -a /home/lmr/.ccm/scylla-1/node1/logs/system.log\r\n    exec /home/lmr/.ccm/scylla-1/node1/bin/scylla --options-file /home/lmr/.ccm/scylla-1/node1/conf/scylla.yaml --api-address 127.0.0.1 --collectd-hostname lmr-XPS-13-9350.node1 --smp 1 --memory 512M --default-log-level info --collectd 0 --cpuset 0\r\n    /usr/include/boost/smart_ptr/detail/shared_count.hpp:443:36: runtime error: member call on address 0x60300000d840 which does not point to an object of type 'sp_counted_base'\r\n    0x60300000d840: note: object is of type 'boost::detail::sp_counted_impl_p<boost::program_options::option_description>'\r\n     02 00 80 79  90 62 4e f6 f8 7f 00 00  03 00 00 00 01 00 00 00  80 b6 00 00 c0 60 00 00  00 00 00 00\r\n                  ^~~~~~~~~~~~~~~~~~~~~~~\r\n                  vptr for 'boost::detail::sp_counted_impl_p<boost::program_options::option_description>'\r\n    /usr/include/boost/smart_ptr/detail/shared_count.hpp:454:41: runtime error: member call on address 0x60300000dd50 which does not point to an object of type 'sp_counted_base'\r\n    0x60300000dd50: note: object is of type 'boost::detail::sp_counted_impl_p<boost::program_options::option_description>'\r\n     02 00 80 79  90 62 4e f6 f8 7f 00 00  01 00 00 00 01 00 00 00  40 bd 00 00 c0 60 00 00  00 00 00 00\r\n                  ^~~~~~~~~~~~~~~~~~~~~~~\r\n                  vptr for 'boost::detail::sp_counted_impl_p<boost::program_options::option_description>'\r\n    /usr/include/cryptopp/cryptlib.h:530:39: runtime error: member call on address 0x7ffe8ae24880 which does not point to an object of type 'Algorithm'\r\n    0x7ffe8ae24880: note: object is of type 'CryptoPP::HashTransformation'\r\n     00 00 00 00  a0 45 5b f5 f8 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00\r\n                  ^~~~~~~~~~~~~~~~~~~~~~~\r\n                  vptr for 'CryptoPP::HashTransformation'\r\n    ASAN:SIGSEGV\r\n    =================================================================\r\n    ==10073==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000325 (pc 0x7ff8f66f501a bp 0x7ffe8ae24400 sp 0x7ffe8ae23ee0 T0)\r\n        #0 0x7ff8f66f5019  (/usr/lib/x86_64-linux-gnu/libubsan.so.0+0xa019)\r\n        #1 0x7ff8f66f44d5  (/usr/lib/x86_64-linux-gnu/libubsan.so.0+0x94d5)\r\n        #2 0x7ff8f66f4d12 in __ubsan_handle_dynamic_type_cache_miss (/usr/lib/x86_64-linux-gnu/libubsan.so.0+0x9d12)\r\n        #3 0xab8dc4 in boost::io::basic_oaltstringstream<char, std::char_traits<char>, std::allocator<char> >::basic_oaltstringstream(boost::io::basic_altstringbuf<char, std::char_traits<char>, std::allocator<char> >*) /usr/include/boost/format/alt_sstream.hpp:146\r\n        #4 0xa92b25 in void boost::io::detail::put<char, std::char_traits<char>, std::allocator<char>, boost::io::detail::put_holder<char, std::char_traits<char> > const&>(boost::io::detail::put_holder<char, std::char_traits<char> > const&, boost::io::detail::format_item<char, std::char_traits<char>, std::allocator<char> > const&, boost::basic_format<char, std::char_traits<char>, std::allocator<char> >::string_type&, boost::basic_format<char, std::char_traits<char>, std::allocator<char> >::internal_streambuf_t&, std::locale*) /usr/include/boost/format/feed_args.hpp:174\r\n        #5 0xa75937 in void boost::io::detail::distribute<char, std::char_traits<char>, std::allocator<char>, boost::io::detail::put_holder<char, std::char_traits<char> > const&>(boost::basic_format<char, std::char_traits<char>, std::allocator<char> >&, boost::io::detail::put_holder<char, std::char_traits<char> > const&) /usr/include/boost/format/feed_args.hpp:285\r\n        #6 0xa5550d in boost::basic_format<char, std::char_traits<char>, std::allocator<char> >& boost::io::detail::feed_impl<char, std::char_traits<char>, std::allocator<char>, boost::io::detail::put_holder<char, std::char_traits<char> > const&>(boost::basic_format<char, std::char_traits<char>, std::allocator<char> >&, boost::io::detail::put_holder<char, std::char_traits<char> > const&) /usr/include/boost/format/feed_args.hpp:295\r\n        #7 0x2b72113 in boost::basic_format<char, std::char_traits<char>, std::allocator<char> >& boost::io::detail::feed<char, std::char_traits<char>, std::allocator<char>, char const (&) [12]>(boost::basic_format<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [12]) /usr/include/boost/format/feed_args.hpp:307\r\n        #8 0x2b71e7a in boost::basic_format<char, std::char_traits<char>, std::allocator<char> >& boost::basic_format<char, std::char_traits<char>, std::allocator<char> >::operator%<char [12]>(char const (&) [12]) /usr/include/boost/format/format_class.hpp:64\r\n        #9 0x2b71d73 in void apply_format<char const (&) [12], char const (&) [17]>(boost::basic_format<char, std::char_traits<char>, std::allocator<char> >&, char const (&) [12], char const (&) [17]) /home/lmr/Code/scylla/seastar/core/print.hh:38\r\n        #10 0x2b71a58 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > sprint<char const (&) [12], char const (&) [17]>(char const*, char const (&) [12], char const (&) [17]) (/dtest/.ccm/scylla-1/node1/bin/scylla+0x2b71a58)\r\n        #11 0x2b7128e in scylla_version[abi:cxx11]() /home/lmr/Code/scylla/release.cc:31\r\n        #12 0x2b71424 in __static_initialization_and_destruction_0 /home/lmr/Code/scylla/release.cc:36\r\n        #13 0x2b71766 in _GLOBAL__sub_I_release.cc /home/lmr/Code/scylla/release.cc:36\r\n        #14 0xbf4566c in __libc_csu_init (/dtest/.ccm/scylla-1/node1/bin/scylla+0xbf4566c)\r\n        #15 0x7ff8f30329ce in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x209ce)\r\n        #16 0x4a09f8 in _start (/dtest/.ccm/scylla-1/node1/bin/scylla+0x4a09f8)\r\n\r\n    AddressSanitizer can not provide additional info.\r\n    SUMMARY: AddressSanitizer: SEGV ??:0 ??\r\n    ==10073==ABORTING\r\n    + pkill -f com.sun.management.jmxremote.port=7100\r\n    + true\r\n    + sleep 2\r\n    + sleep 6\r\n    + exec java -Dapiaddress=127.0.0.1 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7100 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar /home/lmr/.ccm/scylla-1/node1/bin/scylla-jmx-1.0.jar\r\n    + tee -a /home/lmr/.ccm/scylla-1/node1/logs/system.log.jmx\r\n    Using config file: /home/lmr/.ccm/scylla-1/node1/conf/scylla.yaml\r\n    Connecting to http://127.0.0.1:10000\r\n    Starting the JMX server\r\n    SLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".\r\n    SLF4J: Defaulting to no-operation (NOP) logger implementation\r\n    SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\r\n     getDroppedMessages()\r\n    Dec 23, 2015 1:12:42 AM org.apache.cassandra.net.MessagingService log\r\n    INFO:  getDroppedMessages()\r\n    Exception in thread \"Dropped messages\" javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused\r\n    \tat org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287)\r\n    \tat org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:255)\r\n    \tat org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:700)\r\n    \tat org.glassfish.jersey.internal.Errors.process(Errors.java:315)\r\n    \tat org.glassfish.jersey.internal.Errors.process(Errors.java:297)\r\n    \tat org.glassfish.jersey.internal.Errors.process(Errors.java:228)\r\n    \tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)\r\n    \tat org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:696)\r\n    \tat org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:420)\r\n    \tat org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:316)\r\n    \tat com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:152)\r\n    \tat com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:169)\r\n    \tat com.scylladb.jmx.api.APIClient.getReader(APIClient.java:196)\r\n    \tat com.scylladb.jmx.api.APIClient.getJsonArray(APIClient.java:600)\r\n    \tat com.scylladb.jmx.api.APIClient.getJsonArray(APIClient.java:607)\r\n    \tat org.apache.cassandra.net.MessagingService.getDroppedMessages(MessagingService.java:192)\r\n    \tat org.apache.cassandra.net.MessagingService$CheckDroppedMessages.run(MessagingService.java:130)\r\n    \tat java.util.TimerThread.mainLoop(Timer.java:555)\r\n    \tat java.util.TimerThread.run(Timer.java:505)\r\n    Caused by: java.net.ConnectException: Connection refused\r\n    \tat java.net.PlainSocketImpl.socketConnect(Native Method)\r\n    \tat java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)\r\n    \tat java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)\r\n    \tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)\r\n    \tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\r\n    \tat java.net.Socket.connect(Socket.java:579)\r\n    \tat sun.net.NetworkClient.doConnect(NetworkClient.java:175)\r\n    \tat sun.net.www.http.HttpClient.openServer(HttpClient.java:432)\r\n    \tat sun.net.www.http.HttpClient.openServer(HttpClient.java:527)\r\n    \tat sun.net.www.http.HttpClient.<init>(HttpClient.java:211)\r\n    \tat sun.net.www.http.HttpClient.New(HttpClient.java:308)\r\n    \tat sun.net.www.http.HttpClient.New(HttpClient.java:326)\r\n    \tat sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)\r\n    \tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)\r\n    \tat sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)\r\n    \tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)\r\n    \tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)\r\n    \tat org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:394)\r\n    \tat org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285)\r\n    \t... 18 more\r\n\r\n    Error starting scylla node\r\n\r\nSteps to reproduce\r\n\r\n1. Build scylla with the debug build\r\n\r\n    ./configure.py --mode debug\r\n    ninja -v\r\n\r\n2. Create a scylla cluster on ccm:\r\n\r\n    ccm create scylla-3 --scylla --vnodes -n 3 --install-dir=/path/to/scylla-source/build/debug/\r\n    ccm start\r\n\r\nThat will trigger the bug."}}