{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak/issues/790","labels_url":"https://api.github.com/repos/basho/riak/issues/790/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak/issues/790/comments","events_url":"https://api.github.com/repos/basho/riak/issues/790/events","html_url":"https://github.com/basho/riak/issues/790","id":115897857,"number":790,"title":"Riak doesn't start anymore - Waiting for service riak_kv to start (540 seconds) [JIRA: RIAK-2273]","user":{"login":"Zerebokep","id":15731988,"avatar_url":"https://avatars.githubusercontent.com/u/15731988?v=3","gravatar_id":"","url":"https://api.github.com/users/Zerebokep","html_url":"https://github.com/Zerebokep","followers_url":"https://api.github.com/users/Zerebokep/followers","following_url":"https://api.github.com/users/Zerebokep/following{/other_user}","gists_url":"https://api.github.com/users/Zerebokep/gists{/gist_id}","starred_url":"https://api.github.com/users/Zerebokep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zerebokep/subscriptions","organizations_url":"https://api.github.com/users/Zerebokep/orgs","repos_url":"https://api.github.com/users/Zerebokep/repos","events_url":"https://api.github.com/users/Zerebokep/events{/privacy}","received_events_url":"https://api.github.com/users/Zerebokep/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/basho/riak/labels/JIRA:%20To%20Do","name":"JIRA: To Do","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-11-09T15:17:43Z","updated_at":"2015-11-10T13:36:49Z","closed_at":null,"body":"After a server crash riak doesn't seem to start anymore. There are no errors logged.\r\n\r\nHere is the console.log:\r\n\r\n2015-11-09 16:04:31.030 [info] <0.7.0> Application lager started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.038 [info] <0.7.0> Application sasl started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.039 [info] <0.7.0> Application asn1 started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.040 [info] <0.7.0> Application crypto started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.040 [info] <0.7.0> Application public_key started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.044 [info] <0.7.0> Application ssl started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.046 [info] <0.7.0> Application riak_sysmon started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.051 [info] <0.7.0> Application os_mon started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.054 [info] <0.73.0> alarm_handler: {set,{{disk_almost_full,\"/\"},[]}}\r\n2015-11-09 16:04:31.059 [info] <0.7.0> Application runtime_tools started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.068 [info] <0.7.0> Application erlang_js started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.068 [info] <0.7.0> Application xmerl started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.082 [info] <0.7.0> Application inets started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.084 [info] <0.7.0> Application mochiweb started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.089 [info] <0.7.0> Application webmachine started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.089 [info] <0.7.0> Application basho_stats started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.093 [info] <0.7.0> Application bitcask started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.101 [info] <0.7.0> Application clique started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.101 [info] <0.7.0> Application eleveldb started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.101 [info] <0.7.0> Application pbkdf2 started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.101 [info] <0.7.0> Application poolboy started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.120 [info] <0.161.0>@exometer_report:do_start_reporters:613 Starting reporters with []\r\n2015-11-09 16:04:31.120 [info] <0.7.0> Application exometer_core started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.254 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,vnode_routing} = proxy\r\n2015-11-09 16:04:31.261 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,staged_joins} = true\r\n2015-11-09 16:04:31.269 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,resizable_ring} = true\r\n2015-11-09 16:04:31.285 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,fold_req_version} = v2\r\n2015-11-09 16:04:31.301 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,security} = true\r\n2015-11-09 16:04:31.317 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,bucket_types} = true\r\n2015-11-09 16:04:31.334 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,net_ticktime} = true\r\n2015-11-09 16:04:31.426 [info] <0.7.0> Application riak_core started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.430 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_pipe,trace_format} = ordsets\r\n2015-11-09 16:04:31.439 [notice] <0.47.0>@lager_app:132 Deprecated lager_file_backend config detected, please consider updating it\r\n2015-11-09 16:04:31.442 [info] <0.7.0> Application riak_pipe started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.444 [info] <0.7.0> Application sidejob started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.445 [info] <0.7.0> Application riak_dt started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.445 [info] <0.7.0> Application protobuffs started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.445 [info] <0.7.0> Application riak_pb started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.450 [warning] <0.305.0>@riak_api_web:get_listeners:40 Setting riak_core/http is deprecated, please use riak_api/http\r\n2015-11-09 16:04:31.465 [info] <0.7.0> Application riak_api started on node 'riak@127.0.0.1'\r\n2015-11-09 16:04:31.494 [info] <0.331.0>@riak_kv_env:doc_env:46 Environment and OS variables:\r\n2015-11-09 16:04:31.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (0 seconds)\r\n2015-11-09 16:04:31.549 [info] <0.385.0>@riak_kv_entropy_manager:set_aae_throttle_limits:790 Setting AAE throttle limits: [{-1,0},{200,10},{500,50},{750,250},{900,1000},{1100,5000}]\r\n2015-11-09 16:04:31.586 [info] <0.394.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.394.0>)\r\n2015-11-09 16:04:31.588 [info] <0.395.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.395.0>)\r\n2015-11-09 16:04:31.591 [info] <0.396.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.396.0>)\r\n2015-11-09 16:04:31.593 [info] <0.397.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.397.0>)\r\n2015-11-09 16:04:31.596 [info] <0.398.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.398.0>)\r\n2015-11-09 16:04:31.598 [info] <0.399.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.399.0>)\r\n2015-11-09 16:04:31.600 [info] <0.400.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.400.0>)\r\n2015-11-09 16:04:31.603 [info] <0.401.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.401.0>)\r\n2015-11-09 16:04:31.606 [info] <0.403.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.403.0>)\r\n2015-11-09 16:04:31.609 [info] <0.404.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.404.0>)\r\n2015-11-09 16:04:31.612 [info] <0.405.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.405.0>)\r\n2015-11-09 16:04:31.615 [info] <0.406.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.406.0>)\r\n2015-11-09 16:04:31.618 [info] <0.407.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.407.0>)\r\n2015-11-09 16:04:31.622 [info] <0.408.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.408.0>)\r\n2015-11-09 16:04:31.625 [info] <0.410.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting (<0.410.0>)\r\n2015-11-09 16:04:31.628 [info] <0.411.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting (<0.411.0>)\r\n2015-11-09 16:04:31.632 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,vnode_vclocks} = true\r\n2015-11-09 16:04:31.634 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,legacy_keylisting} = false\r\n2015-11-09 16:04:31.635 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,listkeys_backpressure} = true\r\n2015-11-09 16:04:31.637 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,index_backpressure} = true\r\n2015-11-09 16:04:31.639 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,mapred_system} = pipe\r\n2015-11-09 16:04:31.646 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,mapred_2i_pipe} = true\r\n2015-11-09 16:04:31.648 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,anti_entropy} = enabled_v1\r\n2015-11-09 16:04:31.650 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,handoff_data_encoding} = encode_raw\r\n2015-11-09 16:04:31.652 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,object_format} = v1\r\n2015-11-09 16:04:31.653 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,secondary_index_version} = v3\r\n2015-11-09 16:04:31.655 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,vclock_data_encoding} = encode_zlib\r\n2015-11-09 16:04:31.657 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,crdt} = [pncounter,riak_dt_pncounter,riak_dt_orswot,riak_dt_map]\r\n2015-11-09 16:04:31.659 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,crdt_epoch_versions} = [{riak_dt_map,2},{riak_dt_orswot,2},{riak_dt_pncounter,2}]\r\n2015-11-09 16:04:31.661 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,put_fsm_ack_execute} = enabled\r\n2015-11-09 16:04:31.705 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_control,member_info_version} = v1\r\n2015-11-09 16:05:32.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (60 seconds)\r\n2015-11-09 16:06:32.739 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (120 seconds)\r\n2015-11-09 16:07:33.339 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (180 seconds)\r\n2015-11-09 16:08:33.939 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (240 seconds)\r\n2015-11-09 16:09:34.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (300 seconds)\r\n2015-11-09 16:10:35.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (360 seconds)\r\n2015-11-09 16:11:35.739 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (420 seconds)\r\n2015-11-09 16:12:36.339 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (480 seconds)\r\n2015-11-09 16:13:36.939 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (540 seconds)\r\n2015-11-09 16:14:37.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (600 seconds)\r\n2015-11-09 16:15:38.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (660 seconds)"},"comment":{"url":"https://api.github.com/repos/basho/riak/issues/comments/155420692","html_url":"https://github.com/basho/riak/issues/790#issuecomment-155420692","issue_url":"https://api.github.com/repos/basho/riak/issues/790","id":155420692,"user":{"login":"Zerebokep","id":15731988,"avatar_url":"https://avatars.githubusercontent.com/u/15731988?v=3","gravatar_id":"","url":"https://api.github.com/users/Zerebokep","html_url":"https://github.com/Zerebokep","followers_url":"https://api.github.com/users/Zerebokep/followers","following_url":"https://api.github.com/users/Zerebokep/following{/other_user}","gists_url":"https://api.github.com/users/Zerebokep/gists{/gist_id}","starred_url":"https://api.github.com/users/Zerebokep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zerebokep/subscriptions","organizations_url":"https://api.github.com/users/Zerebokep/orgs","repos_url":"https://api.github.com/users/Zerebokep/repos","events_url":"https://api.github.com/users/Zerebokep/events{/privacy}","received_events_url":"https://api.github.com/users/Zerebokep/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T13:36:49Z","updated_at":"2015-11-10T13:36:49Z","body":"That's weird, even the status commands are telling me that everything is fine.\r\n\r\n# riak-admin member-status\r\n================================= Membership ==================================\r\nStatus     Ring    Pending    Node\r\n-------------------------------------------------------------------------------\r\nvalid     100.0%      --      'riak@127.0.0.1'\r\n-------------------------------------------------------------------------------\r\nValid:1 / Leaving:0 / Exiting:0 / Joining:0 / Down:0\r\n\r\n\r\n# riak-admin ring-status\r\n================================== Claimant ===================================\r\nClaimant:  'riak@127.0.0.1'\r\nStatus:     up\r\nRing Ready: true\r\n\r\n============================== Ownership Handoff ==============================\r\nNo pending changes.\r\n\r\n============================== Unreachable Nodes ==============================\r\nAll nodes are up and reachable\r\n"}}