{"action":"created","issue":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5263","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5263/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5263/comments","events_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5263/events","html_url":"https://github.com/orientechnologies/orientdb/issues/5263","id":114608396,"number":5263,"title":"Database unresponsive, looks like twice creating the same index may have caused this.","user":{"login":"wouterv","id":616997,"avatar_url":"https://avatars.githubusercontent.com/u/616997?v=3","gravatar_id":"","url":"https://api.github.com/users/wouterv","html_url":"https://github.com/wouterv","followers_url":"https://api.github.com/users/wouterv/followers","following_url":"https://api.github.com/users/wouterv/following{/other_user}","gists_url":"https://api.github.com/users/wouterv/gists{/gist_id}","starred_url":"https://api.github.com/users/wouterv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wouterv/subscriptions","organizations_url":"https://api.github.com/users/wouterv/orgs","repos_url":"https://api.github.com/users/wouterv/repos","events_url":"https://api.github.com/users/wouterv/events{/privacy}","received_events_url":"https://api.github.com/users/wouterv/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/bug","name":"bug","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-02T14:47:45Z","updated_at":"2015-11-02T18:50:22Z","closed_at":null,"body":"Our database is unresponsive when it comes to writes. Looks like an index was created twice in a short period of time. The second time the index wasn't finished creating and our logs never show that the index is finished, neither does it show progress.\n\nThe following are the last relevant logrows on the servers (3 nodes). \nTest1:\n```\n2015-11-02 14:27:02:154 FINE  [test1-dev]<-[test3-dev] processing request=id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:27:02:154 FINE  [test1-dev]->[test3-dev] received request: id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 [ODistributedWorker]\n2015-11-02 14:27:02:202 FINE  Created record #1:668 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:02:203 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: 0 [OFileClassic]\n2015-11-02 14:27:02:234 FINE  Created record #1:669 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:02:235 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: 0 [OFileClassic]\n2015-11-02 14:27:02:236 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n...\n2015-11-02 14:27:02:633 FINE  Updated record #0:2 v.1336 size=184884 [OLocalPaginatedStorage]\n2015-11-02 14:27:02:939 FINE  [test1-dev]->[test3-dev] sending back response '0' to request 4908 (command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING)) [ODistributedWorker]\n2015-11-02 14:27:03:728 FINE  [test1-dev]<-[test3-dev] processing request=id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels) user=#5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:27:03:728 FINE  [test1-dev]->[test3-dev] received request: id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels) user=#5:0 [ODistributedWorker]\n2015-11-02 14:27:03:730 FINE  Created record #1:670 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:03:731 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: 0 [OFileClassic]\n2015-11-02 14:27:03:732 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n```\n\nTest2:\n```\n2015-11-02 14:27:01:675 FINE  [test2-dev]->[test3-dev] sending back response '8' to request 4907 (command_sql(create property BDE_kanters_nl_4_artikel.OmschrijvingEngels String)\n) [ODistributedWorker]\n2015-11-02 14:27:02:151 FINE  [test2-dev]<-[test3-dev] processing request=id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEn\ngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:27:02:151 FINE  [test2-dev]->[test3-dev] received request: id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEng\nels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 [ODistributedWorker]\n2015-11-02 14:27:02:165 FINE  Created record #1:668 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:02:166 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:27:02:171 FINE  Created record #1:669 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:02:171 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:27:02:172 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n...\n2015-11-02 14:27:02:513 FINE  Updated record #0:2 v.1336 size=184884 [OLocalPaginatedStorage]\n2015-11-02 14:27:02:758 FINE  [test2-dev]->[test3-dev] sending back response '0' to request 4908 (command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels\n on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING)) [ODistributedWorker]\n2015-11-02 14:27:03:726 FINE  [test2-dev]<-[test3-dev] processing request=id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingE\nngels) user=#5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:27:03:726 FINE  [test2-dev]->[test3-dev] received request: id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEn\ngels) user=#5:0 [ODistributedWorker]\n2015-11-02 14:27:03:728 FINE  Created record #1:670 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:27:03:728 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:27:03:729 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n```\nTest3:\n```\n2015-11-02 14:26:41:084 FINE  Updated record #0:1 v.17272 size=807982 [OLocalPaginatedStorage]\n2015-11-02 14:26:41:404 FINE  [test3-dev] sending back response '8' to request 4907 (command_sql(create property BDE_kanters_nl_4_artikel.OmschrijvingEngels String)) [ODistribut\nedWorker]\n2015-11-02 14:26:41:700 FINE  Read record #0:1 v.17272 size=807982 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:41:797 FINE  Read record #0:1 v.17272 size=807982 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:41:994 FINE  [test3-dev]->[[test3-dev, test2-dev, test1-dev]] sending request id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_a\nrtikel_OmschrijvingEngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 [OHazelcastDistributedDatabase]\n2015-11-02 14:26:41:996 FINE  [test3-dev] processing request=id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_k\nanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:26:41:996 FINE  [test3-dev] received request: id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_ka\nnters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 [ODistributedWorker]\n2015-11-02 14:26:42:001 FINE  [test3-dev]->[[test3-dev, test2-dev, test1-dev]] sent request id=4908 from=test3-dev task=command_sql(create index BDE_unique_BDE_kanters_nl_4_arti\nkel_OmschrijvingEngels on BDE_kanters_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING) user=#5:0 [OHazelcastDistributedDatabase]\n2015-11-02 14:26:42:041 FINE  Created record #1:668 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:42:114 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:26:42:142 FINE  Created record #1:669 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:42:143 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:26:42:144 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n...\n2015-11-02 14:26:42:934 FINE  Updated record #0:2 v.1336 size=184884 [OLocalPaginatedStorage]\n2015-11-02 14:26:43:541 FINE  [test3-dev] sending back response '0' to request 4908 (command_sql(create index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels on BDE_kante\nrs_nl_4_artikel ( OmschrijvingEngels ) UNIQUE STRING)) [ODistributedWorker]\n2015-11-02 14:26:43:544 FINE  Read record #0:2 v.1336 size=184884 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:43:569 FINE  [test3-dev]->[[test3-dev, test2-dev, test1-dev]] sending request id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_\nartikel_OmschrijvingEngels) user=#5:0 [OHazelcastDistributedDatabase]\n2015-11-02 14:26:43:571 FINE  [test3-dev] processing request=id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels) user=#\n5:0 sourceNode=test3-dev [ODistributedWorker]\n2015-11-02 14:26:43:571 FINE  [test3-dev] received request: id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels) user=#5\n:0 [ODistributedWorker]\n2015-11-02 14:26:43:574 FINE  Created record #1:670 v.1 size=0 bytes [OLocalPaginatedStorage]\n2015-11-02 14:26:43:574 FINE  [OFile.openChannel] opening channel for file '/opt/orientdb/databases/beeyond/BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels.sbt' of size: \n0 [OFileClassic]\n2015-11-02 14:26:43:575 FINE  [test3-dev]->[[test3-dev, test2-dev, test1-dev]] sent request id=4909 from=test3-dev task=command_sql(rebuild index BDE_unique_BDE_kanters_nl_4_art\nikel_OmschrijvingEngels) user=#5:0 [OHazelcastDistributedDatabase]\n2015-11-02 14:26:43:576 INFO  - Rebuilding index beeyond.BDE_unique_BDE_kanters_nl_4_artikel_OmschrijvingEngels (estimated 6,444 items)... [OIndexRebuildOutputListener]\n```\n\nI can email logs with threaddumps"},"comment":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/comments/153122451","html_url":"https://github.com/orientechnologies/orientdb/issues/5263#issuecomment-153122451","issue_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5263","id":153122451,"user":{"login":"lvca","id":312606,"avatar_url":"https://avatars.githubusercontent.com/u/312606?v=3","gravatar_id":"","url":"https://api.github.com/users/lvca","html_url":"https://github.com/lvca","followers_url":"https://api.github.com/users/lvca/followers","following_url":"https://api.github.com/users/lvca/following{/other_user}","gists_url":"https://api.github.com/users/lvca/gists{/gist_id}","starred_url":"https://api.github.com/users/lvca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvca/subscriptions","organizations_url":"https://api.github.com/users/lvca/orgs","repos_url":"https://api.github.com/users/lvca/repos","events_url":"https://api.github.com/users/lvca/events{/privacy}","received_events_url":"https://api.github.com/users/lvca/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T18:50:22Z","updated_at":"2015-11-02T18:50:22Z","body":"What's the status now? Is the index ready on all the nodes? If not, could you rebuild it on the node where wasn't built?,"}}