{"action":"created","issue":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/3412","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/3412/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/3412/comments","events_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/3412/events","html_url":"https://github.com/orientechnologies/orientdb/issues/3412","id":54653375,"number":3412,"title":"DB building suspended with message 'Max cache limit is reached (3000 vs. 3001), sync flush is performed'","user":{"login":"adithyank","id":9428921,"avatar_url":"https://avatars.githubusercontent.com/u/9428921?v=3","gravatar_id":"","url":"https://api.github.com/users/adithyank","html_url":"https://github.com/adithyank","followers_url":"https://api.github.com/users/adithyank/followers","following_url":"https://api.github.com/users/adithyank/following{/other_user}","gists_url":"https://api.github.com/users/adithyank/gists{/gist_id}","starred_url":"https://api.github.com/users/adithyank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adithyank/subscriptions","organizations_url":"https://api.github.com/users/adithyank/orgs","repos_url":"https://api.github.com/users/adithyank/repos","events_url":"https://api.github.com/users/adithyank/events{/privacy}","received_events_url":"https://api.github.com/users/adithyank/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/core","name":"core","color":"207de5"}],"state":"closed","locked":false,"assignee":{"login":"laa","id":105066,"avatar_url":"https://avatars.githubusercontent.com/u/105066?v=3","gravatar_id":"","url":"https://api.github.com/users/laa","html_url":"https://github.com/laa","followers_url":"https://api.github.com/users/laa/followers","following_url":"https://api.github.com/users/laa/following{/other_user}","gists_url":"https://api.github.com/users/laa/gists{/gist_id}","starred_url":"https://api.github.com/users/laa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laa/subscriptions","organizations_url":"https://api.github.com/users/laa/orgs","repos_url":"https://api.github.com/users/laa/repos","events_url":"https://api.github.com/users/laa/events{/privacy}","received_events_url":"https://api.github.com/users/laa/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/33","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/33/labels","id":897335,"number":33,"title":"2.0 Final","description":"2.0 Final version after the 2.0-RC","creator":{"login":"lvca","id":312606,"avatar_url":"https://avatars.githubusercontent.com/u/312606?v=3","gravatar_id":"","url":"https://api.github.com/users/lvca","html_url":"https://github.com/lvca","followers_url":"https://api.github.com/users/lvca/followers","following_url":"https://api.github.com/users/lvca/following{/other_user}","gists_url":"https://api.github.com/users/lvca/gists{/gist_id}","starred_url":"https://api.github.com/users/lvca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvca/subscriptions","organizations_url":"https://api.github.com/users/lvca/orgs","repos_url":"https://api.github.com/users/lvca/repos","events_url":"https://api.github.com/users/lvca/events{/privacy}","received_events_url":"https://api.github.com/users/lvca/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":57,"state":"open","created_at":"2014-12-09T14:13:06Z","updated_at":"2015-01-20T12:51:02Z","due_on":"2015-01-19T23:00:00Z","closed_at":null},"comments":7,"created_at":"2015-01-17T06:44:23Z","updated_at":"2015-01-20T13:16:26Z","closed_at":"2015-01-19T17:13:57Z","body":"OrientDB Version : 2.0 RC-2\r\nOS : CentOS 6.4\r\nJRE : Oracle 1.7.0_55\r\n\r\nIn our server application we use only one Thread for adding records in OreintDB and multiple Threads will be reading the records. This is how we have designed\r\n\r\nWe started building our inventory database. After 20% of building, following message got printed in nohup.out continuously for say 110 times and then overall operation halted. The thread dump stayed in the same place when we took the thread dump multiple times.\r\n\r\n```\r\n2015-01-17 00:06:15:187 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:06:25:390 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:06:36:070 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:00:294 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:11:683 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:23:506 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:32:049 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:43:620 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n2015-01-17 00:07:55:173 INFO  Max cache limit is reached (3000 vs. 3001), sync flush is performed. [ODiskWriteAheadLog$LogSegment]\r\n```\r\n\r\nThe thread dump of the writing thread is given below\r\n\r\n```\r\n\"InventoryDataPersister\" prio=10 tid=0x00007f9cfc14c800 nid=0x4165 waiting on condition [0x00007f9b9affb000]\r\n   java.lang.Thread.State: WAITING (parking)\r\n        at sun.misc.Unsafe.park(Native Method)\r\n        - parking to wait for  <0x00007fa1df3a39a8> (a java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)\r\n        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)\r\n        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)\r\n        at java.util.concurrent.FutureTask.get(FutureTask.java:187)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.addAllocatedSpace(OWOWCache.java:193)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.cacheFileContent(OWOWCache.java:914)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.load(OWOWCache.java:425)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.updateCache(OReadWriteDiskCache.java:676)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.doLoad(OReadWriteDiskCache.java:302)\r\n        at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.load(OReadWriteDiskCache.java:270)\r\n        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.addEntry(OPaginatedCluster.java:1338)\r\n        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:387)\r\n        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:1582)\r\n        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:2064)\r\n        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:907)\r\n        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:483)\r\n        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:147)\r\n        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2369)\r\n        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2339)\r\n        at in.co.nmsworks.documenttdb.db.DocDB.commit(DocDB.java:594)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.store.RcaInvWriter.addAll(RcaInvWriter.java:164)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.store.RcaInvWriter.addAll(RcaInvWriter.java:117)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.dao.InventoryDAO.save(InventoryDAO.java:106)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.queue.InventoryDataPersister.saveDAOData(InventoryDataPersister.java:356)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.queue.InventoryDataPersister.processAddition(InventoryDataPersister.java:331)\r\n        at in.co.nmsworks.cygnet.telecom.fault.rcasia.inventory.queue.InventoryDataPersister.run(InventoryDataPersister.java:72)\r\n\r\n   Locked ownable synchronizers:\r\n        - <0x00007fa1c07dfda0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1c10871f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1cb056038> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded819f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded81d68> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded81fa8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded82270> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded82a40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded82cc0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded82dd0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded82f90> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded83188> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded83380> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded83578> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded83770> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded83ff0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded85460> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded88ad8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded8d130> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded912a8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded91a58> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1ded95148> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1deda09c0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dedaa518> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dedaf918> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dedb0e00> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dedb1040> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dedc90d8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dee2d3b0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dee34288> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n        - <0x00007fa1dee344c8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)\r\n```\r\n\r\nIn this stage, we checked the size of the orientdb directory, it was 163 GB. Usually, the size of the DB will be 60GB after the inventory building is 100% over. Now, even with 20% of inventory building, the size of the DB is 163 GB. The database directory had around 700 `*.wal` files totaling to 153 GB. This is another unusual behavior we observed. The free disk space was 50GB at this stage.\r\n\r\nThen we did shutdown our application gracefully using kill <pid> (not kill -9 <pid>). After that, we got message in nohup that orient shutdown complete.\r\n\r\nThen when we restarted the application, all indices started rebuilding.\r\n\r\nSo, to avoid long time in index rebuilding, we had to delete the database directory and starting the inventory building again from starting. Since, it is a testing environment, we could do this. But, this may occur again as we did not do any other change.\r\n\r\nAnother Input:\r\nWe explicit set below two variables\r\n``` java\r\nOGlobalConfiguration.STORAGE_LOCK_TIMEOUT.setValue(5 * 60 * 1000);\r\nOGlobalConfiguration.MVRBTREE_TIMEOUT.setValue(5 * 60 * 1000);\r\n```\r\n\r\nMy Questions\r\n\r\n1. Why the `*.wal` files occupy 153 DB when the inventory building is just 20% over.\r\n2. Even after proper shutdown of Orient, why the indices started rebuilding?"},"comment":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/comments/70651280","html_url":"https://github.com/orientechnologies/orientdb/issues/3412#issuecomment-70651280","issue_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/3412","id":70651280,"user":{"login":"adithyank","id":9428921,"avatar_url":"https://avatars.githubusercontent.com/u/9428921?v=3","gravatar_id":"","url":"https://api.github.com/users/adithyank","html_url":"https://github.com/adithyank","followers_url":"https://api.github.com/users/adithyank/followers","following_url":"https://api.github.com/users/adithyank/following{/other_user}","gists_url":"https://api.github.com/users/adithyank/gists{/gist_id}","starred_url":"https://api.github.com/users/adithyank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adithyank/subscriptions","organizations_url":"https://api.github.com/users/adithyank/orgs","repos_url":"https://api.github.com/users/adithyank/repos","events_url":"https://api.github.com/users/adithyank/events{/privacy}","received_events_url":"https://api.github.com/users/adithyank/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T13:16:26Z","updated_at":"2015-01-20T13:16:26Z","body":"It got suspended... Attaching the thread dumps\r\n\r\nhttps://www.dropbox.com/s/mdq020iuuki6tbu/td.zip?dl=0\r\n"}}