{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/4671","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/4671/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/4671/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/4671/events","html_url":"https://github.com/hazelcast/hazelcast/issues/4671","id":58780684,"number":4671,"title":"Two issues with local near cache","user":{"login":"lukasblu","id":3501420,"avatar_url":"https://avatars.githubusercontent.com/u/3501420?v=3","gravatar_id":"","url":"https://api.github.com/users/lukasblu","html_url":"https://github.com/lukasblu","followers_url":"https://api.github.com/users/lukasblu/followers","following_url":"https://api.github.com/users/lukasblu/following{/other_user}","gists_url":"https://api.github.com/users/lukasblu/gists{/gist_id}","starred_url":"https://api.github.com/users/lukasblu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukasblu/subscriptions","organizations_url":"https://api.github.com/users/lukasblu/orgs","repos_url":"https://api.github.com/users/lukasblu/repos","events_url":"https://api.github.com/users/lukasblu/events{/privacy}","received_events_url":"https://api.github.com/users/lukasblu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"eb6420"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Defect","name":"Type: Defect","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"ahmetmircik","id":2960360,"avatar_url":"https://avatars.githubusercontent.com/u/2960360?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/67","html_url":"https://github.com/hazelcast/hazelcast/milestones/3.5.2","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/67/labels","id":1217906,"number":67,"title":"3.5.2","description":"","creator":{"login":"mdogan","id":1201712,"avatar_url":"https://avatars.githubusercontent.com/u/1201712?v=3","gravatar_id":"","url":"https://api.github.com/users/mdogan","html_url":"https://github.com/mdogan","followers_url":"https://api.github.com/users/mdogan/followers","following_url":"https://api.github.com/users/mdogan/following{/other_user}","gists_url":"https://api.github.com/users/mdogan/gists{/gist_id}","starred_url":"https://api.github.com/users/mdogan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdogan/subscriptions","organizations_url":"https://api.github.com/users/mdogan/orgs","repos_url":"https://api.github.com/users/mdogan/repos","events_url":"https://api.github.com/users/mdogan/events{/privacy}","received_events_url":"https://api.github.com/users/mdogan/received_events","type":"User","site_admin":false},"open_issues":40,"closed_issues":15,"state":"open","created_at":"2015-07-21T10:52:39Z","updated_at":"2015-07-31T13:21:52Z","due_on":null,"closed_at":null},"comments":36,"created_at":"2015-02-24T18:16:48Z","updated_at":"2015-07-31T15:23:56Z","closed_at":null,"body":"Hi,\r\n\r\nwhile stress testing our application, we discovered two issues with using local near caches:\r\n\r\n1. Even on a single node, the local near cache is not synchronously invalidated after the map is modified.\r\n\r\n2. Also on a single node, it may happen that near cache invalidations are lost.\r\n\r\nTo 1:\r\nI understand that you do not give full consistency guarantees when using a near cache, however I kind of expected that at least the local node always sees its latest changes.\r\nWhile adding full consistency guarantees on a distributed near cache might be very expensive (in terms of synchronization and operation latency), providing such guarantees on the local node only should be much cheaper.\r\n\r\nTo 2:\r\nThis is certainly a bug.\r\n\r\nPlease find below a test case which shows both issues.\r\n\r\nIf using a ConcurrentHashMap, the test always succeeds. If using a distributed Hazelcast map with local near cache enabled, the test either fails with\r\n  \"Assertion violated.\"\r\nwhich means that the near cache showed old data, or the test fails with\r\n  \"Near cache did *not* become consistent.\"\r\nwhich means that even after waiting for some time, the near cache did not show the latest data - hence near cache invalidations were lost.\r\n\r\nIt would be great to hear your opinion on 1 and a fix for 2 would be amazing! :-)\r\n\r\nThanks and best,\r\nLukas\r\n\r\n\r\n```java\r\npackage com.nm.test.hazelcast.nearcache;\r\n\r\nimport com.hazelcast.config.*;\r\nimport com.hazelcast.core.Hazelcast;\r\nimport com.hazelcast.core.HazelcastInstance;\r\nimport com.nm.test.hazelcast.TestHazelcast;\r\nimport org.apache.log4j.BasicConfigurator;\r\nimport org.apache.log4j.Logger;\r\nimport java.util.ArrayList;\r\nimport java.util.List;\r\nimport java.util.concurrent.ConcurrentHashMap;\r\nimport java.util.concurrent.ConcurrentMap;\r\nimport java.util.concurrent.atomic.AtomicBoolean;\r\nimport java.util.concurrent.atomic.AtomicInteger;\r\nimport junit.framework.TestCase;\r\n\r\n/**\r\n * A test to ensure the near cache eventually reflects the data from the map.\r\n */\r\npublic class TestLocalNearCache6 extends TestCase {\r\n\r\n\tprivate static final Logger logger = Logger.getLogger(TestLocalNearCache6.class);\r\n\r\n\tprivate static final int numGetters = 7;\r\n\r\n\tprivate static final String mapName = \"testMap\" + TestLocalNearCache6.class.getSimpleName();\r\n\r\n\tprivate static final String key = \"key123\";\r\n\r\n\tprivate AtomicInteger valuePut = new AtomicInteger(0);\r\n\r\n\tprivate AtomicBoolean stop = new AtomicBoolean(false);\r\n\r\n\tprivate AtomicBoolean failed = new AtomicBoolean(false);\r\n\r\n\t@Override\r\n\tprotected void setUp() throws Exception {\r\n\r\n\t\t// configure logging\r\n\t\tif (!TestHazelcast.loggingInitialized) {\r\n\t\t\tTestHazelcast.loggingInitialized = true;\r\n\t\t\tBasicConfigurator.configure();\r\n\t\t}\r\n\t}\r\n\r\n\tpublic void testConsistentNearCache() throws Exception {\r\n\r\n\t\t// create concurrent hash map\r\n\t\tConcurrentMap<String, String> map = new ConcurrentHashMap<String, String>();\r\n\r\n\t\t// run test\r\n\t\ttestInternal(map);\r\n\r\n\t\t// fail if failed\r\n\t\tif (failed.get()) {\r\n\t\t\tfail(\"Assertion violated.\");\r\n\t\t}\r\n\t}\r\n\r\n\tpublic void testConsistentNearCacheHz() throws Exception {\r\n\r\n\t\t// create hazelcast config\r\n\t\tConfig config = new XmlConfigBuilder().build();\r\n\t\tconfig.setProperty(\"hazelcast.logging.type\", \"log4j\");\r\n\r\n\t\t// configure near cache\r\n\t\tNearCacheConfig nearCacheConfig = new NearCacheConfig();\r\n\t\tnearCacheConfig.setCacheLocalEntries(true); // this enables caching of local entries\r\n\t\tnearCacheConfig.setInMemoryFormat(InMemoryFormat.OBJECT);\r\n\r\n\t\t// enable near cache\r\n\t\tMapConfig mapConfig = config.getMapConfig(mapName);\r\n\t\tmapConfig.setNearCacheConfig(nearCacheConfig);\r\n\r\n\t\t// create Hazelcast instance\r\n\t\tHazelcastInstance hcInstance = Hazelcast.newHazelcastInstance(config);\r\n\r\n\t\t// get hazelcast map\r\n\t\tConcurrentMap<String, String> map = hcInstance.getMap(mapName);\r\n\r\n\t\t// run test\r\n\t\ttestInternal(map);\r\n\r\n\t\t// test eventually consistent\r\n\t\tThread.sleep(1000);\r\n\t\tint valuePutLast = valuePut.get();\r\n\t\tString valueMapStr = map.get(key);\r\n\t\tint valueMap = Integer.parseInt(valueMapStr);\r\n\r\n\t\t// stop hazelcast\r\n\t\thcInstance.getLifecycleService().terminate();\r\n\r\n\t\t// test eventually consistent\r\n\t\tif (valueMap < valuePutLast) {\r\n\t\t\tString msg = \"Near cache did *not* become consistent. (valueMap = \" + valueMap + \", valuePut = \" + valuePutLast + \")\";\r\n\t\t\tlogger.warn(msg);\r\n\t\t\tfail(msg);\r\n\t\t}\r\n\r\n\t\t// fail if failed\r\n\t\tif (failed.get()) {\r\n\t\t\tfail(\"Assertion violated.\");\r\n\t\t}\r\n\t}\r\n\r\n\tprivate void testInternal(ConcurrentMap<String, String> map) throws Exception {\r\n\r\n\t\t// start 1 putter thread\r\n\t\tThread threadPut = new Thread(new PutRunnable(map), \"put\");\r\n\t\tthreadPut.start();\r\n\t\tThread.sleep(1000);\r\n\r\n\t\t// start numGetters getter threads\r\n\t\tList<Thread> threads = new ArrayList<Thread>();\r\n\t\tfor (int i = 0; i < numGetters; i++) {\r\n\t\t\tThread thread = new Thread(new GetRunnable(map), \"get\" + i);\r\n\t\t\tthreads.add(thread);\r\n\t\t}\r\n\t\tfor (Thread thread : threads) {\r\n\t\t\tthread.start();\r\n\t\t}\r\n\r\n\t\t// stop after 10s\r\n\t\tint i = 0;\r\n\t\twhile (!stop.get() && i++ < 10) {\r\n\t\t\tThread.sleep(1000);\r\n\t\t}\r\n\t\tif (i == 20) {\r\n\t\t\tlogger.info(\"Problem did not occur within 10s.\");\r\n\t\t}\r\n\t\tstop.set(true);\r\n\t\tthreadPut.join();\r\n\t\tfor (Thread thread : threads) {\r\n\t\t\tthread.join();\r\n\t\t}\r\n\t}\r\n\r\n\tprivate class PutRunnable implements Runnable {\r\n\r\n\t\tprivate final ConcurrentMap<String, String> map;\r\n\r\n\t\tpublic PutRunnable(ConcurrentMap<String, String> map) {\r\n\t\t\tthis.map = map;\r\n\t\t}\r\n\r\n\t\t@Override\r\n\t\tpublic void run() {\r\n\t\t\tlogger.info(Thread.currentThread().getName() + \" started.\");\r\n\t\t\tint i = 0;\r\n\t\t\twhile (!stop.get()) {\r\n\r\n\t\t\t\t// put new value and update last state\r\n\t\t\t\t// note: the value in the map/near cache is *always* larger or equal to valuePut\r\n\t\t\t\t// assert: valueMap >= valuePut\r\n\t\t\t\ti++;\r\n\t\t\t\tmap.put(key, String.valueOf(i));\r\n\t\t\t\tvaluePut.set(i);\r\n\r\n\t\t\t\t// sleep\r\n\t\t\t\ttry {\r\n\t\t\t\t\tThread.sleep(50);\r\n\t\t\t\t} catch (InterruptedException e) {\r\n\t\t\t\t\tlogger.warn(\"Interrupted: \" + e.getMessage());\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t\tlogger.info(Thread.currentThread().getName() + \" performed \" + i + \" operations.\");\r\n\t\t}\r\n\r\n\t}\r\n\r\n\tprivate class GetRunnable implements Runnable {\r\n\r\n\t\tprivate final ConcurrentMap<String, String> map;\r\n\r\n\t\tpublic GetRunnable(ConcurrentMap<String, String> map) {\r\n\t\t\tthis.map = map;\r\n\t\t}\r\n\r\n\t\t@Override\r\n\t\tpublic void run() {\r\n\t\t\tlogger.info(Thread.currentThread().getName() + \" started.\");\r\n\t\t\tint n = 0;\r\n\t\t\twhile (!stop.get()) {\r\n\r\n\t\t\t\t// compare value from near cache with last put value\r\n\t\t\t\tint valuePutLast = valuePut.get();\r\n\t\t\t\tString valueMapStr = map.get(key);\r\n\t\t\t\tint valueMap = Integer.parseInt(valueMapStr);\r\n\t\t\t\tn++;\r\n\r\n\t\t\t\t// ensure assertion holds\r\n\t\t\t\tif (valueMap < valuePutLast) {\r\n\t\t\t\t\tlogger.warn(\"Assertion violated! (valueMap = \" + valueMap + \", valuePut = \" + valuePutLast + \")\");\r\n\t\t\t\t\tfailed.set(true);\r\n\t\t\t\t\tstop.set(true);\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t\tlogger.info(Thread.currentThread().getName() + \" performed \" + n + \" operations.\");\r\n\t\t}\r\n\r\n\t}\r\n\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/126724031","html_url":"https://github.com/hazelcast/hazelcast/issues/4671#issuecomment-126724031","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/4671","id":126724031,"user":{"login":"ahmetmircik","id":2960360,"avatar_url":"https://avatars.githubusercontent.com/u/2960360?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T15:23:56Z","updated_at":"2015-07-31T15:23:56Z","body":"Hi @lukasblu, currently some solutions exist around but still we are not happy with any one of them. Lets continue to keep an eye on it. \r\n\r\n\r\n"}}