{"action":"closed","issue":{"url":"https://api.github.com/repos/jankotek/mapdb/issues/584","labels_url":"https://api.github.com/repos/jankotek/mapdb/issues/584/labels{/name}","comments_url":"https://api.github.com/repos/jankotek/mapdb/issues/584/comments","events_url":"https://api.github.com/repos/jankotek/mapdb/issues/584/events","html_url":"https://github.com/jankotek/mapdb/issues/584","id":106935595,"number":584,"title":"Degraded performance when using Memory map with Overflow","user":{"login":"yuval-rikover","id":3227117,"avatar_url":"https://avatars.githubusercontent.com/u/3227117?v=3","gravatar_id":"","url":"https://api.github.com/users/yuval-rikover","html_url":"https://github.com/yuval-rikover","followers_url":"https://api.github.com/users/yuval-rikover/followers","following_url":"https://api.github.com/users/yuval-rikover/following{/other_user}","gists_url":"https://api.github.com/users/yuval-rikover/gists{/gist_id}","starred_url":"https://api.github.com/users/yuval-rikover/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuval-rikover/subscriptions","organizations_url":"https://api.github.com/users/yuval-rikover/orgs","repos_url":"https://api.github.com/users/yuval-rikover/repos","events_url":"https://api.github.com/users/yuval-rikover/events{/privacy}","received_events_url":"https://api.github.com/users/yuval-rikover/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/jankotek/mapdb/labels/2.0","name":"2.0","color":"009800"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-17T08:33:40Z","updated_at":"2015-10-01T09:02:55Z","closed_at":"2015-10-01T09:02:55Z","body":"Testing the overflow-to-disk feature results in worse performance compared to a regular file-map with cache:\r\n* *Inserts* are about 4 times slower,\r\n* *Fetches* are about 20 times slower, but because of the *get()* problem (https://github.com/jankotek/mapdb/issues/583), it's not a very good indication,\r\nas I had to call *remove* on each in-memory key in order to get it.\r\n\r\nHere's the test I used to get these results:\r\n\r\n```java\r\nimport com.google.common.base.Stopwatch;\r\nimport com.google.common.collect.ImmutableSet;\r\nimport com.google.common.collect.Sets;\r\nimport com.google.common.primitives.Ints;\r\nimport com.google.common.primitives.Longs;\r\nimport org.apache.commons.io.FileUtils;\r\nimport org.apache.commons.lang.math.RandomUtils;\r\nimport org.junit.After;\r\nimport org.junit.Before;\r\nimport org.junit.Test;\r\nimport org.mapdb.DB;\r\nimport org.mapdb.DBMaker;\r\nimport org.mapdb.HTreeMap;\r\nimport org.mapdb.Serializer;\r\n\r\nimport java.io.ByteArrayOutputStream;\r\nimport java.io.File;\r\nimport java.io.IOException;\r\nimport java.nio.file.Files;\r\nimport java.nio.file.Path;\r\nimport java.util.Map;\r\nimport java.util.Set;\r\nimport java.util.concurrent.TimeUnit;\r\n\r\nimport static org.hamcrest.Matchers.is;\r\nimport static org.junit.Assert.assertNotNull;\r\nimport static org.junit.Assert.assertThat;\r\n\r\npublic class SimpleDiskBackedMapPerformanceTest {\r\n\r\n    private static final String MAP = \"map\";\r\n    private static final int CACHE_SIZE = 100_000;\r\n    private static final int ENTRIES_COUNT = CACHE_SIZE * 20;\r\n\r\n    private Path tempDirectory;\r\n\r\n    @Before\r\n    public void createTempDir() throws IOException {\r\n        tempDirectory = Files.createTempDirectory(\"eh-cache\");\r\n    }\r\n\r\n    @After\r\n    public void deleteTempDir() throws IOException {\r\n        FileUtils.deleteDirectory(tempDirectory.toFile());\r\n    }\r\n\r\n    @Test\r\n    public void testWithOverflow() throws IOException {\r\n\r\n        DB diskDb = DBMaker.fileDB(new File(tempDirectory.toFile(), \"mapdb\"))\r\n                .fileMmapEnable()\r\n                .transactionDisable()\r\n                .closeOnJvmShutdown()\r\n                .deleteFilesAfterClose()\r\n                .make();\r\n\r\n        DB memoryDb = DBMaker.memoryDB()\r\n                .transactionDisable()\r\n                .make();\r\n\r\n        HTreeMap<Integer, byte[]> diskMap = diskDb.hashMapCreate(MAP)\r\n                .keySerializer(Serializer.INTEGER)\r\n                .valueSerializer(Serializer.BYTE_ARRAY)\r\n                .make();\r\n\r\n        HTreeMap<Integer, byte[]> memoryMap = memoryDb.hashMapCreate(MAP)\r\n                .expireMaxSize(CACHE_SIZE)\r\n                .expireOverflow(diskMap, true)\r\n                .make();\r\n\r\n        Stopwatch stopwatch = new Stopwatch();\r\n        populate(memoryMap, ENTRIES_COUNT, stopwatch);\r\n        System.out.println(\"Overflow put - \" + stopwatch.elapsed(TimeUnit.SECONDS) + \" seconds\");\r\n\r\n        int counter = 0;\r\n        Set<Integer> keySet = ImmutableSet.copyOf(Sets.union(memoryMap.keySet(), diskMap.keySet()));\r\n\r\n        stopwatch.reset();\r\n        for (Integer key : keySet) {\r\n            stopwatch.start();\r\n            byte[] value = extractValue(memoryMap, diskMap, key);\r\n            stopwatch.stop();\r\n            assertNotNull(value);\r\n            counter++;\r\n        }\r\n\r\n        assertThat(counter, is(ENTRIES_COUNT));\r\n        System.out.println(\"Overflow get - \" + stopwatch.elapsed(TimeUnit.SECONDS) + \" seconds\");\r\n    }\r\n\r\n    @Test\r\n    public void testWithoutOverflow() throws IOException {\r\n\r\n        DB db = DBMaker.fileDB(new File(tempDirectory.toFile(), \"mapdb\"))\r\n                .fileMmapEnable()\r\n                .transactionDisable()\r\n                .cacheSize(CACHE_SIZE)\r\n                .deleteFilesAfterClose()\r\n                .make();\r\n\r\n        Map<Integer, byte[]> map = db.hashMapCreate(MAP)\r\n                .keySerializer(Serializer.INTEGER)\r\n                .valueSerializer(Serializer.BYTE_ARRAY)\r\n                .make();\r\n\r\n        Stopwatch stopwatch = new Stopwatch();\r\n        populate(map, ENTRIES_COUNT, stopwatch);\r\n\r\n        System.out.println(\"DiskMap put - \" + stopwatch.elapsed(TimeUnit.SECONDS) + \" seconds\");\r\n\r\n        stopwatch.reset();\r\n        int counter = 0;\r\n        for (Integer key : map.keySet()) {\r\n            stopwatch.start();\r\n            byte[] value = map.get(key);\r\n            stopwatch.stop();\r\n            assertNotNull(value);\r\n            counter++;\r\n        }\r\n        assertThat(counter, is(ENTRIES_COUNT));\r\n        System.out.println(\"DiskMap get - \" + stopwatch.elapsed(TimeUnit.SECONDS) + \" seconds\");\r\n    }\r\n\r\n    private byte[] extractValue(HTreeMap<Integer, byte[]> memoryMap, HTreeMap<Integer, byte[]> diskMap, Integer key) {\r\n        byte[] value = memoryMap.get(key);\r\n        if (value != null) {\r\n            return value;\r\n        }\r\n\r\n        if (memoryMap.containsKey(key)) {\r\n            return memoryMap.remove(key);\r\n        }\r\n\r\n        return null;\r\n    }\r\n\r\n    public void populate(Map<Integer, byte[]> map, int entries, Stopwatch timer) throws IOException {\r\n        for (int i = 0; i < entries; i++) {\r\n\r\n            ByteArrayOutputStream baos = new ByteArrayOutputStream(20);\r\n            baos.write(Longs.toByteArray(RandomUtils.nextLong()));\r\n            baos.write(Longs.toByteArray(RandomUtils.nextLong()));\r\n            baos.write(Ints.toByteArray(RandomUtils.nextInt()));\r\n\r\n            timer.start();\r\n            map.put(i, baos.toByteArray());\r\n            timer.stop();\r\n        }\r\n    }\r\n\r\n}\r\n```"}}