{"action":"created","issue":{"url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/315","labels_url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/315/labels{/name}","comments_url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/315/comments","events_url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/315/events","html_url":"https://github.com/HaxeFoundation/hxcpp/issues/315","id":111710378,"number":315,"title":"hxcpp garbage collector leaks memory","user":{"login":"bjitivo","id":4976999,"avatar_url":"https://avatars.githubusercontent.com/u/4976999?v=3","gravatar_id":"","url":"https://api.github.com/users/bjitivo","html_url":"https://github.com/bjitivo","followers_url":"https://api.github.com/users/bjitivo/followers","following_url":"https://api.github.com/users/bjitivo/following{/other_user}","gists_url":"https://api.github.com/users/bjitivo/gists{/gist_id}","starred_url":"https://api.github.com/users/bjitivo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjitivo/subscriptions","organizations_url":"https://api.github.com/users/bjitivo/orgs","repos_url":"https://api.github.com/users/bjitivo/repos","events_url":"https://api.github.com/users/bjitivo/events{/privacy}","received_events_url":"https://api.github.com/users/bjitivo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-15T21:22:53Z","updated_at":"2015-10-16T19:11:58Z","closed_at":null,"body":"Here is a test program that I wrote a long time ago to stress test the hxcpp garbage collector:\r\n\r\n```\r\n/** *************************************************************************\r\n * StressGC.hx\r\n *\r\n * Copyright 2013 TiVo, Inc.\r\n ************************************************************************** **/\r\n\r\nclass StressGC\r\n{\r\n    public static function main()\r\n    {\r\n        var argv = Sys.args();\r\n\r\n        var maxArrays = 2000;\r\n        var maxStrings = 2000;\r\n        var maxObjects = 2000;\r\n        var clear_after = 10000;\r\n        var iterations = -1;\r\n        var thread_count = 1;\r\n\r\n        var seed : Int = 0;\r\n        var seeded = false;\r\n\r\n        var iter = 0 ... argv.length;\r\n        for (i in iter) {\r\n            var arg = argv[i];\r\n            switch (arg) {\r\n            case \"-s\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -s requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                seed = Std.parseInt(argv[i + 1]);\r\n                i = iter.next();\r\n                seeded = true;\r\n\r\n            case \"-i\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -i requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                iterations = Std.parseInt(argv[i + 1]);\r\n                i = iter.next();\r\n                \r\n            case \"-c\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -c requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                clear_after = Std.parseInt(argv[i + 1]);\r\n                if (clear_after < 0) {\r\n                    Sys.stderr().writeString(\"ERROR: -c requires an \" +\r\n                                             \"argument >= 0\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                i = iter.next();\r\n\r\n            case \"-am\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -am requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                maxArrays = Std.parseInt(argv[i + 1]);\r\n                if (maxArrays < 0) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -am requires an argument > 0\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                i = iter.next();\r\n\r\n            case \"-sm\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -sm requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                maxStrings = Std.parseInt(argv[i + 1]);\r\n                if (maxStrings < 0) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -sm requires an argument > 0\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                i = iter.next();\r\n\r\n            case \"-om\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -om requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                maxObjects = Std.parseInt(argv[i + 1]);\r\n                if (maxObjects < 0) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -om requires an argument > 0\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                i = iter.next();\r\n\r\n            case \"-tc\":\r\n                if (i == (argv.length - 1)) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -tc requires an argument\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                thread_count = Std.parseInt(argv[i + 1]);\r\n                if (thread_count < 1) {\r\n                    Sys.stderr().writeString\r\n                        (\"ERROR: -tc requires an argument > 0\\n\");\r\n                    Sys.exit(-1);\r\n                }\r\n                i = iter.next();\r\n\r\n            default:\r\n                Sys.stderr().writeString\r\n                    (\"ERROR: unknown argumnt: \" + arg + \"\\n\");\r\n                Sys.exit(-1);\r\n            }\r\n        }\r\n\r\n        if ((maxArrays == 0) && (maxStrings == 0) && (maxObjects == 0)) {\r\n            Sys.stderr().writeString(\"ERROR: At least one of -am, -sm or -om \" +\r\n                                     \"must have an argument > 0\\n\");\r\n            Sys.exit(-1);\r\n        }\r\n\r\n        Runner.setMaxArrays(maxArrays);\r\n        Runner.setMaxStrings(maxStrings);\r\n        Runner.setMaxObjects(maxObjects);\r\n        Runner.setClearAfter(clear_after);\r\n\r\n        if (!seeded) {\r\n            seed = Std.int(Sys.time() * 100);\r\n        }\r\n\r\n        Sys.println(\"Using random seed: \" + seed);\r\n\r\n        // Create the Runners\r\n        var arr = new Array<Runner>();\r\n        for (i in 0 ... thread_count) {\r\n            arr.push(new Runner(i, iterations, seed + i));\r\n        }\r\n\r\n        // Start the Runners\r\n        for (i in 0 ... arr.length) {\r\n            arr[i].start();\r\n        }\r\n\r\n        // Wait for all to be done\r\n        for (i in 0 ... arr.length) {\r\n            arr[i].join();\r\n        }\r\n\r\n#if cpp\r\n        cpp.vm.Gc.run(true);\r\n#elseif neko\r\n        neko.vm.Gc.run(true);\r\n#end\r\n\r\n        return 0;\r\n    }\r\n}\r\n\r\n\r\nclass Runner\r\n{\r\n    public static function setMaxArrays(max : Int)\r\n    {\r\n        gMaxArrays = max;\r\n    }\r\n\r\n    public static function setMaxStrings(max : Int)\r\n    {\r\n        gMaxStrings = max;\r\n    }\r\n\r\n    public static function setMaxObjects(max : Int)\r\n    {\r\n        gMaxObjects = max;\r\n    }\r\n\r\n    public static function setClearAfter(clear_after : Int)\r\n    {\r\n        gClearAfter = clear_after;\r\n    }\r\n\r\n    public function new(num : Int, iters : Int, seed : Int)\r\n    {\r\n        mNum = num;\r\n        mIters = iters;\r\n        mRandom = new Random(seed);\r\n        mObject = new Object(mRandom, gMaxArrays, gMaxStrings, gMaxObjects);\r\n#if cpp\r\n        mJoinQ = new cpp.vm.Deque<Int>();\r\n        mThread = cpp.vm.Thread.create(thread_main);\r\n#elseif neko\r\n        mJoinQ = new neko.vm.Deque<Int>();\r\n        mThread = neko.vm.Thread.create(thread_main);\r\n#end\r\n    }\r\n\r\n    public function start()\r\n    {\r\n#if (cpp || neko)\r\n        // Send the runner to the thread to start it\r\n        mThread.sendMessage(this);\r\n#else\r\n        this.run();\r\n#end\r\n    }\r\n\r\n    public function join()\r\n    {\r\n#if (cpp || neko)\r\n        // Read the message send at the end of run()\r\n        mJoinQ.pop(true);\r\n#end\r\n    }\r\n\r\n    private function run()\r\n    {\r\n        var iter_num = 0;\r\n\r\n        while ((mIters == -1) || (--mIters > 0)) {\r\n            iter_num += 1;\r\n            mObject.iterate();\r\n            \r\n            if ((iter_num % 1000) == 0) {\r\n                mObject.dump(\"\" + mNum + \" @ \" + iter_num + \": \");\r\n            }\r\n\r\n            if ((gClearAfter > 0) && ((iter_num % gClearAfter) == 0)) {\r\n                mObject.clear(\"\" + mNum + \" @ \" + iter_num + \": \");\r\n            }\r\n        }\r\n        \r\n        mObject.clear(\"\" + mNum + \" @ \" + iter_num + \": \");\r\n    }\r\n\r\n#if (cpp || neko)\r\n    private static function thread_main()\r\n    {\r\n        // Wait for start() to send the Runner\r\n        var runner : Runner = cpp.vm.Thread.readMessage(true);\r\n\r\n        // Run the runner\r\n        runner.run();\r\n\r\n        // Send a message which join() will read\r\n        runner.mJoinQ.push(0);\r\n    }\r\n#end\r\n\r\n    static private var gMaxArrays : Int;\r\n    static private var gMaxStrings : Int;\r\n    static private var gMaxObjects : Int;\r\n    private static var gClearAfter : Int;\r\n\r\n    private var mNum : Int;\r\n    private var mIters : Int;\r\n    private var mRandom : Random;\r\n    private var mObject : Object;\r\n\r\n#if cpp\r\n    private var mJoinQ : cpp.vm.Deque<Int>;\r\n    private var mThread : cpp.vm.Thread;\r\n#elseif neko\r\n    private var mJoinQ : neko.vm.Deque<Int>;\r\n    private var mThread : neko.vm.Thread;\r\n#end\r\n}\r\n\r\n\r\nclass Object\r\n{\r\n    public function new(random : Random, maxArrays : Int, maxStrings : Int,\r\n                        maxObjects : Int)\r\n    {\r\n        mRandom = random;\r\n        mMaxArrays = maxArrays;\r\n        mMaxStrings = maxStrings;\r\n        mMaxObjects = maxObjects;\r\n        this.allocate();\r\n    }\r\n\r\n    public function iterate()\r\n    {\r\n        if (mIterators.length > 0) {\r\n            mIterators[mRandom.random() % mIterators.length]();\r\n        }\r\n    }\r\n\r\n    public function clear(prefix : String)\r\n    {\r\n        var clear_count : Int = 0;\r\n        if (mObjects != null) {\r\n            for (i in 0 ... mObjects.length) {\r\n                if (mObjects[i] != null) {\r\n                    clear_count += 1;\r\n                }\r\n            }\r\n        }\r\n        Sys.println(prefix + \"Cleared \" + clear_count + \" objects\");\r\n        mArrays = [ ];\r\n        mStrings = [ ];\r\n        mObjects = [ ];\r\n#if cpp\r\n        cpp.vm.Gc.run(true);\r\n#elseif neko\r\n        neko.vm.Gc.run(true);\r\n#end\r\n        this.allocate();\r\n    }\r\n\r\n    public function dump(prefix : String)\r\n    {\r\n#if cpp\r\n        // Testing enterGCFreeZone and enable\r\n        cpp.vm.Gc.enable(false);\r\n        cpp.vm.Gc.enterGCFreeZone();\r\n        cpp.vm.Gc.exitGCFreeZone();\r\n#end\r\n        var arrayCount = this.countArrays();\r\n        var stringCount = this.countStrings();\r\n        var objectCount = this.countObjects();\r\n\r\n        Sys.println(prefix + \"a: \" + arrayCount + \", s: \" + stringCount +\r\n                    \", o: \" + objectCount);\r\n\r\n#if cpp\r\n        cpp.vm.Gc.enable(true);\r\n#end\r\n    }\r\n\r\n    private function countArrays() : Int\r\n    {\r\n        var ret = 0;\r\n        if (mArrays != null) {\r\n            for (a in mArrays) {\r\n                if (a != null) {\r\n                    ret += 1;\r\n                }\r\n            }\r\n        }\r\n        if (mObjects != null) {\r\n            for (o in mObjects) {\r\n                if (o != null) {\r\n                    ret += o.countArrays();\r\n                }\r\n            }\r\n        }\r\n        return ret;\r\n    }\r\n\r\n    private function countStrings() : Int\r\n    {\r\n        var ret = 0;\r\n        if (mStrings != null) {\r\n            for (s in mStrings) {\r\n                if (s != null) {\r\n                    ret += 1;\r\n                }\r\n            }\r\n        }\r\n        if (mObjects != null) {\r\n            for (o in mObjects) {\r\n                if (o != null) {\r\n                    ret += o.countStrings();\r\n                }\r\n            }\r\n        }\r\n        return ret;\r\n    }\r\n\r\n    private function countObjects() : Int\r\n    {\r\n        var ret = 0;\r\n        if (mObjects != null) {\r\n            for (o in mObjects) {\r\n                if (o != null) {\r\n                    ret += o.countObjects() + 1;\r\n                }\r\n            }\r\n        }\r\n        return ret;\r\n    }\r\n\r\n    private function iterateArray()\r\n    {\r\n        var which = (mArrays.length == 0) ? 0 :\r\n            (mRandom.random() % mArrays.length);\r\n\r\n        var set = (mArrays[which] == null) ? \r\n            true : ((mRandom.random() % 2) == 0);\r\n\r\n        var array : Array<Int> = null;\r\n\r\n        if (set) {\r\n            array = new Array<Int>();\r\n            var element_count = mRandom.random() % 1000;\r\n            for (i in 0 ... element_count) {\r\n                array.push(0);\r\n            }\r\n        }\r\n\r\n        mArrays[which] = array;\r\n    }\r\n\r\n    private function iterateString()\r\n    {\r\n        var which = (mStrings.length == 0) ? 0 :\r\n            (mRandom.random() % mStrings.length);\r\n\r\n        var set = (mStrings[which] == null) ? \r\n            true : ((mRandom.random() % 2) == 0);\r\n\r\n        var string : String = null;\r\n\r\n        if (set) {\r\n            string = new String(gStrings[mRandom.random() % gStrings.length]);\r\n        }\r\n\r\n        mStrings[which] = string;\r\n    }\r\n\r\n    private function iterateObject()\r\n    {\r\n        var which = (mObjects.length == 0) ? 0 :\r\n            (mRandom.random() % mObjects.length);\r\n\r\n        var object : Object = null;\r\n\r\n        if (mObjects[which] == null) {\r\n            object = this.createObject();\r\n        }\r\n        else {\r\n            switch (mRandom.random() % 7) {\r\n            case 0:\r\n                object = null;\r\n                \r\n            case 1:\r\n                object = this.createObject();\r\n                \r\n            default:\r\n                mObjects[which].iterate();\r\n                return;\r\n            }\r\n        }\r\n\r\n        mObjects[which] = object;\r\n    }\r\n\r\n    private function createObject() : Object\r\n    {\r\n        return new Object(mRandom, Std.int(mMaxArrays / 2),\r\n                          Std.int(mMaxStrings / 2), Std.int(mMaxObjects / 2));\r\n    }\r\n\r\n    private function allocate()\r\n    {\r\n        mIterators = new Array<Dynamic>();\r\n\r\n        var arrays : Array<Array<Int>> = null;\r\n        var strings : Array<String> = null;\r\n        var objects : Array<Object> = null;\r\n\r\n        var arrayCount =\r\n            (mMaxArrays > 0) ? (mRandom.random() % mMaxArrays) : 0;\r\n        if (arrayCount > 0) {\r\n            arrays = new Array<Array<Int>>();\r\n            for (i in 0 ... arrayCount) {\r\n                arrays.push(null);\r\n            }\r\n            mIterators.push(iterateArray);\r\n        }\r\n\r\n        var stringCount = \r\n            (mMaxStrings > 0) ? (mRandom.random() % mMaxStrings) : 0;\r\n        if (stringCount > 0) {\r\n            strings = new Array<String>();\r\n            for (i in 0 ... stringCount) {\r\n                strings.push(null);\r\n            }\r\n            mIterators.push(iterateString);\r\n        }\r\n\r\n        var objectCount = \r\n            (mMaxObjects > 0) ? (mRandom.random() % mMaxObjects) : 0;\r\n        if (objectCount > 0) {\r\n            objects = new Array<Object>();\r\n            for (i in 0 ... objectCount) {\r\n                objects.push(null);\r\n            }\r\n            mIterators.push(iterateObject);\r\n        }\r\n\r\n        mArrays = arrays;\r\n        mStrings = strings;\r\n        mObjects = objects;\r\n    }\r\n\r\n    private var mRandom : Random;\r\n    private var mIterators : Array<Dynamic>;\r\n    private var mArrays : Array<Array<Int>>;\r\n    private var mStrings : Array<String>;\r\n    private var mObjects : Array<Object>;\r\n    private var mMaxArrays : Int;\r\n    private var mMaxStrings : Int;\r\n    private var mMaxObjects : Int;\r\n\r\n    private static var gStrings = [ \"\", \"1\", \"12\", \"123\", \"1234\", \"12345\",\r\n                                    \"123456\", \"1234567\", \"12345678\",\r\n                                    \"123456789\", \"1234567890\" ];\r\n}\r\n\r\n\r\nclass Random\r\n{\r\n    public function new(seed : Int)\r\n    {\r\n        this.seed(seed);\r\n    }\r\n\r\n    public function random() : Int\r\n    {\r\n        // What are these values? I don't know -- they were in the sample code\r\n        // from wikipedia.\r\n        var a : Int = 18782;\r\n        var r : Int = 0xfffffffe;\r\n        var t : Int;\r\n        var x : Int;\r\n        \r\n        mRotation = (mRotation + 1) & (RESIDUES - 1);\r\n        t = a * mQ[mRotation] + mC;\r\n        mC = (t >> 32);\r\n        x = t + mC;\r\n        if (x < mC) {\r\n            x++;\r\n            mC++;\r\n        }\r\n        mQ[mRotation] = r - x;\r\n\r\n        return (mQ[mRotation] < 0) ? -mQ[mRotation] : mQ[mRotation];\r\n    }\r\n    \r\n    private function seed(seed : Int)\r\n    {\r\n        mQ = new Array<Int>();\r\n        \r\n        mQ.push(seed);\r\n        mQ.push(seed + PHI);\r\n        mQ.push(seed + PHI + PHI);\r\n        \r\n        for (i in 3 ... RESIDUES)\r\n        {\r\n            mQ.push(mQ[i - 3] ^ mQ[i - 2] ^ PHI ^ i);\r\n        }\r\n        \r\n        mRotation = RESIDUES - 1;\r\n    }\r\n\r\n    // This is the number of pregenerated starting points that are rotated\r\n    // through\r\n    private static inline var RESIDUES : Int = 4096;\r\n\r\n    // I don't know what these next two numbers are. They are from the sample\r\n    // code on Wikipedia. If you want to sort through the formulas there, you\r\n    // can figure it out.\r\n    private static inline var PHI : Int = 0x9e3779b9;\r\n\r\n    private var mC : Int = 362436;\r\n\r\n    // This is the current index of the pregenerated starting point\r\n    private var mRotation : Int;\r\n        \r\n    // This is the array of pregenerated starting points.\r\n    private var mQ : Array<Int>;\r\n}\r\n```\r\n\r\nWhen run with the newest version of hxcpp, it demonstrates that the GC is allowing memory to leak.\r\n\r\nWhen run like so:\r\n\r\n```\r\nStressGC -i 10000000 -s 0\r\n```\r\n\r\nI see memory usage of the program start out at around 35 MB, and grow continuously as the program runs, to a maximum memory usage of nearly 60 MB by the time it completes.\r\n\r\nAs a comparison, the hxcpp garbage collector that we use at TiVo, that we wrote ourselves, stays around 29 - 31 MB continuously over the same run, never growing in size beyond that.\r\n\r\nFor what it's worth, the TiVo garbage collector is still faster than the hxcpp garbage collector for single threaded workloads, although not by much (48.5s for the TiVo GC versus 51.2s for the stock GC).  Its implementation is also vastly simpler.  Another drawback of the stock GC is that it uses over a minute of CPU time due to its multithreaded nature in order to finish the test in 51.2 seconds, so it's burning alot more CPU than the TiVo GC, which is single threaded and takes 48.5 seconds of total CPU time.\r\n\r\nThe TiVo GC used to be tons faster than the stock GC on multithreaded workloads, but is now about 50% as fast as the stock GC, so that's an area that we could work on in the TiVo GC if we want to achieve parity or better in all respects when compared to the stock GC.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/comments/148809838","html_url":"https://github.com/HaxeFoundation/hxcpp/issues/315#issuecomment-148809838","issue_url":"https://api.github.com/repos/HaxeFoundation/hxcpp/issues/315","id":148809838,"user":{"login":"bjitivo","id":4976999,"avatar_url":"https://avatars.githubusercontent.com/u/4976999?v=3","gravatar_id":"","url":"https://api.github.com/users/bjitivo","html_url":"https://github.com/bjitivo","followers_url":"https://api.github.com/users/bjitivo/followers","following_url":"https://api.github.com/users/bjitivo/following{/other_user}","gists_url":"https://api.github.com/users/bjitivo/gists{/gist_id}","starred_url":"https://api.github.com/users/bjitivo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjitivo/subscriptions","organizations_url":"https://api.github.com/users/bjitivo/orgs","repos_url":"https://api.github.com/users/bjitivo/repos","events_url":"https://api.github.com/users/bjitivo/events{/privacy}","received_events_url":"https://api.github.com/users/bjitivo/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T19:11:58Z","updated_at":"2015-10-16T19:11:58Z","body":"Are you saying you're not seeing any leaking when running this StressGC program on Windows?\r\n\r\nI can't imagine what Windows vs. Linux has to do with the leak.  I would expect it to occur on all platforms similarly as the low level memory management is similar.  I just double checked and StressGC clearly grows unbounded on Linux, for whatever reason. \r\n\r\nI also checked on Mac OS X:\r\n\r\n- Install haxe 3.2.1 for Mac OS X\r\n- Install hxcpp haxelib via git\r\n- Build StressGC\r\n\r\nThe result is that StressGC also demonstrates a memory leak on Mac OS X as the application grows over time.  I was able to use the XCode \"Instruments\" program to analyze the memory usage of StressGC on Mac OS X and interestingly, the unbounded growth is in 256 byte allocations.  Allocations in all other sizes fluctuated between reasonable bounds, but the 256 byte allocations grew unbounded.  Hope that helps.\r\n"}}