{"action":"created","issue":{"url":"https://api.github.com/repos/techfort/LokiJS/issues/185","labels_url":"https://api.github.com/repos/techfort/LokiJS/issues/185/labels{/name}","comments_url":"https://api.github.com/repos/techfort/LokiJS/issues/185/comments","events_url":"https://api.github.com/repos/techfort/LokiJS/issues/185/events","html_url":"https://github.com/techfort/LokiJS/issues/185","id":93110845,"number":185,"title":"Scaling on multiple CPUs for load balancing","user":{"login":"Student007","id":5132566,"avatar_url":"https://avatars.githubusercontent.com/u/5132566?v=3","gravatar_id":"","url":"https://api.github.com/users/Student007","html_url":"https://github.com/Student007","followers_url":"https://api.github.com/users/Student007/followers","following_url":"https://api.github.com/users/Student007/following{/other_user}","gists_url":"https://api.github.com/users/Student007/gists{/gist_id}","starred_url":"https://api.github.com/users/Student007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Student007/subscriptions","organizations_url":"https://api.github.com/users/Student007/orgs","repos_url":"https://api.github.com/users/Student007/repos","events_url":"https://api.github.com/users/Student007/events{/privacy}","received_events_url":"https://api.github.com/users/Student007/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-05T13:21:41Z","updated_at":"2015-07-06T07:40:57Z","closed_at":null,"body":"Hi Joe,\r\n\r\nFirst, I love your awesome work - thanks for making this free! \r\nIn your roadmap you plan `replication and horizontal scaling`. Does this mean you will give the ability to add multiple machines what sync their JSON-Files ?\r\nI like to do `vertical scaling` like Nodejs http clustering - means: Using all my CPUs. I did this my own using [MapRed](https://www.npmjs.com/package/mapred) for mapReduce, but it would be interesting to see full clustering support by LokiJS itself.\r\nIt is about concurrency of the persistence level, I cannot fork myself the easy way. Because it is the same memory (I think) it could sync inMemory and persist all in the master instance. In my opinion about how LokiJS works, I think, it should be possible to fork the `insert,update,get` and organize the persistence sync by the master by giving `API control about the number of forks`.\r\nIn the case I missed something, please apologize and put my nose on it :smile: \r\nI wouldn't hang on isomorphic thoughts instead of getting much more performance by clustering.\r\nThe browser has web workers for threading - are you using them (couldn't find something in the doc - oh wait - the dynamicViews are web workers maybe) ?\r\n\r\nBtw: I love to see you changed your licensing more MIT like\r\n\r\nMy goal is to get an inMemoryDB is working within all instances of Nodejs answering a http request (load balancing). I am thinking about to write a KoaJS Middleware to load multiple instances based on numbers of idle CPUs. That way the Event-Stack wouldn't block the chaining of koa middleware longer than necessary - DB is a bottleneck.\r\nIn the case there is not resource to add the features like I need for my plans. It would be nice to get an explanation about how to manage this my own (maybe setting semaphores in the app itself - bound on the events LokiJS fires).\r\n\r\nDan"},"comment":{"url":"https://api.github.com/repos/techfort/LokiJS/issues/comments/118760799","html_url":"https://github.com/techfort/LokiJS/issues/185#issuecomment-118760799","issue_url":"https://api.github.com/repos/techfort/LokiJS/issues/185","id":118760799,"user":{"login":"Student007","id":5132566,"avatar_url":"https://avatars.githubusercontent.com/u/5132566?v=3","gravatar_id":"","url":"https://api.github.com/users/Student007","html_url":"https://github.com/Student007","followers_url":"https://api.github.com/users/Student007/followers","following_url":"https://api.github.com/users/Student007/following{/other_user}","gists_url":"https://api.github.com/users/Student007/gists{/gist_id}","starred_url":"https://api.github.com/users/Student007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Student007/subscriptions","organizations_url":"https://api.github.com/users/Student007/orgs","repos_url":"https://api.github.com/users/Student007/repos","events_url":"https://api.github.com/users/Student007/events{/privacy}","received_events_url":"https://api.github.com/users/Student007/received_events","type":"User","site_admin":false},"created_at":"2015-07-06T07:40:57Z","updated_at":"2015-07-06T07:40:57Z","body":"@obeliskos thanks for the detailed first response. That's a good idea to put the JSON to a remote server like AWS. But here comes up a new question:\r\nDoes the saving only update the JSON parts in the file that have been changed since last timestamp - or is it written completely new ? I think frequently saving 2GB will make a difference here. Same as sending data over network. It would be better only to send data based on timestamp later than x.\r\nIf you didn't, here is a short idea of optimization: Knowing the position of each dataset by reference pointer within the `database.json` would make persistence much faster. Same as saving network traffic, you only need to write a service for updating the database.json ...\r\n\r\nAbout the idea to share collections: This would be a good idea if they are copies not referenced collections - isn't it ? \r\n\r\nOr is there a way to use the reference to save memory ? It is about the imagination the hole database file is in memory when getting the collection out of it. In my opinion the gotten collection is referenced, not copied. I know from the docs there is a way to copy them (if I remember right). Writing to the same collection (if it is referenced not copied) would maybe make a concurrency situation (by clustering). I'm unsure about this point. I do not know what part of the hole (operating) system is managing this. Maybe the operating system prevents read/write access to memory when it is used by another thread. \r\n\r\n**Edit**: I did one more search and found [this article on medium.com](https://medium.com/@tech_fort/an-introduction-to-lokijs-the-idiomatic-way-d24a4c546f7)  where it is talked about **branchResultset()** what I couldn't find on your official documentation at lokijs.org. It looks like the concurrency problem is solved by merging in some magic way - isn't it ? So limited copies could be referenced by condition of requested resource by the slave instance of the answering server. After answering and performing the action on the resource, the final act would be to trigger merging.\r\n\r\nBut all the thoughts might be wrong and cause in wasting capacity. (Blowing up memory is more expensive than waiting for access to a single instance). When the process forks for nodejs load balancing, the hole app (without what is triggered within the `if(cluster.isMaster){...}` part ) is copied in memory. So it would be the strategy to fork the lokiJS part within the master. At this point the hole communication runs over the cluster API of NodeJS. \r\n(Btw: You could use the NodeJS-Cluster like WebWorkers ! - it works the similar way)\r\n"}}