{"action":"created","issue":{"url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/2","labels_url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/2/comments","events_url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/2/events","html_url":"https://github.com/kawanet/msgpack-lite/issues/2","id":103352382,"number":2,"title":"Use of Buffer is killing performance.","user":{"login":"fearthecowboy","id":133736,"avatar_url":"https://avatars.githubusercontent.com/u/133736?v=3","gravatar_id":"","url":"https://api.github.com/users/fearthecowboy","html_url":"https://github.com/fearthecowboy","followers_url":"https://api.github.com/users/fearthecowboy/followers","following_url":"https://api.github.com/users/fearthecowboy/following{/other_user}","gists_url":"https://api.github.com/users/fearthecowboy/gists{/gist_id}","starred_url":"https://api.github.com/users/fearthecowboy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fearthecowboy/subscriptions","organizations_url":"https://api.github.com/users/fearthecowboy/orgs","repos_url":"https://api.github.com/users/fearthecowboy/repos","events_url":"https://api.github.com/users/fearthecowboy/events{/privacy}","received_events_url":"https://api.github.com/users/fearthecowboy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-08-26T19:59:01Z","updated_at":"2015-08-31T18:50:33Z","closed_at":null,"body":"Howdy!\r\n\r\nI'm using *msgpack-lite* in a *very* high performance situation, and I've discovered a bit about some of the way that you're doing some stuff that's robbing you of performance greatly.\r\n\r\nIt turns out that *any* use of `new Buffer(...)` or `Buffer.concat()`is terrible for performance.  \r\n\r\nReallocating a buffer during `flush()` pretty much guarantees that you're losing some very valuable performance, and returning objects from `encode()` kills it too.\r\n\r\nI started tweaking msgpack-lite and in my case exposed the Encoder type out to the consumer, and have it keep it's internal buffer, and only reallocate if the size of the current `encode()` is exceeding it.\r\n\r\nI use it like this now:\r\n\r\n``` javascript\r\n    var msgpack = require(\"msgpack-lite\");\r\n    // ....\r\n\r\n    var enc = new Encoder();\r\n    // ... \r\n\r\n    /// in a tight loop somewhere\r\n    // msg is some object from somewhere\r\n    var buf = enc.encode( msg ); \r\n\r\n    // buf is .slice() 'd  from the internal buffer in the object \r\n    // slice() is a cheap operation, since it doesn't actually allocate more data\r\n    // it just creates a view\r\n\r\n   // at that point, you can do what you want with it...\r\n   somestream.write( buf );\r\n\r\n```\r\n\r\nUsing *msgpack-lite* in this way, I'm able to encode and process around **200,000 messages per second** in on a single core in a VM.\r\n\r\nI'm working on a pull request for the changes, once I clean up the hacking mess I did :)\r\n\r\nIncidentally, in `buffer-lite.js` it mentions:\r\n```\r\n    // new Buffer(string, \"utf-8\") is SLOWER then below\r\n```\r\n\r\nI imagine that Buffer allocation speed is what's really killing that. If you pre-allocate and re-use the buffer, and just use `buffer.write(string, 'utf-8') ` it appears to be much better.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/comments/136463515","html_url":"https://github.com/kawanet/msgpack-lite/issues/2#issuecomment-136463515","issue_url":"https://api.github.com/repos/kawanet/msgpack-lite/issues/2","id":136463515,"user":{"login":"fearthecowboy","id":133736,"avatar_url":"https://avatars.githubusercontent.com/u/133736?v=3","gravatar_id":"","url":"https://api.github.com/users/fearthecowboy","html_url":"https://github.com/fearthecowboy","followers_url":"https://api.github.com/users/fearthecowboy/followers","following_url":"https://api.github.com/users/fearthecowboy/following{/other_user}","gists_url":"https://api.github.com/users/fearthecowboy/gists{/gist_id}","starred_url":"https://api.github.com/users/fearthecowboy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fearthecowboy/subscriptions","organizations_url":"https://api.github.com/users/fearthecowboy/orgs","repos_url":"https://api.github.com/users/fearthecowboy/repos","events_url":"https://api.github.com/users/fearthecowboy/events{/privacy}","received_events_url":"https://api.github.com/users/fearthecowboy/received_events","type":"User","site_admin":false},"created_at":"2015-08-31T18:50:33Z","updated_at":"2015-08-31T18:50:33Z","body":"I'll see if I can get some time in the next day or so to take a look at the newest version and see how it plays with my code :)\r\n\r\nAnother day or two, and I should have my `EventEmitter`-style usage cleaned up, and maybe we can put that in there beside the `Transform` implementation to give developers an couple easy means for tying it to streams. \r\n\r\nI've switched over to working with the `Decode` side of my app today. We'll see what happens there :)"}}