{"action":"created","issue":{"url":"https://api.github.com/repos/antirez/redis/issues/2509","labels_url":"https://api.github.com/repos/antirez/redis/issues/2509/labels{/name}","comments_url":"https://api.github.com/repos/antirez/redis/issues/2509/comments","events_url":"https://api.github.com/repos/antirez/redis/issues/2509/events","html_url":"https://github.com/antirez/redis/pull/2509","id":67308423,"number":2509,"title":"sds size classes - memory optimization","user":{"login":"oranagra","id":7045099,"avatar_url":"https://avatars.githubusercontent.com/u/7045099?v=3","gravatar_id":"","url":"https://api.github.com/users/oranagra","html_url":"https://github.com/oranagra","followers_url":"https://api.github.com/users/oranagra/followers","following_url":"https://api.github.com/users/oranagra/following{/other_user}","gists_url":"https://api.github.com/users/oranagra/gists{/gist_id}","starred_url":"https://api.github.com/users/oranagra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oranagra/subscriptions","organizations_url":"https://api.github.com/users/oranagra/orgs","repos_url":"https://api.github.com/users/oranagra/repos","events_url":"https://api.github.com/users/oranagra/events{/privacy}","received_events_url":"https://api.github.com/users/oranagra/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/antirez/redis/labels/state-proposed-feature","name":"state-proposed-feature","color":"f172e8"},{"url":"https://api.github.com/repos/antirez/redis/labels/state-work-needed","name":"state-work-needed","color":"0b02e1"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2015-04-09T08:32:54Z","updated_at":"2015-07-24T06:54:00Z","closed_at":"2015-07-24T06:54:00Z","pull_request":{"url":"https://api.github.com/repos/antirez/redis/pulls/2509","html_url":"https://github.com/antirez/redis/pull/2509","diff_url":"https://github.com/antirez/redis/pull/2509.diff","patch_url":"https://github.com/antirez/redis/pull/2509.patch"},"body":"sds strings were using 8 byte header, this has two implications:\r\n1. wasting memory - even on short strings (of less than 256 chars) we have a full 4 bytes len field.\r\n2. we are not able to hold strings that are larger than 2^31 chars. (see issues with MIGRAGE https://github.com/antirez/redis/issues/757)\r\n\r\nWith this patch, each sds instance can be one of 4 different internal types:\r\n* strings with 8bit length - 3 bytes header\r\n* strings with 16bit length - 5 bytes header\r\n* strings with 32bit length - 9 bytes header\r\n* strings with 64bit length - 17 bytes header\r\n\r\n**Test (on relatively short strings)**\r\ndebug populate 10000000\r\nused_memory of original code: 1254709872\r\nused_memory with new code: 1078723024\r\nmemory optimization: **16%**\r\n\r\n*Notice that considering internal fragmentation, there are some string lengths which will now fit into a smaller jemalloc bin, for these the result will be more 2x factor of memory reduction.\r\n"},"comment":{"url":"https://api.github.com/repos/antirez/redis/issues/comments/124367790","html_url":"https://github.com/antirez/redis/pull/2509#issuecomment-124367790","issue_url":"https://api.github.com/repos/antirez/redis/issues/2509","id":124367790,"user":{"login":"antirez","id":65632,"avatar_url":"https://avatars.githubusercontent.com/u/65632?v=3","gravatar_id":"","url":"https://api.github.com/users/antirez","html_url":"https://github.com/antirez","followers_url":"https://api.github.com/users/antirez/followers","following_url":"https://api.github.com/users/antirez/following{/other_user}","gists_url":"https://api.github.com/users/antirez/gists{/gist_id}","starred_url":"https://api.github.com/users/antirez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antirez/subscriptions","organizations_url":"https://api.github.com/users/antirez/orgs","repos_url":"https://api.github.com/users/antirez/repos","events_url":"https://api.github.com/users/antirez/events{/privacy}","received_events_url":"https://api.github.com/users/antirez/received_events","type":"User","site_admin":false},"created_at":"2015-07-24T06:54:00Z","updated_at":"2015-07-24T06:54:00Z","body":"This PR was just merged, thanks @oranagra. We verified that the speed hint is not very significative in most commands, with the exception of commands having a workload depending a lot of sdscatlen() or sdcatfmt(), where it is possible to use up to 25% of performances. However:\r\n\r\n1. We only identified `CLIENT LIST` so far having such a big loss.\r\n2. The other commands in general show a very small performance hit that sometimes could be compensated by the fact the memory footprint of the whole Redis instance is smaller.\r\n3. The commit actually adds an important functionality to SDS: the ability to don't crash when the user appends over 4GB, which was a concern in the past and is going to be a concern even more in the future as computers have more memory.\r\n4. The commit is trivial to revert if we'll change idea later for some reason, since it is very local to `sds.c` and does not affect the exported API."}}