{"action":"created","issue":{"url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/89","labels_url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/89/labels{/name}","comments_url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/89/comments","events_url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/89/events","html_url":"https://github.com/chrsmithdemos/leveldb/issues/89","id":58997553,"number":89,"title":"Feature request: stream into the database","user":{"login":"GoogleCodeExporter","id":9614759,"avatar_url":"https://avatars.githubusercontent.com/u/9614759?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCodeExporter","html_url":"https://github.com/GoogleCodeExporter","followers_url":"https://api.github.com/users/GoogleCodeExporter/followers","following_url":"https://api.github.com/users/GoogleCodeExporter/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCodeExporter/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCodeExporter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCodeExporter/subscriptions","organizations_url":"https://api.github.com/users/GoogleCodeExporter/orgs","repos_url":"https://api.github.com/users/GoogleCodeExporter/repos","events_url":"https://api.github.com/users/GoogleCodeExporter/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCodeExporter/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/chrsmithdemos/leveldb/labels/auto-migrated","name":"auto-migrated","color":"ededed"},{"url":"https://api.github.com/repos/chrsmithdemos/leveldb/labels/Priority-Medium","name":"Priority-Medium","color":"ededed"},{"url":"https://api.github.com/repos/chrsmithdemos/leveldb/labels/Type-Defect","name":"Type-Defect","color":"ededed"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-26T00:42:19Z","updated_at":"2015-02-26T00:42:21Z","closed_at":"2015-02-26T00:42:20Z","body":"```\nFor IndexedDB in Chrome, we need to be able to store large binary blobs in our \nLevelDB instance.  However, we don't want to have to have each entire blob in \nmemory at once; these could be multiple-gigabyte objects.  We're fine with \nchopping them into chunks manually and adding them under separate keys, but we \nstill want all chunks to be committed in a single atomic action.\n\nCurrently WriteBatch is an in-memory construct, so even if we break up the Blob \ninto chunks, it's all going to be kept in memory anyway.  I'd like to request \nthat you add either a WriteBatch that spills to disk as you build it, or some \nother way to stream large data into the db.\n\nWhen reading it back out, we'll have the same contraints--we can't afford to \nhave the whole thing in memory, and will just want to grab a limited byte range \nat a time.\n```\n\n-----\nOriginal issue reported on code.google.com by `ericu@chromium.org` on 8 May 2012 at 10:31"},"comment":{"url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/comments/76099471","html_url":"https://github.com/chrsmithdemos/leveldb/issues/89#issuecomment-76099471","issue_url":"https://api.github.com/repos/chrsmithdemos/leveldb/issues/89","id":76099471,"user":{"login":"GoogleCodeExporter","id":9614759,"avatar_url":"https://avatars.githubusercontent.com/u/9614759?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCodeExporter","html_url":"https://github.com/GoogleCodeExporter","followers_url":"https://api.github.com/users/GoogleCodeExporter/followers","following_url":"https://api.github.com/users/GoogleCodeExporter/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCodeExporter/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCodeExporter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCodeExporter/subscriptions","organizations_url":"https://api.github.com/users/GoogleCodeExporter/orgs","repos_url":"https://api.github.com/users/GoogleCodeExporter/repos","events_url":"https://api.github.com/users/GoogleCodeExporter/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCodeExporter/received_events","type":"User","site_admin":false},"created_at":"2015-02-26T00:42:21Z","updated_at":"2015-02-26T00:42:21Z","body":"```\nMy bad. It's not Level-0, but the memtable. If your records are that big they'd \neffectively cause churning on the memtables. One could conceivably have an API \nthat bypasses the memtables, but you'd still have pretty massive churn as it \nseems likely you'd trigger compactions of Level-0 and Level-1, if not Level-2. \nThe net effect of all this might not be that different from locking down \nupdates to the database while you stuff in this giant record.\n\nWould you then pass in these bits of memory as mmap'd memory regions to avoid \nthem necessarily being pulled in to RAM?\n\nIt almost seems like after a certain size, you would be much better off having \na blob store which is outside of LevelDB: write to the filesystem and merely \nstore the path to the file in LevelDB. This is something that could be \nconstructed as a layer on top of LevelDB. Just a thought.\n```\n\n-----\nOriginal comment by `cbsm...@gmail.com` on 9 May 2012 at 1:28"}}