{"action":"created","issue":{"url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/37","labels_url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/37/comments","events_url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/37/events","html_url":"https://github.com/NeowayLabs/neosearch/issues/37","id":104267460,"number":37,"title":"OOM crash when indexing","user":{"login":"tiago4orion","id":251478,"avatar_url":"https://avatars.githubusercontent.com/u/251478?v=3","gravatar_id":"","url":"https://api.github.com/users/tiago4orion","html_url":"https://github.com/tiago4orion","followers_url":"https://api.github.com/users/tiago4orion/followers","following_url":"https://api.github.com/users/tiago4orion/following{/other_user}","gists_url":"https://api.github.com/users/tiago4orion/gists{/gist_id}","starred_url":"https://api.github.com/users/tiago4orion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiago4orion/subscriptions","organizations_url":"https://api.github.com/users/tiago4orion/orgs","repos_url":"https://api.github.com/users/tiago4orion/repos","events_url":"https://api.github.com/users/tiago4orion/events{/privacy}","received_events_url":"https://api.github.com/users/tiago4orion/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NeowayLabs/neosearch/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"ppizarro","id":154961,"avatar_url":"https://avatars.githubusercontent.com/u/154961?v=3","gravatar_id":"","url":"https://api.github.com/users/ppizarro","html_url":"https://github.com/ppizarro","followers_url":"https://api.github.com/users/ppizarro/followers","following_url":"https://api.github.com/users/ppizarro/following{/other_user}","gists_url":"https://api.github.com/users/ppizarro/gists{/gist_id}","starred_url":"https://api.github.com/users/ppizarro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppizarro/subscriptions","organizations_url":"https://api.github.com/users/ppizarro/orgs","repos_url":"https://api.github.com/users/ppizarro/repos","events_url":"https://api.github.com/users/ppizarro/events{/privacy}","received_events_url":"https://api.github.com/users/ppizarro/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/NeowayLabs/neosearch/milestones/1","html_url":"https://github.com/NeowayLabs/neosearch/milestones/v0.1","labels_url":"https://api.github.com/repos/NeowayLabs/neosearch/milestones/1/labels","id":1090191,"number":1,"title":"v0.1","description":"Our first release :-)","creator":{"login":"katcipis","id":157634,"avatar_url":"https://avatars.githubusercontent.com/u/157634?v=3","gravatar_id":"","url":"https://api.github.com/users/katcipis","html_url":"https://github.com/katcipis","followers_url":"https://api.github.com/users/katcipis/followers","following_url":"https://api.github.com/users/katcipis/following{/other_user}","gists_url":"https://api.github.com/users/katcipis/gists{/gist_id}","starred_url":"https://api.github.com/users/katcipis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/katcipis/subscriptions","organizations_url":"https://api.github.com/users/katcipis/orgs","repos_url":"https://api.github.com/users/katcipis/repos","events_url":"https://api.github.com/users/katcipis/events{/privacy}","received_events_url":"https://api.github.com/users/katcipis/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":10,"state":"open","created_at":"2015-04-30T01:20:17Z","updated_at":"2015-10-04T21:57:05Z","due_on":"2015-12-31T02:00:00Z","closed_at":null},"comments":16,"created_at":"2015-09-01T13:31:31Z","updated_at":"2015-10-06T12:32:33Z","closed_at":null,"body":"The current implementation of MergeSet operation has a know bug that causes an out-of-memory.\r\n\r\nhttps://github.com/NeowayLabs/neosearch/blob/master/lib/neosearch/store/store_leveldb.go#L143\r\n\r\nThe LevelDB driver throw an OOM when trying to GET a key that has a value bigger than the available memory. It's a very common problem because NeoSearch doesn't support any stopwords analyzer yet, then the reverse-index increases very fast for very frequent words like \"a\", \"the\", \"has\", \"that\", etc.\r\n\r\nAn idea to solve that problem is split the keys in fixed-size segments (e.g 1GB) but then we have to discuss the way we will manage that segments.\r\n\r\nAny ideas?\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/comments/145842553","html_url":"https://github.com/NeowayLabs/neosearch/issues/37#issuecomment-145842553","issue_url":"https://api.github.com/repos/NeowayLabs/neosearch/issues/37","id":145842553,"user":{"login":"tiago4orion","id":251478,"avatar_url":"https://avatars.githubusercontent.com/u/251478?v=3","gravatar_id":"","url":"https://api.github.com/users/tiago4orion","html_url":"https://github.com/tiago4orion","followers_url":"https://api.github.com/users/tiago4orion/followers","following_url":"https://api.github.com/users/tiago4orion/following{/other_user}","gists_url":"https://api.github.com/users/tiago4orion/gists{/gist_id}","starred_url":"https://api.github.com/users/tiago4orion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiago4orion/subscriptions","organizations_url":"https://api.github.com/users/tiago4orion/orgs","repos_url":"https://api.github.com/users/tiago4orion/repos","events_url":"https://api.github.com/users/tiago4orion/events{/privacy}","received_events_url":"https://api.github.com/users/tiago4orion/received_events","type":"User","site_admin":false},"created_at":"2015-10-06T12:32:33Z","updated_at":"2015-10-06T12:32:33Z","body":"Sorry for the delay.\r\n\r\nThanks for the detailed look. The code I put before is wrong =( but the idea is quite the same.\r\n\r\nThe correct pseudo code (I think) is the below:\r\n\r\n```go\r\nfunc Search(index *index.Index, dsl map[string]string, limit uint) ([]string, uint64, error) {\r\n    var (\r\n        resultDocIDs  []uint64\r\n    )\r\n\r\n    for field, value := range dsl {\r\n        // index.FilterSegments returns an iterator used to sequentially get the\r\n        // ids of each segment (already sorted)\r\n        docIterator, err := index.FilterSegments([]byte(field), []byte(value))\r\n\r\n        if err != nil {\r\n            return nil, 0, err\r\n        }\r\n\r\n        if len(resultDocIDs) == 0 {\r\n                resultDocIDs = getAllSegments(docIterator)\r\n                continue\r\n        }\r\n\r\n        for segmentIDs := docIterator.First(); docIterator.Valid(); docIterator.Next() {\r\n        // segmentIDs is an []uint64 with lenght less than (or equal to) segment_size\r\n            \r\n             // this \"and\" can led to out of memory eventually \r\n             resultDocIDs = and(resultDocIDs, segmentIDs)\r\n        }\r\n    }\r\n\r\n    results, err := ind.GetDocs(resultDocIDs, limit)\r\n    return results, uint64(len(resultDocIDs)), err\r\n}\r\n```\r\n\r\nYou can use with code below:\r\n\r\n```go\r\nsearch(index, map[string]string{\r\n                          \"name\": \"jonh\",\r\n                          \"lastname\": \"carter\",\r\n                      }, 10)\r\n```\r\n\r\nThe function getAllSegments will iterate the docIterator and get the all of the ids, but only for the first filter. The subsequent filters will apply the \"and\" function between the resultDocIDs and each segments data. Then, the implementation above has *space complexity* directly proportional to the number of integers returned by first query (name = 'john'). \r\n\r\nThis way, it's impossible to get an OOM at indexing time (original issue), but it's still possible to get an OOM in the call to function getAllSegments (or something like that).\r\n\r\nWe can avoid load the entire first filter IDs in memory re-designing the way of store segments, using fixed-size bitset for segment values. Then we can apply \"and\" and \"or\" operations only between 2 (two) segments loaded in memory, and appending the results to an output list (this output list can grow indefinitely too, but now the problem is in neosearch controlled code, we can avoid crash). But we can have other tradeoffs that I'm not yet aware this way...\r\n\r\nUsing the leveldb (skiplist) or boltdb b+tree directly to execute the intersection function we can avoid any kind of OOM?"}}