{"action":"created","issue":{"url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/139","labels_url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/139/comments","events_url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/139/events","html_url":"https://github.com/fergiemcdowall/search-index/issues/139","id":98671009,"number":139,"title":"Indexing the same corpus twice will triple the storage","user":{"login":"lannka","id":8496897,"avatar_url":"https://avatars.githubusercontent.com/u/8496897?v=3","gravatar_id":"","url":"https://api.github.com/users/lannka","html_url":"https://github.com/lannka","followers_url":"https://api.github.com/users/lannka/followers","following_url":"https://api.github.com/users/lannka/following{/other_user}","gists_url":"https://api.github.com/users/lannka/gists{/gist_id}","starred_url":"https://api.github.com/users/lannka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lannka/subscriptions","organizations_url":"https://api.github.com/users/lannka/orgs","repos_url":"https://api.github.com/users/lannka/repos","events_url":"https://api.github.com/users/lannka/events{/privacy}","received_events_url":"https://api.github.com/users/lannka/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/fergiemcdowall/search-index/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-08-03T04:49:46Z","updated_at":"2015-08-04T10:15:00Z","closed_at":null,"body":"I also tried to delete the doc before adding it back, but still getting the index file bigger and bigger. The search is working fine though, it's not returning duplicated entries, as I have manually assign the same doc IDs to those entries.\r\n\r\nIs that working as intended? In my case, I do quite a lot doc update, so I'd have a cronjob updating the indices hourly/daily. How can I make sure it'll not quickly blow up the disk?"},"comment":{"url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/comments/127556767","html_url":"https://github.com/fergiemcdowall/search-index/issues/139#issuecomment-127556767","issue_url":"https://api.github.com/repos/fergiemcdowall/search-index/issues/139","id":127556767,"user":{"login":"fergiemcdowall","id":1122333,"avatar_url":"https://avatars.githubusercontent.com/u/1122333?v=3","gravatar_id":"","url":"https://api.github.com/users/fergiemcdowall","html_url":"https://github.com/fergiemcdowall","followers_url":"https://api.github.com/users/fergiemcdowall/followers","following_url":"https://api.github.com/users/fergiemcdowall/following{/other_user}","gists_url":"https://api.github.com/users/fergiemcdowall/gists{/gist_id}","starred_url":"https://api.github.com/users/fergiemcdowall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fergiemcdowall/subscriptions","organizations_url":"https://api.github.com/users/fergiemcdowall/orgs","repos_url":"https://api.github.com/users/fergiemcdowall/repos","events_url":"https://api.github.com/users/fergiemcdowall/events{/privacy}","received_events_url":"https://api.github.com/users/fergiemcdowall/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T10:15:00Z","updated_at":"2015-08-04T10:15:00Z","body":"So actually the contents of the indexes are not always the same. There is some reference duplication happening in the reverse index ('RI~') fields when you add and delete documents. "}}