{"action":"created","issue":{"url":"https://api.github.com/repos/sahib/rmlint/issues/109","labels_url":"https://api.github.com/repos/sahib/rmlint/issues/109/labels{/name}","comments_url":"https://api.github.com/repos/sahib/rmlint/issues/109/comments","events_url":"https://api.github.com/repos/sahib/rmlint/issues/109/events","html_url":"https://github.com/sahib/rmlint/issues/109","id":64112251,"number":109,"title":"Bounded resource usage (feature)","user":{"login":"vvs-","id":5171934,"avatar_url":"https://avatars.githubusercontent.com/u/5171934?v=3","gravatar_id":"","url":"https://api.github.com/users/vvs-","html_url":"https://github.com/vvs-","followers_url":"https://api.github.com/users/vvs-/followers","following_url":"https://api.github.com/users/vvs-/following{/other_user}","gists_url":"https://api.github.com/users/vvs-/gists{/gist_id}","starred_url":"https://api.github.com/users/vvs-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vvs-/subscriptions","organizations_url":"https://api.github.com/users/vvs-/orgs","repos_url":"https://api.github.com/users/vvs-/repos","events_url":"https://api.github.com/users/vvs-/events{/privacy}","received_events_url":"https://api.github.com/users/vvs-/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/sahib/rmlint/labels/Feature","name":"Feature","color":"02e10c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-03-24T22:11:35Z","updated_at":"2015-03-30T08:21:59Z","closed_at":null,"body":"Consider this use case. I'm really stuck here trying to deduplicate over 5 million files. The memory usage goes far beyond available RAM and using large swap won't help because processing takes eons. I think that using some database, like e.g. sqlite, to hold files metadata would be a lifesaver. This is similar to #25 but with different scope: to limit memory usage and make rmlint scalable on huge number of files."},"comment":{"url":"https://api.github.com/repos/sahib/rmlint/issues/comments/87589190","html_url":"https://github.com/sahib/rmlint/issues/109#issuecomment-87589190","issue_url":"https://api.github.com/repos/sahib/rmlint/issues/109","id":87589190,"user":{"login":"sahib","id":376651,"avatar_url":"https://avatars.githubusercontent.com/u/376651?v=3","gravatar_id":"","url":"https://api.github.com/users/sahib","html_url":"https://github.com/sahib","followers_url":"https://api.github.com/users/sahib/followers","following_url":"https://api.github.com/users/sahib/following{/other_user}","gists_url":"https://api.github.com/users/sahib/gists{/gist_id}","starred_url":"https://api.github.com/users/sahib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sahib/subscriptions","organizations_url":"https://api.github.com/users/sahib/orgs","repos_url":"https://api.github.com/users/sahib/repos","events_url":"https://api.github.com/users/sahib/events{/privacy}","received_events_url":"https://api.github.com/users/sahib/received_events","type":"User","site_admin":false},"created_at":"2015-03-30T08:21:59Z","updated_at":"2015-03-30T08:21:59Z","body":"```rmlint``` has first to know if it's a hardlink before it can ignore it. ```-L``` won't do anything here of course. The solution is rather to fix the relevant code, not to work around it.\r\n\r\n@BTW: What is `it`? The ```progressbar```? It is already a formatter. Please be a bit more specific when writing answers, especially including the command you run. It takes me sometimes way too much time to figure out what you mean sometimes."}}