{"action":"created","issue":{"url":"https://api.github.com/repos/oniony/TMSU/issues/17","labels_url":"https://api.github.com/repos/oniony/TMSU/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/oniony/TMSU/issues/17/comments","events_url":"https://api.github.com/repos/oniony/TMSU/issues/17/events","html_url":"https://github.com/oniony/TMSU/issues/17","id":53882718,"number":17,"title":"(Re)introduce directory fingerprinting","user":{"login":"oniony","id":5895,"avatar_url":"https://avatars.githubusercontent.com/u/5895?v=3","gravatar_id":"","url":"https://api.github.com/users/oniony","html_url":"https://github.com/oniony","followers_url":"https://api.github.com/users/oniony/followers","following_url":"https://api.github.com/users/oniony/following{/other_user}","gists_url":"https://api.github.com/users/oniony/gists{/gist_id}","starred_url":"https://api.github.com/users/oniony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oniony/subscriptions","organizations_url":"https://api.github.com/users/oniony/orgs","repos_url":"https://api.github.com/users/oniony/repos","events_url":"https://api.github.com/users/oniony/events{/privacy}","received_events_url":"https://api.github.com/users/oniony/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-09T16:01:35Z","updated_at":"2015-01-21T09:45:48Z","closed_at":null,"body":"TMSU use to store directory fingerprints by recursively enumerating the directory, calculating fingerprints for every file within it, building a report of these fingerprints and then creating a further fingerprint of this report. For large directories this was particularly slow and not very intuitive to the user when a tagging operation was suddenly inexplicably slow.\r\n\r\nHowever, I've since realised it might be sufficient to build a directory fingerprint based only on the sizes of the files within it. This should be pretty fast and provide a largely useful fingerprint for identifying duplicate or moved directories."},"comment":{"url":"https://api.github.com/repos/oniony/TMSU/issues/comments/70809839","html_url":"https://github.com/oniony/TMSU/issues/17#issuecomment-70809839","issue_url":"https://api.github.com/repos/oniony/TMSU/issues/17","id":70809839,"user":{"login":"oniony","id":5895,"avatar_url":"https://avatars.githubusercontent.com/u/5895?v=3","gravatar_id":"","url":"https://api.github.com/users/oniony","html_url":"https://github.com/oniony","followers_url":"https://api.github.com/users/oniony/followers","following_url":"https://api.github.com/users/oniony/following{/other_user}","gists_url":"https://api.github.com/users/oniony/gists{/gist_id}","starred_url":"https://api.github.com/users/oniony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oniony/subscriptions","organizations_url":"https://api.github.com/users/oniony/orgs","repos_url":"https://api.github.com/users/oniony/repos","events_url":"https://api.github.com/users/oniony/events{/privacy}","received_events_url":"https://api.github.com/users/oniony/received_events","type":"User","site_admin":false},"created_at":"2015-01-21T09:45:48Z","updated_at":"2015-01-21T09:45:48Z","body":"I've implemented the algorithm for generating the fingerprints without any problem. However using the fingerprints in a repair is a bit more problematic: to identify where a directory has moved to would require fingerprinting every directory in the search paths specified which would potentially be very expensive.\r\n\r\nYou might think this would be a preexisting problem when repairing files but TMSU is able to build a shortlist of candidates by only considering files with an identical size as a file cannot be identical to another if its filesize is different. The size check (stat) is a relatively cheap operation compared to calculating a fingerprinting.\r\n\r\nA synonym to the file size shortcut might be to consider the number of items within the directory. This might be cheap if the directory is small but could potentially be more expensive than the fingerprint calculation if the directory has millions of items. It might be I would have to cap the number of items just like the directory fingerprint algorithm stops if the directory has too many items."}}