{"action":"created","issue":{"url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/23","labels_url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/23/comments","events_url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/23/events","html_url":"https://github.com/siacs/HttpUploadComponent/pull/23","id":97354509,"number":23,"title":"Add expiry and user quotas","user":{"login":"chaotix-","id":9901421,"avatar_url":"https://avatars.githubusercontent.com/u/9901421?v=3","gravatar_id":"","url":"https://api.github.com/users/chaotix-","html_url":"https://github.com/chaotix-","followers_url":"https://api.github.com/users/chaotix-/followers","following_url":"https://api.github.com/users/chaotix-/following{/other_user}","gists_url":"https://api.github.com/users/chaotix-/gists{/gist_id}","starred_url":"https://api.github.com/users/chaotix-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chaotix-/subscriptions","organizations_url":"https://api.github.com/users/chaotix-/orgs","repos_url":"https://api.github.com/users/chaotix-/repos","events_url":"https://api.github.com/users/chaotix-/events{/privacy}","received_events_url":"https://api.github.com/users/chaotix-/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-26T22:17:47Z","updated_at":"2015-07-26T22:54:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/siacs/HttpUploadComponent/pulls/23","html_url":"https://github.com/siacs/HttpUploadComponent/pull/23","diff_url":"https://github.com/siacs/HttpUploadComponent/pull/23.diff","patch_url":"https://github.com/siacs/HttpUploadComponent/pull/23.patch"},"body":"Expiry is controlled by 'expire_interval' and 'expire_maxage' in the config file.\r\n\r\n'expire_interval' sets how often old files should be expired (in\r\n    seconds). A '0' (or not setting the entry at all) disables expiry.\r\n\r\n'expire_maxage' sets the max age in seconds before files get expired.\r\n\r\n'user_quota' sets the maximum space in bytes that a user might use.\r\n    Uploads that will exceed this limit will be refused.\r\n    A '0' (or not setting the entry at all) disables user quotas.\r\n\r\nPlease note that the line \"from threading import Thread\" that was removed in pull request #22 because it was not necessary is needed for this PR. \r\n\r\nExpiry also deletes empty directories (as opposed to a simple cron job as in PR #20) to keeps things tidy."},"comment":{"url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/comments/125046662","html_url":"https://github.com/siacs/HttpUploadComponent/pull/23#issuecomment-125046662","issue_url":"https://api.github.com/repos/siacs/HttpUploadComponent/issues/23","id":125046662,"user":{"login":"chaotix-","id":9901421,"avatar_url":"https://avatars.githubusercontent.com/u/9901421?v=3","gravatar_id":"","url":"https://api.github.com/users/chaotix-","html_url":"https://github.com/chaotix-","followers_url":"https://api.github.com/users/chaotix-/followers","following_url":"https://api.github.com/users/chaotix-/following{/other_user}","gists_url":"https://api.github.com/users/chaotix-/gists{/gist_id}","starred_url":"https://api.github.com/users/chaotix-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chaotix-/subscriptions","organizations_url":"https://api.github.com/users/chaotix-/orgs","repos_url":"https://api.github.com/users/chaotix-/repos","events_url":"https://api.github.com/users/chaotix-/events{/privacy}","received_events_url":"https://api.github.com/users/chaotix-/received_events","type":"User","site_admin":false},"created_at":"2015-07-26T22:54:00Z","updated_at":"2015-07-26T22:54:00Z","body":"I thought about that but I saw some problems and i didn't know if solving them was worth the IO or RAM cost.\r\n\r\n1. Expiry has to be more or less instantaneous. If there is an expiry run only every x hours a user may exceed the quote by a great margin. If only the files exceeding the quote are deleted there will almost always be an expiry after a user filled up his quota for the first time. So with every upload there needs to be a complete directory scan of the users files or the metadata of all files for all users must be kept in memory at all times. I don't know if this is wise for scalability reasons. An alternative would be to store the metadata in a database. But that would add maybe unwanted complexity and dependencies in setting this up.\r\n\r\n2. If files only get deleted when the user quota is exceeded old stuff of inactive users will hang around indefinately (although a seperate expiry mechanism like the one in this PR would solve this problem).\r\n\r\nSo what do you propose? Let a user exceed the quota for x hours, keep all metadata in memory or store it in a database?"}}