{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17","labels_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/comments","events_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/events","html_url":"https://github.com/mozilla-services/shavar-list-creation/issues/17","id":102246582,"number":17,"title":"Put a maximum file size on the resulting lists","user":{"login":"fmarier","id":167821,"avatar_url":"https://avatars.githubusercontent.com/u/167821?v=3","gravatar_id":"","url":"https://api.github.com/users/fmarier","html_url":"https://github.com/fmarier","followers_url":"https://api.github.com/users/fmarier/followers","following_url":"https://api.github.com/users/fmarier/following{/other_user}","gists_url":"https://api.github.com/users/fmarier/gists{/gist_id}","starred_url":"https://api.github.com/users/fmarier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmarier/subscriptions","organizations_url":"https://api.github.com/users/fmarier/orgs","repos_url":"https://api.github.com/users/fmarier/repos","events_url":"https://api.github.com/users/fmarier/events{/privacy}","received_events_url":"https://api.github.com/users/fmarier/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-20T21:58:33Z","updated_at":"2015-08-24T20:37:35Z","closed_at":null,"body":"As a fail-safe, we should cap the total file size of each list.\r\n\r\nI suggest the following starting values:\r\n\r\n- tracking protection blacklists: 100k\r\n- tracking protection entity whiteliste: 2M\r\n\r\nIf the generated list is too large. Abort, throw an error, notify some people, etc. but keep serving the old list."},"comment":{"url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/comments/134372280","html_url":"https://github.com/mozilla-services/shavar-list-creation/issues/17#issuecomment-134372280","issue_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17","id":134372280,"user":{"login":"fmarier","id":167821,"avatar_url":"https://avatars.githubusercontent.com/u/167821?v=3","gravatar_id":"","url":"https://api.github.com/users/fmarier","html_url":"https://github.com/fmarier","followers_url":"https://api.github.com/users/fmarier/followers","following_url":"https://api.github.com/users/fmarier/following{/other_user}","gists_url":"https://api.github.com/users/fmarier/gists{/gist_id}","starred_url":"https://api.github.com/users/fmarier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmarier/subscriptions","organizations_url":"https://api.github.com/users/fmarier/orgs","repos_url":"https://api.github.com/users/fmarier/repos","events_url":"https://api.github.com/users/fmarier/events{/privacy}","received_events_url":"https://api.github.com/users/fmarier/received_events","type":"User","site_admin":false},"created_at":"2015-08-24T20:37:35Z","updated_at":"2015-08-24T20:37:35Z","body":"> Should we cap the size of each list or for each new chunk generated?  While this script doesn't currently generate multiple chunks the publishing process should do that as necessary.\r\n\r\nFor the purposes of detecting mistakes in list generation or in the upstream list, I would suggest only looking at the total list size for now.\r\n\r\n> My only concern there is that some research is needed to identify an optimum max size for chunks based various factors including download efficiency for mobile devices and cost efficiency for serving the data at least.\r\n\r\nIndeed and Safe Browsing v4 promises to improve on that, but I guess we'll find out once we start implementing it."}}