{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17","labels_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/comments","events_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17/events","html_url":"https://github.com/mozilla-services/shavar-list-creation/issues/17","id":102246582,"number":17,"title":"Put a minimum and maximum file size on the resulting lists","user":{"login":"fmarier","id":167821,"avatar_url":"https://avatars.githubusercontent.com/u/167821?v=3","gravatar_id":"","url":"https://api.github.com/users/fmarier","html_url":"https://github.com/fmarier","followers_url":"https://api.github.com/users/fmarier/followers","following_url":"https://api.github.com/users/fmarier/following{/other_user}","gists_url":"https://api.github.com/users/fmarier/gists{/gist_id}","starred_url":"https://api.github.com/users/fmarier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmarier/subscriptions","organizations_url":"https://api.github.com/users/fmarier/orgs","repos_url":"https://api.github.com/users/fmarier/repos","events_url":"https://api.github.com/users/fmarier/events{/privacy}","received_events_url":"https://api.github.com/users/fmarier/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-08-20T21:58:33Z","updated_at":"2015-11-04T21:49:11Z","closed_at":null,"body":"As a fail-safe, we should ensure that the total size of each list is within a reasonable bound.\r\n\r\nI suggest the following starting values:\r\n\r\n- tracking protection blacklists: 50-70k\r\n- tracking protection entity whitelist: 250-400k\r\n\r\nIf the generated list is too large. Abort, throw an error, notify some people, etc. but keep serving the old list."},"comment":{"url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/comments/153877004","html_url":"https://github.com/mozilla-services/shavar-list-creation/issues/17#issuecomment-153877004","issue_url":"https://api.github.com/repos/mozilla-services/shavar-list-creation/issues/17","id":153877004,"user":{"login":"kontaxis","id":6628990,"avatar_url":"https://avatars.githubusercontent.com/u/6628990?v=3","gravatar_id":"","url":"https://api.github.com/users/kontaxis","html_url":"https://github.com/kontaxis","followers_url":"https://api.github.com/users/kontaxis/followers","following_url":"https://api.github.com/users/kontaxis/following{/other_user}","gists_url":"https://api.github.com/users/kontaxis/gists{/gist_id}","starred_url":"https://api.github.com/users/kontaxis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kontaxis/subscriptions","organizations_url":"https://api.github.com/users/kontaxis/orgs","repos_url":"https://api.github.com/users/kontaxis/repos","events_url":"https://api.github.com/users/kontaxis/events{/privacy}","received_events_url":"https://api.github.com/users/kontaxis/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T21:49:11Z","updated_at":"2015-11-04T21:49:11Z","body":"Instead of checking the size of the list we could check how much it differs from older lists. This not only looks for huge or tiny lists but also for lists that have radically changed in terms of content without changing in size.\r\n\r\nThe Jaccard index or Dice coefficient look at how many entries two lists have in common and return their similarity. So for instance we can say that the old list and the new list are 90% similar. This makes more sense when providing quality assurance. If the new list is less than x% similar to the old one then something might be wrong. To determine the warning threshold we can see how much the list has changed in the past.\r\n\r\nhttps://en.wikipedia.org/wiki/Jaccard_index"}}