{"action":"created","issue":{"url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/78","labels_url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/78/labels{/name}","comments_url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/78/comments","events_url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/78/events","html_url":"https://github.com/Datafable/gbif-dataset-metrics/issues/78","id":62626342,"number":78,"title":"Update metrics for all datasets","user":{"login":"peterdesmet","id":600993,"avatar_url":"https://avatars.githubusercontent.com/u/600993?v=3","gravatar_id":"","url":"https://api.github.com/users/peterdesmet","html_url":"https://github.com/peterdesmet","followers_url":"https://api.github.com/users/peterdesmet/followers","following_url":"https://api.github.com/users/peterdesmet/following{/other_user}","gists_url":"https://api.github.com/users/peterdesmet/gists{/gist_id}","starred_url":"https://api.github.com/users/peterdesmet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterdesmet/subscriptions","organizations_url":"https://api.github.com/users/peterdesmet/orgs","repos_url":"https://api.github.com/users/peterdesmet/repos","events_url":"https://api.github.com/users/peterdesmet/events{/privacy}","received_events_url":"https://api.github.com/users/peterdesmet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"niconoe","id":386387,"avatar_url":"https://avatars.githubusercontent.com/u/386387?v=3","gravatar_id":"","url":"https://api.github.com/users/niconoe","html_url":"https://github.com/niconoe","followers_url":"https://api.github.com/users/niconoe/followers","following_url":"https://api.github.com/users/niconoe/following{/other_user}","gists_url":"https://api.github.com/users/niconoe/gists{/gist_id}","starred_url":"https://api.github.com/users/niconoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/niconoe/subscriptions","organizations_url":"https://api.github.com/users/niconoe/orgs","repos_url":"https://api.github.com/users/niconoe/repos","events_url":"https://api.github.com/users/niconoe/events{/privacy}","received_events_url":"https://api.github.com/users/niconoe/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/milestones/14","html_url":"https://github.com/Datafable/gbif-dataset-metrics/milestones/Challenge%20round%202","labels_url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/milestones/14/labels","id":1248141,"number":14,"title":"Challenge round 2","description":"","creator":{"login":"peterdesmet","id":600993,"avatar_url":"https://avatars.githubusercontent.com/u/600993?v=3","gravatar_id":"","url":"https://api.github.com/users/peterdesmet","html_url":"https://github.com/peterdesmet","followers_url":"https://api.github.com/users/peterdesmet/followers","following_url":"https://api.github.com/users/peterdesmet/following{/other_user}","gists_url":"https://api.github.com/users/peterdesmet/gists{/gist_id}","starred_url":"https://api.github.com/users/peterdesmet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterdesmet/subscriptions","organizations_url":"https://api.github.com/users/peterdesmet/orgs","repos_url":"https://api.github.com/users/peterdesmet/repos","events_url":"https://api.github.com/users/peterdesmet/events{/privacy}","received_events_url":"https://api.github.com/users/peterdesmet/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":12,"state":"open","created_at":"2015-08-10T13:05:13Z","updated_at":"2015-08-31T22:45:21Z","due_on":"2015-09-03T22:00:00Z","closed_at":null},"comments":18,"created_at":"2015-03-18T08:43:07Z","updated_at":"2015-09-01T12:16:38Z","closed_at":null,"body":"This is the status so far:\r\n\r\ntype | occ_api | occ | updated | newly_created | count | remarks\r\n--- | --- | --- | --- | --- | --- | ---\r\nOCCURRENCE | false | true | true | true | 49 | OK, new datasets\r\nOCCURRENCE | true | false | false | false | 672 | Still to update\r\nOCCURRENCE | false | false | false | false | 379 | OK, no real need to update this\r\nOCCURRENCE | true | true | false | false | 1790 | Were once updated, should be done again\r\nOCCURRENCE | false | true | true | false | 19 | Since when did these datasets get occ?\r\nOCCURRENCE | false | true | false | false | 1 | Added last time, should be updated\r\nOCCURRENCE | true | true | true | false | 9472 | OK, updated datasets\r\nMETADATA | true | true | true | false | 3 | OK, odd to such datasets with occ though\r\nMETADATA | false | false | false | false | 21 | OK, no real need to update this\r\nMETADATA | true | false | false | false | 1 | OK, no real need to update this\r\nMETADATA | true | true | false | false | 1 | OK, apparently updated last time\r\nCHECKLIST | false | true | true | false | 122 | Since when did these datasets get occ?\r\nCHECKLIST | true | true | true | false | 479 | OK, recently updated\r\nCHECKLIST | true | true | false | false | 183 | OK, no real need to update this\r\nCHECKLIST | false | true | false | false | 9 | OK, apparently these datasets got occ last time\r\nCHECKLIST | false | false | false | false | 750 | OK, no real need to update this\r\nCHECKLIST | true | false | false | false | 48 | OK, no real need to update this\r\n\r\nSQL used:\r\n\r\n```SQL\r\nSELECT\r\n\ttype,\r\n\tCASE\r\n\t\tWHEN occurrences_api > 0 THEN True\r\n\t\tELSE False\r\n\tEND AS occ_api,\r\n\tCASE\r\n\t\tWHEN occurrences > 0 THEN True\r\n\t\tELSE False\r\n\tEND AS occ,\r\n\tCASE\r\n\t\tWHEN updated_at >= '2015-03-12' THEN True\r\n\t\tELSE False\r\n\tEND AS updated,\r\n\tCASE\r\n\t\tWHEN created_at >= '2015-03-12' THEN True\r\n\t\tELSE False\r\n\tEND AS newly_created,\r\n\tcount(*)\r\nFROM gbif_dataset_metrics_test\r\nGROUP BY\r\n\ttype,\r\n\tocc_api,\r\n\tocc,\r\n\tupdated,\r\n\tnewly_created\r\nORDER BY\r\n\ttype DESC\r\n```"},"comment":{"url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/comments/136692486","html_url":"https://github.com/Datafable/gbif-dataset-metrics/issues/78#issuecomment-136692486","issue_url":"https://api.github.com/repos/Datafable/gbif-dataset-metrics/issues/78","id":136692486,"user":{"login":"niconoe","id":386387,"avatar_url":"https://avatars.githubusercontent.com/u/386387?v=3","gravatar_id":"","url":"https://api.github.com/users/niconoe","html_url":"https://github.com/niconoe","followers_url":"https://api.github.com/users/niconoe/followers","following_url":"https://api.github.com/users/niconoe/following{/other_user}","gists_url":"https://api.github.com/users/niconoe/gists{/gist_id}","starred_url":"https://api.github.com/users/niconoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/niconoe/subscriptions","organizations_url":"https://api.github.com/users/niconoe/orgs","repos_url":"https://api.github.com/users/niconoe/repos","events_url":"https://api.github.com/users/niconoe/events{/privacy}","received_events_url":"https://api.github.com/users/niconoe/received_events","type":"User","site_admin":false},"created_at":"2015-09-01T12:16:38Z","updated_at":"2015-09-01T12:16:38Z","body":"Other suggestion of trying to unzip on your computer: yeah, why not. The link to my version is http://www.gbif.org/occurrence/download/0000313-150831095807960, yours would be perfect too if that works.\r\n\r\nTo do this you'll need an unzip tool that support large files (ZIP64). Around 600Gb on free space, on a removable drive that you can pass me later. You don't need to extract verbatim.txt, but all the other files are necessary.\r\n\r\nGiven the time everything takes (just transfer for example), it has a high chance of being unnecessary (if I succeed before you). But doing in in parallel maybe useful, just in case. I keep you in touch of the progress!"}}