{"action":"created","issue":{"url":"https://api.github.com/repos/datasets/registry/issues/92","labels_url":"https://api.github.com/repos/datasets/registry/issues/92/labels{/name}","comments_url":"https://api.github.com/repos/datasets/registry/issues/92/comments","events_url":"https://api.github.com/repos/datasets/registry/issues/92/events","html_url":"https://github.com/datasets/registry/pull/92","id":72008888,"number":92,"title":"Fix scraper","user":{"login":"Deiz","id":84350,"avatar_url":"https://avatars.githubusercontent.com/u/84350?v=3","gravatar_id":"","url":"https://api.github.com/users/Deiz","html_url":"https://github.com/Deiz","followers_url":"https://api.github.com/users/Deiz/followers","following_url":"https://api.github.com/users/Deiz/following{/other_user}","gists_url":"https://api.github.com/users/Deiz/gists{/gist_id}","starred_url":"https://api.github.com/users/Deiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Deiz/subscriptions","organizations_url":"https://api.github.com/users/Deiz/orgs","repos_url":"https://api.github.com/users/Deiz/repos","events_url":"https://api.github.com/users/Deiz/events{/privacy}","received_events_url":"https://api.github.com/users/Deiz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-29T22:37:15Z","updated_at":"2015-04-30T13:49:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/datasets/registry/pulls/92","html_url":"https://github.com/datasets/registry/pull/92","diff_url":"https://github.com/datasets/registry/pull/92.diff","patch_url":"https://github.com/datasets/registry/pull/92.patch"},"body":"It came up earlier [in #okfn](https://botbot.me/freenode/okfn/2015-04-29/?msg=37903679&page=1) that GitHub's code search appears to have changed. It returns far more results and seems to now match filenames very fuzzily.\r\n\r\nI constructed the most restrictive search I could manage and made the scraper somewhat more fault-tolerant. It now verifies that the result for each repo is in fact a datapackage.json file, and will ignore everything else, eventually exiting when five full pages of irrelevant results have been encountered.\r\n\r\nIt's still far from perfect, and is mostly useful as a coarse discovery tool, not a way to build a definitive list."},"comment":{"url":"https://api.github.com/repos/datasets/registry/issues/comments/97799852","html_url":"https://github.com/datasets/registry/pull/92#issuecomment-97799852","issue_url":"https://api.github.com/repos/datasets/registry/issues/92","id":97799852,"user":{"login":"pdehaye","id":3274335,"avatar_url":"https://avatars.githubusercontent.com/u/3274335?v=3","gravatar_id":"","url":"https://api.github.com/users/pdehaye","html_url":"https://github.com/pdehaye","followers_url":"https://api.github.com/users/pdehaye/followers","following_url":"https://api.github.com/users/pdehaye/following{/other_user}","gists_url":"https://api.github.com/users/pdehaye/gists{/gist_id}","starred_url":"https://api.github.com/users/pdehaye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdehaye/subscriptions","organizations_url":"https://api.github.com/users/pdehaye/orgs","repos_url":"https://api.github.com/users/pdehaye/repos","events_url":"https://api.github.com/users/pdehaye/events{/privacy}","received_events_url":"https://api.github.com/users/pdehaye/received_events","type":"User","site_admin":false},"created_at":"2015-04-30T13:49:22Z","updated_at":"2015-04-30T13:49:22Z","body":"FWIW, I get 216 datasets when running the scraper a few times, way more than the 120 currently in `catalog-list.txt`. Any guidance on how to choose which to transition to `catalog-list`? \r\n \r\n```\r\nhttps://github.com/Btux/depute-as-nat-sn\r\nhttps://github.com/CDECatapult/data.gp-practice-england\r\nhttps://github.com/CDECatapult/data.prescription-pca-annual-ts\r\nhttps://github.com/DemocracyClub/UK-Political-Parties\r\nhttps://github.com/Ebnalblad/top-level-domain-names\r\nhttps://github.com/ElenaTomas/FirstTest\r\nhttps://github.com/FREERANGEINC/makerspaces\r\nhttps://github.com/PV-Log/Open-Photovoltaic-Inverter-Database\r\nhttps://github.com/PV-Log/Open-Photovoltaic-Module-Database\r\nhttps://github.com/RDBinns/datactrl\r\nhttps://github.com/Reproducible-Science-Curriculum/rr-init\r\nhttps://github.com/SiciliaHub/albopretoriopa\r\nhttps://github.com/SomethingNewUK/candidates\r\nhttps://github.com/SomethingNewUK/finances\r\nhttps://github.com/TMMV/obitos_em_portugal\r\nhttps://github.com/YeoLab/bodymap2\r\nhttps://github.com/YeoLab/flotilla_test_data\r\nhttps://github.com/YeoLab/shalek2013\r\nhttps://github.com/aliounedia/bce-data\r\nhttps://github.com/aliounedia/senegal-companies\r\nhttps://github.com/amercader/car-fuel-and-emissions\r\nhttps://github.com/bluechi/citiesinoman\r\nhttps://github.com/c4fcm/nj-municipalities\r\nhttps://github.com/cbenz/ccomptes-affected-taxes\r\nhttps://github.com/centraldedados/codigos_postais\r\nhttps://github.com/centraldedados/contratos_publicos\r\nhttps://github.com/centraldedados/eleicoes-autarquicas\r\nhttps://github.com/centraldedados/eleicoes-datas\r\nhttps://github.com/centraldedados/eleicoes-europeias\r\nhttps://github.com/centraldedados/governos\r\nhttps://github.com/centraldedados/incendios\r\nhttps://github.com/centraldedados/nomes_proprios\r\nhttps://github.com/centraldedados/obitos\r\nhttps://github.com/centraldedados/parlamento-datas\r\nhttps://github.com/centraldedados/parlamento-datas_sessoes\r\nhttps://github.com/centraldedados/parlamento-deputados\r\nhttps://github.com/centraldedados/partidos\r\nhttps://github.com/centraldedados/violencia-domestica\r\nhttps://github.com/charliedotau/price-of-solar-panels-australia\r\nhttps://github.com/christophergandrud/euro_membership\r\nhttps://github.com/cihai/cihaidata-unihan\r\nhttps://github.com/codeforamerica/OpenReferral\r\nhttps://github.com/darenasc/presupuesto-2015\r\nhttps://github.com/datasets/airport-codes\r\nhttps://github.com/datasets/bond-yields-uk-10y\r\nhttps://github.com/datasets/bond-yields-us-10y\r\nhttps://github.com/datasets/browser-stats\r\nhttps://github.com/datasets/co2-fossil-global\r\nhttps://github.com/datasets/co2-ppm\r\nhttps://github.com/datasets/cofog\r\nhttps://github.com/datasets/corruption-perceptions-index\r\nhttps://github.com/datasets/country-codes\r\nhttps://github.com/datasets/country-list\r\nhttps://github.com/datasets/cpi\r\nhttps://github.com/datasets/cpi-us\r\nhttps://github.com/datasets/currency-codes\r\nhttps://github.com/datasets/employment-us\r\nhttps://github.com/datasets/euribor\r\nhttps://github.com/datasets/ex-geojson\r\nhttps://github.com/datasets/ex-topojson\r\nhttps://github.com/datasets/finance-vix\r\nhttps://github.com/datasets/gdp\r\nhttps://github.com/datasets/gdp-uk\r\nhttps://github.com/datasets/gdp-us\r\nhttps://github.com/datasets/geo-boundaries-us-110m\r\nhttps://github.com/datasets/geo-boundaries-world-110m\r\nhttps://github.com/datasets/global-temp\r\nhttps://github.com/datasets/glwd\r\nhttps://github.com/datasets/gold-prices\r\nhttps://github.com/datasets/house-prices-uk\r\nhttps://github.com/datasets/house-prices-us\r\nhttps://github.com/datasets/imf-weo\r\nhttps://github.com/datasets/investor-flow-of-funds-us\r\nhttps://github.com/datasets/land-matrix\r\nhttps://github.com/datasets/language-codes\r\nhttps://github.com/datasets/lme-large-marine-ecosystems\r\nhttps://github.com/datasets/media-types\r\nhttps://github.com/datasets/membership-to-copyright-treaties\r\nhttps://github.com/datasets/population\r\nhttps://github.com/datasets/ppp\r\nhttps://github.com/datasets/s-and-p-500\r\nhttps://github.com/datasets/s-and-p-500-companies\r\nhttps://github.com/datasets/uk-sic-2007-condensed\r\nhttps://github.com/datasets/un-locode\r\nhttps://github.com/datasets/unece-package-codes\r\nhttps://github.com/datasets/zopa\r\nhttps://github.com/davetaz/data-publishing-template\r\nhttps://github.com/dd207/odi-open-data-leaders-network-feedback\r\nhttps://github.com/edobejar/geoip2\r\nhttps://github.com/edobejar/presupuesto\r\nhttps://github.com/ewan-klein/natural-neighbourhoods\r\nhttps://github.com/flother/glasgow-cycle-hire\r\nhttps://github.com/flother/landnr\r\nhttps://github.com/fozy81/nuclear-ghost\r\nhttps://github.com/git-data-publisher/Here-is-some-data\r\nhttps://github.com/git-data-publisher/Transactions-over-25-000-in-the-Foreign-and-Commonwealth-Office\r\nhttps://github.com/hdurand/uk-public-spending\r\nhttps://github.com/hhtyo/great-east-japan-earthquake-evacuees\r\nhttps://github.com/hibariba/data.Antihist-Hyposensit-Allergic\r\nhttps://github.com/hibariba/data.Antihist-Hyposensit-Allergic-ts\r\nhttps://github.com/hibariba/data.template\r\nhttps://github.com/investigacion/candidatos-facebook-data\r\nhttps://github.com/investigacion/delitos-data\r\nhttps://github.com/investigacion/desarrollo-humano-data\r\nhttps://github.com/investigacion/divisiones-territoriales-data\r\nhttps://github.com/investigacion/ingreso-turistas-data\r\nhttps://github.com/investigacion/iniciativas-data\r\nhttps://github.com/investigacion/morosos\r\nhttps://github.com/investigacion/reciclaje-data\r\nhttps://github.com/investigacion/salidas-data\r\nhttps://github.com/iwlearn/project-data\r\nhttps://github.com/jacquestardie/library\r\nhttps://github.com/jeff-s-zhou/ai_final_project\r\nhttps://github.com/joehand/nyse-listings\r\nhttps://github.com/johnlsheridan/occupations\r\nhttps://github.com/jrnold/CDB13\r\nhttps://github.com/jrnold/civil_war_era_findata\r\nhttps://github.com/jrnold/playfair-data\r\nhttps://github.com/karwana/ayuda-data\r\nhttps://github.com/matt40k/SchoolsWebsites-England\r\nhttps://github.com/mattcg/language-subtag-registry\r\nhttps://github.com/medialab/toflit18_data\r\nhttps://github.com/michalskop/CZ-Praha-2010-2014-Roll-call-Votes\r\nhttps://github.com/michalskop/SK-municipalities-geocoded\r\nhttps://github.com/mihi-tr/bosnia-tenders\r\nhttps://github.com/mihi-tr/dpkg-medientransparenz\r\nhttps://github.com/mihi-tr/frontex-budgets-dpkg\r\nhttps://github.com/mihi-tr/list-of-deaths-dpkg\r\nhttps://github.com/mihi-tr/parteien-oesterreich-dpkg\r\nhttps://github.com/mihi-tr/tanzania-census-dpkg\r\nhttps://github.com/mikehiggins/hk-govt-land-sales\r\nhttps://github.com/mspratt/OKFN_Data_Challenge_3\r\nhttps://github.com/mxmmargarida/fct-projetos-financiados\r\nhttps://github.com/nesta-uk/UK-makerspaces\r\nhttps://github.com/nporwal/cdb13_ml_analysis\r\nhttps://github.com/okffi/popup-katalogi\r\nhttps://github.com/okfn/opendatasites\r\nhttps://github.com/okfn/publicbodies\r\nhttps://github.com/opendatamali/sos_hotline_data\r\nhttps://github.com/opendatamali/sos_hotline_data2\r\nhttps://github.com/openspending/dpkg-cra\r\nhttps://github.com/paulbradshaw/Islington-Food-Standards-Agency-inspections\r\nhttps://github.com/pdehaye/safe-harbor\r\nhttps://github.com/peterdesmet/traffic-signs-hansbeke\r\nhttps://github.com/pezholio/cid-test\r\nhttps://github.com/pezholio/india-pale-kolsch\r\nhttps://github.com/pezholio/odipa\r\nhttps://github.com/pezholio/transactions-over-25-000-in-the-foreign-and-commonwealth-office\r\nhttps://github.com/ppKrauss/getlex\r\nhttps://github.com/ppKrauss/habemus-papam\r\nhttps://github.com/ppKrauss/schemaOrg-wikidata\r\nhttps://github.com/ppKrauss/standard-URNs\r\nhttps://github.com/prmkbr/data_pkgs\r\nhttps://github.com/pwalsh/il-government-structure\r\nhttps://github.com/pwalsh/il-municipal-budget-data\r\nhttps://github.com/pwalsh/il-municipal-budget-tree\r\nhttps://github.com/pwalsh/il-municipal-statistics\r\nhttps://github.com/radproject/2011-nca-results\r\nhttps://github.com/radproject/electoral-districts\r\nhttps://github.com/radproject/electoral-sub-districts\r\nhttps://github.com/radproject/general-election-statistics\r\nhttps://github.com/radproject/parliamentary-results\r\nhttps://github.com/radproject/pollingcenters\r\nhttps://github.com/radproject/presidential-rnd1-results\r\nhttps://github.com/radproject/presidential-round2-results\r\nhttps://github.com/relet/ssr\r\nhttps://github.com/rgrp/california-finances\r\nhttps://github.com/rgrp/crime-data-sf\r\nhttps://github.com/rgrp/dataset-gla\r\nhttps://github.com/rgrp/hard-drive-prices\r\nhttps://github.com/rgrp/health-data-uk\r\nhttps://github.com/rgrp/music-sales\r\nhttps://github.com/rgrp/search-engine-market-shares\r\nhttps://github.com/rgrp/wheat-us\r\nhttps://github.com/rossmounce/finance-vix\r\nhttps://github.com/sanjaypoyzer/general-election-candidates-2015-datapackage\r\nhttps://github.com/sbreslav/dataset-user-interactions-mimic\r\nhttps://github.com/senegalouvert-datasets/Decoupage-Administratif\r\nhttps://github.com/senegalouvert-datasets/Education\r\nhttps://github.com/senegalouvert-datasets/Indice_National_Prix_Consommation\r\nhttps://github.com/senegalouvert-datasets/Marches_Publics\r\nhttps://github.com/senegalouvert-datasets/Observatoires-Internet\r\nhttps://github.com/senegalouvert-datasets/Repartition-Population-1970-2005\r\nhttps://github.com/senegalouvert/Deces-dus-au-SIDA\r\nhttps://github.com/senegalouvert/Decoupage-Administratif\r\nhttps://github.com/senegalouvert/Enseignement-primaire-CITE-1-inscription-4e-annee\r\nhttps://github.com/senegalouvert/Population-Senegal\r\nhttps://github.com/senegalouvert/arachides-avec-la-coquille\r\nhttps://github.com/senegalouvert/carbureacteurs\r\nhttps://github.com/senegalouvert/commerce-des-marchandises-Cereales\r\nhttps://github.com/senegalouvert/commerce-des-marchandises-animaux-vivants\r\nhttps://github.com/senegalouvert/commerce-des-marchandises-coton\r\nhttps://github.com/senegalouvert/commerce-des-marchandises-et-ouvrages-en-etain\r\nhttps://github.com/senegalouvert/electricite-total\r\nhttps://github.com/senegalouvert/elevage-primaire\r\nhttps://github.com/senegalouvert/elevage-transforme\r\nhttps://github.com/senegalouvert/population-senegal-recensee\r\nhttps://github.com/skitazaki/datasets-locref-jp\r\nhttps://github.com/skitazaki/datasets-zipcode-jp\r\nhttps://github.com/stevesong/dp-afterfibre\r\nhttps://github.com/stevesong/old-afterfibre\r\nhttps://github.com/texttochange/uganda-admin-divisions\r\nhttps://github.com/tfgg/yournextmp-mep-match\r\nhttps://github.com/theodi/cid-test\r\nhttps://github.com/theodi/csv-validation-research\r\nhttps://github.com/theodi/data-publishing-template\r\nhttps://github.com/theodi/dataset-metadata-survey\r\nhttps://github.com/theodi/dataset-mod-disposals\r\nhttps://github.com/theodi/euro-elections\r\nhttps://github.com/theodi/hot-drinks\r\nhttps://github.com/theodi/test-data\r\nhttps://github.com/tlevine/open-data-500\r\nhttps://github.com/tlevine/open-data-index\r\nhttps://github.com/transparenciahackday/acesso-ensino-superior-medicina\r\nhttps://github.com/transporteaberto/osm-bicicletarios\r\nhttps://github.com/ukodi-training/hot-drinks\r\n```\r\n"}}