{"action":"created","issue":{"url":"https://api.github.com/repos/OpenJournal/central/issues/8","labels_url":"https://api.github.com/repos/OpenJournal/central/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/OpenJournal/central/issues/8/comments","events_url":"https://api.github.com/repos/OpenJournal/central/issues/8/events","html_url":"https://github.com/OpenJournal/central/issues/8","id":119598747,"number":8,"title":"Indexing OAI-PMH non-compliant repositories","user":{"login":"wetneb","id":309908,"avatar_url":"https://avatars.githubusercontent.com/u/309908?v=3","gravatar_id":"","url":"https://api.github.com/users/wetneb","html_url":"https://github.com/wetneb","followers_url":"https://api.github.com/users/wetneb/followers","following_url":"https://api.github.com/users/wetneb/following{/other_user}","gists_url":"https://api.github.com/users/wetneb/gists{/gist_id}","starred_url":"https://api.github.com/users/wetneb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wetneb/subscriptions","organizations_url":"https://api.github.com/users/wetneb/orgs","repos_url":"https://api.github.com/users/wetneb/repos","events_url":"https://api.github.com/users/wetneb/events{/privacy}","received_events_url":"https://api.github.com/users/wetneb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-30T22:45:53Z","updated_at":"2015-12-01T07:27:37Z","closed_at":null,"body":"So, BASE (http://www.base-search.net) indexes many repositories that have an OAI-PMH interface, which is already very useful and covers most of the respectable repositories.\r\n\r\nBut unfortunately, many repositories do not provide such an interface. And even when they provide one, the protocol is far from perfect: for instance, it is hard to detect accurately whether a full text is freely available behind a metadata record. Reaching out to the repository admins to encourage them to expose their metadata correctly is, from my experience, not effective at all.\r\n\r\nIf we want to go beyond this, I think we need to crawl!\r\n - One (conceptually) simple option would be to crawl for PDF files, extract metadata from them and dump this in a proper database with indexes. I think this is what CiteSeerX does, but only for papers within a particular field. In my experience the metadata that comes out of this is quite noisy.\r\n\r\n- The other option we have discussed would be to leverage existing scrapers (Zotero) to extract cleaner metadata from HTML pages. Zotero does the scraping pretty well, but I have no clue what crawling software we should use. Any idea? The scrapy framework looks nice but I'm a complete newcommer in this field so I have probably missed better options.\r\n\r\nHow much resources (servers) do we need for this? Where could we get them?"},"comment":{"url":"https://api.github.com/repos/OpenJournal/central/issues/comments/160882106","html_url":"https://github.com/OpenJournal/central/issues/8#issuecomment-160882106","issue_url":"https://api.github.com/repos/OpenJournal/central/issues/8","id":160882106,"user":{"login":"wetneb","id":309908,"avatar_url":"https://avatars.githubusercontent.com/u/309908?v=3","gravatar_id":"","url":"https://api.github.com/users/wetneb","html_url":"https://github.com/wetneb","followers_url":"https://api.github.com/users/wetneb/followers","following_url":"https://api.github.com/users/wetneb/following{/other_user}","gists_url":"https://api.github.com/users/wetneb/gists{/gist_id}","starred_url":"https://api.github.com/users/wetneb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wetneb/subscriptions","organizations_url":"https://api.github.com/users/wetneb/orgs","repos_url":"https://api.github.com/users/wetneb/repos","events_url":"https://api.github.com/users/wetneb/events{/privacy}","received_events_url":"https://api.github.com/users/wetneb/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T07:27:37Z","updated_at":"2015-12-01T07:27:37Z","body":"@cleegiles Thanks for the clarification, it's great that CiteSeerX is not limited to comp. sci. anymore!\r\nDo not get me wrong, the metadata CiteSeerX extracts is very useful. But at http://dissem.in we need to match publications with preprints, which is quite hard as soon as the title or the authors differ by a few words. So, metadata quality is critical for us.\r\n\r\nI can imagine that CiteSeerX requires a lot of resources indeed. We cannot afford this for dissemin, and this is the reason why I thought using scrapers would be cheaper (both in terms of bandwidth, storage, and computing) and could potentially yield cleaner metadata. But this option only works for repositories, not home pages.\r\n\r\nI have been playing around on indexing researchgate.net recently (with crawling and scraping), and I have been in touch with Mike Taylor who has started something along these lines for SSRN."}}