{"action":"created","issue":{"url":"https://api.github.com/repos/okfn/ideas/issues/52","labels_url":"https://api.github.com/repos/okfn/ideas/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/okfn/ideas/issues/52/comments","events_url":"https://api.github.com/repos/okfn/ideas/issues/52/events","html_url":"https://github.com/okfn/ideas/issues/52","id":16445687,"number":52,"title":"Simple (open) PDF to text service","user":{"login":"rgrp","id":180658,"avatar_url":"https://avatars.githubusercontent.com/u/180658?v=3","gravatar_id":"","url":"https://api.github.com/users/rgrp","html_url":"https://github.com/rgrp","followers_url":"https://api.github.com/users/rgrp/followers","following_url":"https://api.github.com/users/rgrp/following{/other_user}","gists_url":"https://api.github.com/users/rgrp/gists{/gist_id}","starred_url":"https://api.github.com/users/rgrp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgrp/subscriptions","organizations_url":"https://api.github.com/users/rgrp/orgs","repos_url":"https://api.github.com/users/rgrp/repos","events_url":"https://api.github.com/users/rgrp/events{/privacy}","received_events_url":"https://api.github.com/users/rgrp/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/okfn/ideas/labels/Apps+%26+Services","name":"Apps & Services","color":"bfe5bf"},{"url":"https://api.github.com/repos/okfn/ideas/labels/Status%3A+Discussion+and+Research","name":"Status: Discussion and Research","color":"f0f0b5"},{"url":"https://api.github.com/repos/okfn/ideas/labels/Value%3A+%E2%98%85%E2%98%85","name":"Value: ★★","color":"D3132F"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2013-07-07T17:33:29Z","updated_at":"2015-01-06T18:29:29Z","closed_at":null,"body":"Looking for a simple online service to convert PDF to plain text. Features wanted:\r\n\r\n- Preferably open (i.e. underlying code is open)\r\n- Service has API\r\n- OCR *not* required (but a bonus) - working on \"normal\" PDFs (not scan PDFs)\r\n\r\nIf it does not exist would consider building.\r\n\r\n*Please add suggestions, ideas and comments in the comments*\r\n\r\n## Research\r\n\r\n### Tools and Libraries (only open-source)\r\n\r\nTaken from this **[gist list of pdf 2 xxx tools](https://gist.github.com/rgrp/5844485)**\r\n\r\ncf also http://okfnlabs.org/dataconverters/ PDF section\r\n\r\n#### Generic (PDF -> text)\r\n\r\n* [PDFMiner][pdfminer] - PDFMiner is a tool for extracting information from PDF documents. Unlike other PDF-related tools, it focuses entirely on getting and analyzing text data. PDFMiner allows one to obtain the exact location of text in a page, as well as other information such as fonts or lines. It includes a PDF converter that can transform PDF files into other text formats (such as HTML). It has an extensible PDF parser that can be used for other purposes than text analysis.\r\n  * Pure python\r\n* [pdftohtml][] - pdftohtml is a utility which converts PDF files into HTML and XML formats. Based on xpdf\r\n* [pdftoxml][] - command line utility to convert PDF to XML built on poppler.\r\n* [docsplit][] - part of DocumentCloud. Docsplit is a command-line utility and Ruby library for splitting apart documents into their component parts: searchable UTF-8 plain text via OCR if necessary, page images or thumbnails in any format, PDFs, single pages, and document metadata (title, author, number of pages...)\r\n* [pypdf2xml][] - convert PDF to XML. Built on pdfminer. Started as an alternative to poppler's pdftoxml, which didn't properly decode CID Type2 fonts in PDFs.\r\n* [pdf2htmlEX][] - Convert PDF to HTML without losing text or format. C++. Fast.\r\n\r\n[pdf2htmlEX]: http://coolwanglu.github.io/pdf2htmlEX/\r\n[pypdf2xml]: https://github.com/zejn/pypdf2xml\r\n[docsplit]: http://documentcloud.github.io/docsplit/\r\n[pdfminer]: http://www.unixuser.org/~euske/python/pdfminer/\r\n[pdftohtml]: http://pdftohtml.sourceforge.net/\r\n[pdftoxml]: http://pdftoxml.sourceforge.net/\r\n\r\n#### Tables from PDF\r\n\r\n* https://github.com/okfn/pdftables (Was scraperwiki but no longer seems to be available so here is a fork)\r\n* <http://tabula.nerdpower.org/> - open-source, designed specifically for tabular data but looks a bit of a pain to set up\r\n* <http://pdftoxml.sourceforge.net/> - one of the better for tables but have not used for a while\r\n* <http://pdftohtml.sourceforge.net/> - linux only afaict\r\n* [pdf.js](http://mozilla.github.io/pdf.js/) - you probably want a fork like [pdf2json](https://github.com/modesty/pdf2json) or [node-pdfreader](https://github.com/jviereck/node-pdfreader) that integrates this better with node. I have not tried this on tables though ...\r\n* Using scraperwiki + pdftoxml - see this recent tutorial [Get Started With Scraping – Extracting Simple Tables from PDF Documents][scoda-simple-tables]\r\n\r\n[scoda-simple-tables]: http://schoolofdata.org/2013/06/18/get-started-with-scraping-extracting-simple-tables-from-pdf-documents/\r\n\r\n### Existing open services\r\n\r\n* http://pdfx.cs.man.ac.uk/ - has a nice command line interface\r\n  * Is this open? Says at [bottom of usage](http://pdfx.cs.man.ac.uk/usage) that it is powered by http://www.utopiadocs.com/\r\n* Scraperwiki - https://views.scraperwiki.com/run/pdf-to-html-preview-1/ and [this tutorial](http://blog.scraperwiki.com/2010/12/17/scraping-pdfs-now-26-less-unpleasant-with-scraperwiki/)\r\n\r\n### Existing proprietary free or paid-for services\r\n\r\n* http://www.newocr.com/ - free, no API\r\n* http://www.free-ocr.com/ - free, no API, captcha\r\n* http://www.onlineocr.net/ - free\r\n* http://captricity.com/\r\n\r\nGoogle app engine used to do this http://developers.google.com/appengine/docs/python/conversion/overview\r\n\r\n### By Language\r\n\r\n@maxogden has this list of Node libraries and tools:\r\n\r\nhttps://gist.github.com/maxogden/5842859\r\n\r\nHere's a gist showing how to use pdf2json: https://gist.github.com/rgrp/5944247\r\n\r\n## Colophon\r\n\r\nOriginally: http://ideas.okfn.org/ideas/106/pdf-tiff-scan-to-text-conversion-service/"},"comment":{"url":"https://api.github.com/repos/okfn/ideas/issues/comments/68908788","html_url":"https://github.com/okfn/ideas/issues/52#issuecomment-68908788","issue_url":"https://api.github.com/repos/okfn/ideas/issues/52","id":68908788,"user":{"login":"mattfullerton","id":4084428,"avatar_url":"https://avatars.githubusercontent.com/u/4084428?v=3","gravatar_id":"","url":"https://api.github.com/users/mattfullerton","html_url":"https://github.com/mattfullerton","followers_url":"https://api.github.com/users/mattfullerton/followers","following_url":"https://api.github.com/users/mattfullerton/following{/other_user}","gists_url":"https://api.github.com/users/mattfullerton/gists{/gist_id}","starred_url":"https://api.github.com/users/mattfullerton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattfullerton/subscriptions","organizations_url":"https://api.github.com/users/mattfullerton/orgs","repos_url":"https://api.github.com/users/mattfullerton/repos","events_url":"https://api.github.com/users/mattfullerton/events{/privacy}","received_events_url":"https://api.github.com/users/mattfullerton/received_events","type":"User","site_admin":false},"created_at":"2015-01-06T18:29:29Z","updated_at":"2015-01-06T18:29:29Z","body":"I have a working version of Tika dev (1.8) with tesseract here: http://beta.offenedaten.de:9998/tika\r\n\r\nThis will give simple text back from a PDF. Nothing fancy.\r\n\r\nTest by doing things like:\r\n\r\n    curl -T mypdf.pdf http:///beta.offenedaten.de:9998/tika\r\n\r\nYou can run your own using Docker by doing:\r\n\r\n    sudo docker build -t tika github.com/mattfullerton/tika-tesseract-docker\r\n    sudo docker run -d -p 9998:9998 tika\r\n\r\nI'm very open to improvements to the Docker build files, I am no expert there.\r\n\r\nOCR is supported (for images), but what is lacking now (AFAIK) is detection that standard text extraction from a PDF 'failed' with a fallback to tesseract. We should look into that."}}