{"action":"created","issue":{"url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/48","labels_url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/48/comments","events_url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/48/events","html_url":"https://github.com/Bookworm-project/BookwormDB/issues/48","id":55912698,"number":48,"title":"Allow ingest from feature count files","user":{"login":"bmschmidt","id":1110758,"avatar_url":"https://avatars.githubusercontent.com/u/1110758?v=3","gravatar_id":"","url":"https://api.github.com/users/bmschmidt","html_url":"https://github.com/bmschmidt","followers_url":"https://api.github.com/users/bmschmidt/followers","following_url":"https://api.github.com/users/bmschmidt/following{/other_user}","gists_url":"https://api.github.com/users/bmschmidt/gists{/gist_id}","starred_url":"https://api.github.com/users/bmschmidt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmschmidt/subscriptions","organizations_url":"https://api.github.com/users/bmschmidt/orgs","repos_url":"https://api.github.com/users/bmschmidt/repos","events_url":"https://api.github.com/users/bmschmidt/events{/privacy}","received_events_url":"https://api.github.com/users/bmschmidt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Bookworm-project/BookwormDB/labels/Feature%20request","name":"Feature request","color":"02e10c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-29T15:26:43Z","updated_at":"2015-02-10T21:35:41Z","closed_at":null,"body":"In addition to ingesting from raw text, we should be able to ingest from feature count files. Some examples include: \r\n\r\n1. Jstor data for research files ([I've created a repo to work with them here](https://github.com/bmschmidt/dfr-bookworm)).\r\n2. Google Ngrams counts files.\r\n3. The internal Hathi feature count files; things like Ted Underwood's page-level feature token-counts.\r\n\r\nMy [last commit](https://github.com/Bookworm-project/BookwormDB/commit/b5cbfc736191052da3ae111c5ccdf9c032cbac2a) started this process. One issue is that there is no standard syntax for a \"feature count\" file, so we need to create one. I'm assuming, for the time being, \r\n1. **named pipes in the directory one above the bookworm installation** in a \r\n2. **tsv format with columns \"documentid,feature,count\".**\r\n\r\nThat's a little wonky, but it should be easy for any specific codebase to use awk or whatever to transform the original format into this.\r\n\r\nOne interesting side-effect is that features need not be \"tokens,\" just some arbitrary strings with any possible meaning. But that will require more granular control over whether you're building unigrams, bigrams, trigrams, etc; most generic feature counts only make sense as unigrams.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/comments/73788759","html_url":"https://github.com/Bookworm-project/BookwormDB/issues/48#issuecomment-73788759","issue_url":"https://api.github.com/repos/Bookworm-project/BookwormDB/issues/48","id":73788759,"user":{"login":"organisciak","id":58351,"avatar_url":"https://avatars.githubusercontent.com/u/58351?v=3","gravatar_id":"","url":"https://api.github.com/users/organisciak","html_url":"https://github.com/organisciak","followers_url":"https://api.github.com/users/organisciak/followers","following_url":"https://api.github.com/users/organisciak/following{/other_user}","gists_url":"https://api.github.com/users/organisciak/gists{/gist_id}","starred_url":"https://api.github.com/users/organisciak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/organisciak/subscriptions","organizations_url":"https://api.github.com/users/organisciak/orgs","repos_url":"https://api.github.com/users/organisciak/repos","events_url":"https://api.github.com/users/organisciak/events{/privacy}","received_events_url":"https://api.github.com/users/organisciak/received_events","type":"User","site_admin":false},"created_at":"2015-02-10T21:35:40Z","updated_at":"2015-02-10T21:35:40Z","body":"Cool. I started this process with the HTRC extracted feature counts. I processed the input, but haven't tried slurping into the DB yet.\r\n\r\nLooking through the code, it looks ready to toy with; is this correct? The one part I'm not clear about is line 242 of [tokenizer.py](BookwormDB/bookworm/tokenizer.py): why does nothing happen if the documentid has been seen? Don't we want to have documentid+feature?"}}