{"action":"created","issue":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/388","labels_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/388/labels{/name}","comments_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/388/comments","events_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/388/events","html_url":"https://github.com/HazyResearch/deepdive/issues/388","id":106116548,"number":388,"title":"Easiest Way Custom Annotations","user":{"login":"vsoch","id":814322,"avatar_url":"https://avatars.githubusercontent.com/u/814322?v=3","gravatar_id":"","url":"https://api.github.com/users/vsoch","html_url":"https://github.com/vsoch","followers_url":"https://api.github.com/users/vsoch/followers","following_url":"https://api.github.com/users/vsoch/following{/other_user}","gists_url":"https://api.github.com/users/vsoch/gists{/gist_id}","starred_url":"https://api.github.com/users/vsoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsoch/subscriptions","organizations_url":"https://api.github.com/users/vsoch/orgs","repos_url":"https://api.github.com/users/vsoch/repos","events_url":"https://api.github.com/users/vsoch/events{/privacy}","received_events_url":"https://api.github.com/users/vsoch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"netj","id":53634,"avatar_url":"https://avatars.githubusercontent.com/u/53634?v=3","gravatar_id":"","url":"https://api.github.com/users/netj","html_url":"https://github.com/netj","followers_url":"https://api.github.com/users/netj/followers","following_url":"https://api.github.com/users/netj/following{/other_user}","gists_url":"https://api.github.com/users/netj/gists{/gist_id}","starred_url":"https://api.github.com/users/netj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/netj/subscriptions","organizations_url":"https://api.github.com/users/netj/orgs","repos_url":"https://api.github.com/users/netj/repos","events_url":"https://api.github.com/users/netj/events{/privacy}","received_events_url":"https://api.github.com/users/netj/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-09-12T00:33:46Z","updated_at":"2015-09-12T03:31:39Z","closed_at":null,"body":"I want to use [RegexNER](http://nlp.stanford.edu/software/regexner/) to define a list of custom words (brain regions and cognitive concepts) that will be tagged as either REGION or CONCEPT.\r\n\r\nAs an example, the top of my `cognitiveatlas-annotations.txt` file looks like this:\r\n\r\n      abductive reasoning\tCONCEPT\r\n      absolute threshold\tCONCEPT\r\n      abstract analogy\tCONCEPT\r\n\r\nThe example on the site to call the extractor is like this:\r\n\r\n      java -mx1g -cp '*' edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators 'tokenize,ssplit,pos,lemma,ner,regexner' -file JuliaGillard.txt -regexner.mapping jg-regexner.txt\r\n\r\nWhere `jg-regexner.txt` would be replaced with `cognitiveatlas-annotations.txt`. Looking at how the extractor is run from deep dive, it's this command in `deepdive.conf`:\r\n\r\n           udf             : \"examples/nlp_extractor/run.sh -k article_id -v text -l 100 -t 4\"\r\n \r\nand the run script seems to export some environmental variables for sbt:\r\n\r\n      #! /usr/bin/env bash\r\n      export SBT_OPTS=\"-Xmx1g\"\r\n      $(dirname $0)/target/start $@\r\n\r\nand the start script sets the project directory and main class:\r\n\r\n      #!/bin/bash\r\n      PROJECT_DIR=$(cd \"${BASH_SOURCE[0]%/*}\" && pwd -P)/..\r\n      MAINCLASS=org.deepdive.udf.nlp.Main\r\n\r\nMy question is: what is the easiest way to integrate my annotations into the pipeline? Is it just as easy as adding the `-regexner.mapping` argument to the SBT_OPTS variable? It would be most ideal to have a solution that stored with my project (external to the deepdive installation). I noticed [this PR](https://github.com/HazyResearch/deepdive/pull/250) that seems to be about adding custom annotations, but the comment seems to be about running things in parallel so I'm not sure if it's relevant. My first thought was just to integrate discovering the concepts in an extractor, but it seems highly inefficient to loop over every word, and every concept or brain region, and check for matches (is that what people sometimes do?). \r\n\r\nI think what I want to do is pretty simple - it's really just specifying regular expressions to get specific NER tags, otherwise the extractor isn't so useful to help find the brain regions and concepts. If it were run in parallel this might not be such a nutty idea, but I haven't even found the documentation for that yet. Any pointers to relevant documentation or tips would be greatly appreciated! The database I'm working with has to be recompiled for nlp_extractor to work and it isn't happening until the 15th, so there is no time urgency. Thanks!"},"comment":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/comments/139706167","html_url":"https://github.com/HazyResearch/deepdive/issues/388#issuecomment-139706167","issue_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/388","id":139706167,"user":{"login":"vsoch","id":814322,"avatar_url":"https://avatars.githubusercontent.com/u/814322?v=3","gravatar_id":"","url":"https://api.github.com/users/vsoch","html_url":"https://github.com/vsoch","followers_url":"https://api.github.com/users/vsoch/followers","following_url":"https://api.github.com/users/vsoch/following{/other_user}","gists_url":"https://api.github.com/users/vsoch/gists{/gist_id}","starred_url":"https://api.github.com/users/vsoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsoch/subscriptions","organizations_url":"https://api.github.com/users/vsoch/orgs","repos_url":"https://api.github.com/users/vsoch/repos","events_url":"https://api.github.com/users/vsoch/events{/privacy}","received_events_url":"https://api.github.com/users/vsoch/received_events","type":"User","site_admin":false},"created_at":"2015-09-12T03:31:39Z","updated_at":"2015-09-12T03:31:39Z","body":"Is there a branch I could look at for inspiration? Even just a push in the right direction I think I could get something working for our analysis."}}