{"action":"created","issue":{"url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/1","labels_url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/1/comments","events_url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/1/events","html_url":"https://github.com/tudarmstadt-lt/structured-topics/issues/1","id":98341228,"number":1,"title":"Implement an initial prototype of the system","user":{"login":"alexanderpanchenko","id":1456830,"avatar_url":"https://avatars.githubusercontent.com/u/1456830?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderpanchenko","html_url":"https://github.com/alexanderpanchenko","followers_url":"https://api.github.com/users/alexanderpanchenko/followers","following_url":"https://api.github.com/users/alexanderpanchenko/following{/other_user}","gists_url":"https://api.github.com/users/alexanderpanchenko/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderpanchenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderpanchenko/subscriptions","organizations_url":"https://api.github.com/users/alexanderpanchenko/orgs","repos_url":"https://api.github.com/users/alexanderpanchenko/repos","events_url":"https://api.github.com/users/alexanderpanchenko/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderpanchenko/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-07-31T08:25:28Z","updated_at":"2015-08-28T10:28:56Z","closed_at":null,"body":"Motivation\r\n-----------\r\n\r\nWe need to develop a prototype of the system that builds structured topics (creates a model) and is able to label new texts according to these topics. This prototype is supposed to have all minimal functionality of the system (input/output) and implemented with the most straightforward set of algorithms. The goal is to make initial validation of the idea and then improve the prototype gradually. In this step we do no evaluation which will be done later as well (during the \"official\" 6 month period reserved for writing the thesis). \r\n\r\nImplementation\r\n--------------------\r\n\r\nThe prototype will build structured topics out of sense clusters i.e. it will cluster word senses which are provided as input. The system will rely on existing developments of the LT team in order to efficiently implement computation of a similarity graph of senses. \r\n\r\nThe overall pipeline of the prototype (to be implemented in Java/Scala or a mix of both):\r\n\r\n0. Download the data -- a Disambiguated Distributional Thesaurus (DDT) build from the JoBimText sense clusters:\r\n\r\n  - http://panchenko.me/data/joint/senses-n30-1600k.csv.gz\r\n  - http://panchenko.me/data/joint/senses-n200-380k.csv.gz \r\n\r\n1. Create a sense-feature matrix out of these files in the following format:\r\n\r\n  ```\r\n  sense-id<TAB>cluster-word<TAB>similarity-of-sense-and-cluster-word\r\n  ```\r\n\r\n  for instance\r\n\r\n  ```\r\n  python#np#2    java#np#4    0.75\r\n  ```\r\n\r\n  here 0.75 is 1/(r+1)^0.33, where r is the rank of cluster word from 1 to n\r\n\r\n  To be more precise, the system should generate the same output as this package: https://github.com/tudarmstadt-lt/noun-sense-induction\r\n\r\n  This includes two additional files: ```sense-id<TAB>freq``` and ```cluster-word<TAB>freq```.\r\n\r\n2.  For each sense retrieve top 200 most similar senses i.e. calculate a similarity graph of senses. You will need to adapt this project to do so efficiently using Apache Spark framework: https://github.com/tudarmstadt-lt/noun-sense-induction-scala. To understand how it works read \r\n\r\n  - http://jlm.ipipan.waw.pl/ojs/index.php/JLM/article/viewFile/60/44\r\n  - https://www.lt.informatik.tu-darmstadt.de/fileadmin/user_upload/Group_LangTech/student_theses/MAThesis_JohannesSimon_WSID.pdf \r\n\r\n  The framework outputs a file in the format:\r\n\r\n  ```\r\n  sense-id<TAB>sense-id<TAB>similarity\r\n  ``` \r\n  for each sense, it would generate 200 nearest neighbours. \r\n\r\n  For start with Spark just look at examples: http://spark.apache.org/\r\n\r\n3. Cluster the graph of sense similarities using Chinese Whispers or/and Markov Chain Clustering algorithm. Use this implementation: https://github.com/johannessimon/chinese-whispers. Alternatively you can use this implementation: http://maggie.lt.informatik.tu-darmstadt.de/jobimtext/documentation/sense-clustering/ \r\n\r\n  In your case, instead of words you will cluster word senses. \r\n  \r\n  http://wortschatz.uni-leipzig.de/~cbiemann/pub/2006/BiemannTextGraph06.pdf\r\n\r\n  ```\r\n  structured-topici-id<TAB>sense-id-1,sense-id-2,sense-id-3,...\r\n  ```\r\n \r\n4. Write a classification module that would use the structured topics, being clusters of senses, to annotate text documents. The module should \r\n  - load the structured topics \r\n\r\n  ```\r\n  structured-topici-id<TAB>cluster-word-1,cluster-word-2,cluster-word-3,...\r\n  ``` \r\n  - for an input document output a set of most relevant topics \r\n  - each output topic should have a confidence of the classification\r\n\r\n  To implement this module you should use ElasticSearch index. One topic would be one document, and then use an input document as search query. The retrieval system will return a list of documents (topics) according to their TF-IDF score. \r\n\r\n "},"comment":{"url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/comments/135729513","html_url":"https://github.com/tudarmstadt-lt/structured-topics/issues/1#issuecomment-135729513","issue_url":"https://api.github.com/repos/tudarmstadt-lt/structured-topics/issues/1","id":135729513,"user":{"login":"alexanderpanchenko","id":1456830,"avatar_url":"https://avatars.githubusercontent.com/u/1456830?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderpanchenko","html_url":"https://github.com/alexanderpanchenko","followers_url":"https://api.github.com/users/alexanderpanchenko/followers","following_url":"https://api.github.com/users/alexanderpanchenko/following{/other_user}","gists_url":"https://api.github.com/users/alexanderpanchenko/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderpanchenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderpanchenko/subscriptions","organizations_url":"https://api.github.com/users/alexanderpanchenko/orgs","repos_url":"https://api.github.com/users/alexanderpanchenko/repos","events_url":"https://api.github.com/users/alexanderpanchenko/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderpanchenko/received_events","type":"User","site_admin":false},"created_at":"2015-08-28T10:28:56Z","updated_at":"2015-08-28T10:28:56Z","body":"```\r\nRefrain#NP   #NP    1\r\n```\r\n\r\n```\r\nRefrain#NP#2   #NP    1\r\n```"}}