{"action":"created","issue":{"url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/329","labels_url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/329/labels{/name}","comments_url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/329/comments","events_url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/329/events","html_url":"https://github.com/dbpedia-spotlight/dbpedia-spotlight/issues/329","id":50182912,"number":329,"title":"Running pignlproc under Amazon EMR","user":{"login":"michellemay","id":5503941,"avatar_url":"https://avatars.githubusercontent.com/u/5503941?v=3","gravatar_id":"","url":"https://api.github.com/users/michellemay","html_url":"https://github.com/michellemay","followers_url":"https://api.github.com/users/michellemay/followers","following_url":"https://api.github.com/users/michellemay/following{/other_user}","gists_url":"https://api.github.com/users/michellemay/gists{/gist_id}","starred_url":"https://api.github.com/users/michellemay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michellemay/subscriptions","organizations_url":"https://api.github.com/users/michellemay/orgs","repos_url":"https://api.github.com/users/michellemay/repos","events_url":"https://api.github.com/users/michellemay/events{/privacy}","received_events_url":"https://api.github.com/users/michellemay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":48,"created_at":"2014-11-26T14:11:49Z","updated_at":"2015-01-30T12:42:27Z","closed_at":null,"body":"Hi, is anybody tried to launch model trainer under amazon elastic map reduce service?\r\nAny automated script for that?\r\n\r\nRight now, I see many hurdles:\r\n1) Amazon Pig versions are 0.9.2, 0.11.1 and 12.0 and Hadoop 2.4, 2.2, 1.0.3, 0.20.205\r\n2) S3 paths\r\n3) Automation using ruby, aws CLI?\r\n4) Working on compressed bz2 wiki dumps\r\n\r\n1) Pig/Hadoop versions:\r\nEach of theses versions of Pig might causes compilation errors. We need to adapt pom.xml in order to target specific hadoop (hadoop-common and hadoop-mapreduce-client-core for 2.2 and later)\r\n\r\n2) S3 output paths.. I got failures running names_and_entities.pig step.  It tries to load temporary surface forms from what seems to an invalid folder in S3.  This is probably due to a file-open in the ./ current folder which does not exist in S3.\r\n\r\n3) Automation.  \r\nSince the training process requires some steps to be run after last pig job, we could use an EC2 with necessarily tools (JDK, Maven, Ruby, s3cmd).\r\n\r\n4) It might not be up-to-date, but I saw somewhere on theses pages that pignlproc should have uncompressed wiki dumps.  What would be needed in order to work directly with compressed dumps (Amazon should do most of the job here according to their documentation)\r\n\r\nAll that said, I'm more than willing to spend some time and amazon money on this granted some support of the project owner.\r\n\r\nMichel"},"comment":{"url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/comments/72195365","html_url":"https://github.com/dbpedia-spotlight/dbpedia-spotlight/issues/329#issuecomment-72195365","issue_url":"https://api.github.com/repos/dbpedia-spotlight/dbpedia-spotlight/issues/329","id":72195365,"user":{"login":"tgalery","id":1846448,"avatar_url":"https://avatars.githubusercontent.com/u/1846448?v=3","gravatar_id":"","url":"https://api.github.com/users/tgalery","html_url":"https://github.com/tgalery","followers_url":"https://api.github.com/users/tgalery/followers","following_url":"https://api.github.com/users/tgalery/following{/other_user}","gists_url":"https://api.github.com/users/tgalery/gists{/gist_id}","starred_url":"https://api.github.com/users/tgalery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgalery/subscriptions","organizations_url":"https://api.github.com/users/tgalery/orgs","repos_url":"https://api.github.com/users/tgalery/repos","events_url":"https://api.github.com/users/tgalery/events{/privacy}","received_events_url":"https://api.github.com/users/tgalery/received_events","type":"User","site_admin":false},"created_at":"2015-01-30T12:42:27Z","updated_at":"2015-01-30T12:42:27Z","body":"The pig script goes on and prints this other error:\r\n```\r\n2015-01-30 11:08:45,211 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1422542657398_0004' doesn't exist in RM.\r\n        at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:284)\r\n        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145)\r\n        at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:415)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)```"}}