{"action":"created","issue":{"url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/16","labels_url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/16/comments","events_url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/16/events","html_url":"https://github.com/RevolutionAnalytics/rhdfs/issues/16","id":99550303,"number":16,"title":"HADOOP_CMD getting lost...","user":{"login":"bhermont","id":7468247,"avatar_url":"https://avatars.githubusercontent.com/u/7468247?v=3","gravatar_id":"","url":"https://api.github.com/users/bhermont","html_url":"https://github.com/bhermont","followers_url":"https://api.github.com/users/bhermont/followers","following_url":"https://api.github.com/users/bhermont/following{/other_user}","gists_url":"https://api.github.com/users/bhermont/gists{/gist_id}","starred_url":"https://api.github.com/users/bhermont/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bhermont/subscriptions","organizations_url":"https://api.github.com/users/bhermont/orgs","repos_url":"https://api.github.com/users/bhermont/repos","events_url":"https://api.github.com/users/bhermont/events{/privacy}","received_events_url":"https://api.github.com/users/bhermont/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-07T00:19:24Z","updated_at":"2015-08-07T01:00:40Z","closed_at":"2015-08-07T00:58:40Z","body":"Hi Antonio,\r\n\r\nI've faced a scenario where I call a mapreduce (rmr) from a Shell script inside a Mapper (This is how Oozie launches a Shell action)\r\n\r\nHere is the flow:\r\nOozie Launcher Job -> Lancher Map only task where shell script (Rscript myscript.r) executes -> StreamJob -> Mapper / Reducers\r\n\r\nmyscript.r\r\n-----------------------------------------------------\r\nSys.setenv(JAVA_HOME=\"/usr/jdk64/jdk1.7.0_67\")\r\nSys.setenv(HADOOP_CMD=\"/usr/bin/hadoop\")\r\nSys.setenv(HADOOP_HOME=\"/usr/hdp/2.2.6.0-2800/hadoop\")\r\nSys.setenv(HADOOP_STREAMING=\"/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-streaming.jar\")\r\nlibrary(\"rhdfs\")\r\nlibrary(\"rmr2\")\r\nhdfs.init()\r\nlibrary(Matrix)\r\n<mapreduce call here>\r\n\r\nLogs \r\n------------------------------------------------------\r\nLauncher Job (Job that launches  Map only task)\r\n\tOn this map, the following shell script is executed as a system call (Rscript myscript.r) - Here HADOOP_CMD is set correctly, but an error on the mr function is logged (Probably due the Streaming Mapper error when a hdfs function is called from inside the mr function). Launcher (Mapper) log:\r\n\r\n\tLoading required package: methods\r\n\tLoading required package: rJava\r\n\r\n\tHADOOP_CMD=/usr/bin/hadoop\r\n\r\n\tBe sure to run hdfs.init()\r\n\tPlease review your hadoop settings. See help(hadoop.settings)\r\n\tError in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : \r\n\t  hadoop streaming failed with error code 15\r\n\tCalls: getTags -> mapreduce -> mr\r\n\t\r\n\t\r\nThe rmr streaming starts a StreamJob which can have Mappers and Reducers. StreamJob (Mapper) log:\r\n\t\t\r\n\t\tLog Type: stderr\r\n\t\tLog Upload Time: Thu Aug 06 19:24:41 -0400 2015\r\n\r\n\t\tLog Length: 2722\r\n\r\n\t\tLoading objects:\r\n\t\tLoading objects:\r\n\t\t  backend.parameters\r\n\t\t  combine\r\n\t\tPlease review your hadoop settings. See help(hadoop.settings)\r\n\t\t  combine.file\r\n\t\t  combine.line\r\n\t\t  debug\r\n\t\t  default.input.format\r\n\t\t  default.output.format\r\n\t\t  in.folder\r\n\t\t  in.memory.combine\r\n\t\t  input.format\r\n\t\t  libs\r\n\t\t  map\r\n\t\t  map.file\r\n\t\t  map.line\r\n\t\t  out.folder\r\n\t\t  output.format\r\n\t\t  pkg.opts\r\n\t\t  postamble\r\n\t\t  preamble\r\n\t\t  profile.nodes\r\n\t\t  reduce\r\n\t\t  reduce.file\r\n\t\t  reduce.line\r\n\t\t  rmr.global.env\r\n\t\t  rmr.local.env\r\n\t\t  save.env\r\n\t\t  tempfile\r\n\t\t  vectorized.reduce\r\n\t\t  verbose\r\n\t\t  work.dir\r\n\t\tLoading required package: methods\r\n\t\tLoading required package: rJava\r\n\t\tLoading required package: rhdfs\r\n\t\tError : .onLoad failed in loadNamespace() for 'rhdfs', details:\r\n\t\t  call: fun(libname, pkgname)\r\n\t\t  error: Environment variable HADOOP_CMD must be set before loading package rhdfs\r\n\t\tWarning in FUN(X[[i]], ...) : can't load rhdfs\r\n\t\tLoading required package: rmr2\r\n\t\tLoading required package: Matrix\r\n\r\nHowever, calling the myscript.r from command line works fine.\r\n\r\nHere is my question:\r\nShould rmr propagate the environment envs in this case, or this should be responsibility of the enviroment to provide the value for the HADOOP_CMD variable?"},"comment":{"url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/comments/128552088","html_url":"https://github.com/RevolutionAnalytics/rhdfs/issues/16#issuecomment-128552088","issue_url":"https://api.github.com/repos/RevolutionAnalytics/rhdfs/issues/16","id":128552088,"user":{"login":"bhermont","id":7468247,"avatar_url":"https://avatars.githubusercontent.com/u/7468247?v=3","gravatar_id":"","url":"https://api.github.com/users/bhermont","html_url":"https://github.com/bhermont","followers_url":"https://api.github.com/users/bhermont/followers","following_url":"https://api.github.com/users/bhermont/following{/other_user}","gists_url":"https://api.github.com/users/bhermont/gists{/gist_id}","starred_url":"https://api.github.com/users/bhermont/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bhermont/subscriptions","organizations_url":"https://api.github.com/users/bhermont/orgs","repos_url":"https://api.github.com/users/bhermont/repos","events_url":"https://api.github.com/users/bhermont/events{/privacy}","received_events_url":"https://api.github.com/users/bhermont/received_events","type":"User","site_admin":false},"created_at":"2015-08-07T01:00:40Z","updated_at":"2015-08-07T01:00:40Z","body":"Done!"}}