{"action":"closed","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/1015","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/1015/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/1015/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/1015/events","html_url":"https://github.com/Yelp/mrjob/issues/1015","id":79038069,"number":1015,"title":"FYI: MRJob job setup fails on MapR 4.x due to bug in their \"hadoop fs -mkdir -p\"","user":{"login":"dpichard","id":11614371,"avatar_url":"https://avatars.githubusercontent.com/u/11614371?v=3","gravatar_id":"","url":"https://api.github.com/users/dpichard","html_url":"https://github.com/dpichard","followers_url":"https://api.github.com/users/dpichard/followers","following_url":"https://api.github.com/users/dpichard/following{/other_user}","gists_url":"https://api.github.com/users/dpichard/gists{/gist_id}","starred_url":"https://api.github.com/users/dpichard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpichard/subscriptions","organizations_url":"https://api.github.com/users/dpichard/orgs","repos_url":"https://api.github.com/users/dpichard/repos","events_url":"https://api.github.com/users/dpichard/events{/privacy}","received_events_url":"https://api.github.com/users/dpichard/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-05-21T15:11:37Z","updated_at":"2015-10-16T20:16:14Z","closed_at":"2015-10-16T20:16:14Z","body":"Just wanted this noted here, since it will bite other MRJob users who are running MapR 4.x.  When MRJob (0.4.3/0.4.4) is making the temporary scratch directory in HDFS, it \"fails\" because MapR's `hadoop fs -mkdir -p` command feigns a failure if the path ends with a trailing `/`, though it actually ends up making the directory anyway.  I have an open bug with MapR about this issue, will happily update once I hear anything about their fix timeline -- for now I'm going to patch the MRJob package we use in our environment to work around the \"failure\".\r\n\r\nExample output:\r\n```\r\nWRAPPER: # store $PWD\r\nWRAPPER: __mrjob_PWD=$PWD\r\nWRAPPER: \r\nWRAPPER: # obtain exclusive file lock\r\nWRAPPER: exec 9>/tmp/wrapper.lock.aggregate_feed.dpichard.20150521.145103.481853\r\nWRAPPER: python -c 'import fcntl; fcntl.flock(9, fcntl.LOCK_EX)'\r\nWRAPPER: \r\nWRAPPER: # setup commands\r\nWRAPPER: {\r\nWRAPPER:   export PYTHONPATH=$__mrjob_PWD/mrjob.tar.gz:$PYTHONPATH\r\nWRAPPER:   export PYTHONPATH=$__mrjob_PWD/cdslite-libs-mapr.tgz:$PYTHONPATH\r\nWRAPPER: } 0</dev/null 1>&2\r\nWRAPPER: \r\nWRAPPER: # release exclusive file lock\r\nWRAPPER: exec 9>&-\r\nWRAPPER: \r\nWRAPPER: # run task from the original working directory\r\nWRAPPER: cd $__mrjob_PWD\r\nWRAPPER: \"$@\"\r\nMaking directory maprfs:///TIP/tmp/mrjob-scratch/aggregate_feed.dpichard.20150521.145103.481853/files/ on HDFS\r\n> /opt/mapr/hadoop/hadoop-2.5.1/bin/hadoop version\r\nUsing Hadoop version 2.5.1\r\n> /opt/mapr/hadoop/hadoop-2.5.1/bin/hadoop fs -mkdir -p maprfs:///TIP/tmp/mrjob-scratch/aggregate_feed.dpichard.20150521.145103.481853/files/\r\nSTDOUT: 2015-05-21 10:51:07,2633 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:1306 Thread: 12680 mkdirs failed for /TIP/tmp/mrjob-scratch/aggregate_feed.dpichard.20150521.145103., error 22\r\nSTDERR: mkdir: Error: Invalid argument(22), file: , user name: dpichard, ID: 246732\r\nTraceback (most recent call last):\r\n  File \"./aggregate_feed.py\", line 160, in <module>\r\n    CDS_Aggregate_Feed.run()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/job.py\", line 461, in run\r\n    mr_job.execute()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/job.py\", line 479, in execute\r\n    super(MRJob, self).execute()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/launch.py\", line 151, in execute\r\n    self.run_job()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/launch.py\", line 214, in run_job\r\n    runner.run()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/runner.py\", line 464, in run\r\n    self._run()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/hadoop.py\", line 236, in _run\r\n    self._upload_local_files_to_hdfs()\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/hadoop.py\", line 263, in _upload_local_files_to_hdfs\r\n    self._mkdir_on_hdfs(self._upload_mgr.prefix)\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/hadoop.py\", line 283, in _mkdir_on_hdfs\r\n    self.invoke_hadoop(['fs', '-mkdir', '-p', path])\r\n  File \"/users/dpichard/env-mrjob-release/lib/python2.6/site-packages/mrjob/fs/hadoop.py\", line 109, in invoke_hadoop\r\n    raise CalledProcessError(proc.returncode, args)\r\nsubprocess.CalledProcessError: Command '['/opt/mapr/hadoop/hadoop-2.5.1/bin/hadoop', 'fs', '-mkdir', '-p', 'maprfs:///TIP/tmp/mrjob-scratch/aggregate_feed.dpichard.20150521.145103.481853/files/']' returned non-zero exit status 1\r\n```\r\n\r\nBasic CLI evidence:\r\n```\r\n% /opt/mapr/hadoop/hadoop-2.5.1/bin/hadoop fs -mkdir -p maprfs:///TIP/tmp/mrjob-scratch/test-2\r\n% /opt/mapr/hadoop/hadoop-2.5.1/bin/hadoop fs -mkdir -p maprfs:///TIP/tmp/mrjob-scratch/test-3/\r\n2015-05-21 10:57:00,6418 ERROR JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:1306 Thread: 15525 mkdirs failed for /TIP/tmp/mrjob-scratch/test-3/, error 22\r\nmkdir: Error: Invalid argument(22), file: , user name: dpichard, ID: 246732\r\n% hadoop fs -ls maprfs:///TIP/tmp/mrjob-scratch/       \r\nFound 3 items\r\ndrwxr-xr-x   - dpichard tip-users          1 2015-05-21 10:51 maprfs:///TIP/tmp/mrjob-scratch/aggregate_feed.dpichard.20150521.145103.481853\r\ndrwxr-xr-x   - dpichard tip-users          0 2015-05-21 10:56 maprfs:///TIP/tmp/mrjob-scratch/test-2\r\ndrwxr-xr-x   - dpichard tip-users          0 2015-05-21 10:57 maprfs:///TIP/tmp/mrjob-scratch/test-3\r\n```"}}