{"action":"created","issue":{"url":"https://api.github.com/repos/NFLabs/zeppelin/issues/391","labels_url":"https://api.github.com/repos/NFLabs/zeppelin/issues/391/labels{/name}","comments_url":"https://api.github.com/repos/NFLabs/zeppelin/issues/391/comments","events_url":"https://api.github.com/repos/NFLabs/zeppelin/issues/391/events","html_url":"https://github.com/NFLabs/zeppelin/issues/391","id":63029729,"number":391,"title":"PySpark Feature not yet working","user":{"login":"dennisproppe","id":1608494,"avatar_url":"https://avatars.githubusercontent.com/u/1608494?v=3","gravatar_id":"","url":"https://api.github.com/users/dennisproppe","html_url":"https://github.com/dennisproppe","followers_url":"https://api.github.com/users/dennisproppe/followers","following_url":"https://api.github.com/users/dennisproppe/following{/other_user}","gists_url":"https://api.github.com/users/dennisproppe/gists{/gist_id}","starred_url":"https://api.github.com/users/dennisproppe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dennisproppe/subscriptions","organizations_url":"https://api.github.com/users/dennisproppe/orgs","repos_url":"https://api.github.com/users/dennisproppe/repos","events_url":"https://api.github.com/users/dennisproppe/events{/privacy}","received_events_url":"https://api.github.com/users/dennisproppe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2015-03-19T16:26:25Z","updated_at":"2015-11-14T00:12:35Z","closed_at":null,"body":"First: Thanks for merging the pyspark interpreter into master! This is really great.\r\n\r\nUnfortunately, I can't use it right now, because there seems to be an issue with the Implementation. When using a very simple example, \r\n\r\n```\r\n%pyspark\r\nsc.parallelize([1,2,3])\r\n```\r\n\r\nit seems to work: ParallelCollectionRDD[11] at parallelize at PythonRDD.scala:376\r\n\r\nBut when I want to actually do something with it:\r\n\r\n```\r\n%pyspark\r\nsc.parallelize([1,2,3]).count() \r\n```\r\n\r\nit crashes: \r\n\r\n(<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred while calling o121.collect.\\n', JavaObject id=o122), <traceback object at 0x330ab48>)\r\n\r\nAm I doing something wrgon here or might this be a bug?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NFLabs/zeppelin/issues/comments/156595626","html_url":"https://github.com/NFLabs/zeppelin/issues/391#issuecomment-156595626","issue_url":"https://api.github.com/repos/NFLabs/zeppelin/issues/391","id":156595626,"user":{"login":"Leemoonsoo","id":1540981,"avatar_url":"https://avatars.githubusercontent.com/u/1540981?v=3","gravatar_id":"","url":"https://api.github.com/users/Leemoonsoo","html_url":"https://github.com/Leemoonsoo","followers_url":"https://api.github.com/users/Leemoonsoo/followers","following_url":"https://api.github.com/users/Leemoonsoo/following{/other_user}","gists_url":"https://api.github.com/users/Leemoonsoo/gists{/gist_id}","starred_url":"https://api.github.com/users/Leemoonsoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Leemoonsoo/subscriptions","organizations_url":"https://api.github.com/users/Leemoonsoo/orgs","repos_url":"https://api.github.com/users/Leemoonsoo/repos","events_url":"https://api.github.com/users/Leemoonsoo/events{/privacy}","received_events_url":"https://api.github.com/users/Leemoonsoo/received_events","type":"User","site_admin":false},"created_at":"2015-11-14T00:12:35Z","updated_at":"2015-11-14T00:12:35Z","body":"Are you running on Zeppelin with JDK7, right? If your system have both JDK7 and JDK8, to make sure I recommend export JAVA_HOME to JDK7, in conf/zeppelin-env.sh and retry.\r\n\r\n\r\nIf exporting JAVA_HOME does not help, we really need digging up the cause.\r\n\r\nZeppelin runs separate process called 'interpreter' to connect backend system and communicate with the process using thrift protocol. \r\n\r\nFor Spark, ZeppelinServer invokes SparkInterpreter process and establish thrift connection to it.\r\nYour error shows there're some problem on establishing that thrift connection.\r\nThere can be few possible reasons i know.\r\n\r\n##### A) creating SparkContext fails\r\n\r\n   ZeppelinInterpreter process tries to create SparkContext, but when it fails to create with some reason, it sometimes terminates SparkContext process. In this case you'll see some error message in log files. Files are under /logs directory\r\n\r\n\r\n##### B) Network configuration problem\r\n\r\n   ZeppelinServer will trying to establish thrift connection using 'localhost:[port]' address. And the SparkInterpreter process is expected to create thrift server and listen on 'tcp4 *.[port]'. Depending on system configuration, i saw very few case that SparkInterpreter process is not listing on all available network interface and ZeppelinServer can not find it using 'localhost' address. To check,\r\n\r\n   1) make sure all ZeppelinServer and Interpreter processes are terminated.\r\n\r\n    ```\r\n    ps -ef | grep ZeppelinServer | grep -v grep | awk '{print $2}' | xargs kill\r\n    ps -ef | grep RemoteInterpreterServer | grep -v grep | awk '{print $2}' | xargs kill\r\n    ```\r\n\r\n   2) Start zeppelin and run any spark command\r\n\r\n    ```\r\n    bin/zeppelin-daemon.sh start\r\n    ```\r\n\r\n    ```\r\n    %spark\r\n    sc.version\r\n    ```\r\n\r\n   3) Check SparkInterpreterServer process and binded address\r\n\r\n    ```\r\n    ps -ef | grep RemoteInterpreterServer\r\n    ```\r\n   \r\n    will print a process, for example\r\n\r\n     ```\r\n     ....\r\n     --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer \\\r\n    /zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar 63549\r\n    ```\r\n\r\n    This is SparkInterpreter process and you can check if it is listening on correct address, against the port \r\n\r\n    ```\r\n    Lees-MacBook:zeppelin moon$ netstat -na | grep 63549\r\n    tcp46      0      0  *.63549 \r\n    ```"}}