{"action":"created","issue":{"url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/1","labels_url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/1/comments","events_url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/1/events","html_url":"https://github.com/TargetHolding/pyspark-cassandra/issues/1","id":62411171,"number":1,"title":"Exception when calling select()","user":{"login":"lrhazi","id":120354,"avatar_url":"https://avatars.githubusercontent.com/u/120354?v=3","gravatar_id":"","url":"https://api.github.com/users/lrhazi","html_url":"https://github.com/lrhazi","followers_url":"https://api.github.com/users/lrhazi/followers","following_url":"https://api.github.com/users/lrhazi/following{/other_user}","gists_url":"https://api.github.com/users/lrhazi/gists{/gist_id}","starred_url":"https://api.github.com/users/lrhazi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lrhazi/subscriptions","organizations_url":"https://api.github.com/users/lrhazi/orgs","repos_url":"https://api.github.com/users/lrhazi/repos","events_url":"https://api.github.com/users/lrhazi/events{/privacy}","received_events_url":"https://api.github.com/users/lrhazi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-17T14:21:30Z","updated_at":"2015-03-17T20:08:10Z","closed_at":null,"body":"```\r\nimport pyspark_cassandra\r\nrdd=sc.cassandraTable('eos','articles')\r\nrdd=rdd.select('title','content')\r\n#.where('sourcecode=?','fbis')\r\n#rdd.take(1)\r\n````\r\n\r\nResults in:\r\n\r\n```\r\nException                                 Traceback (most recent call last)\r\n<ipython-input-5-8b9698b52204> in <module>()\r\n      1 import pyspark_cassandra\r\n      2 rdd=sc.cassandraTable('eos','articles')\r\n----> 3 rdd=rdd.select('title','content')\r\n      4 #.where('sourcecode=?','fbis')\r\n      5 #rdd.take(1)\r\n\r\n/tmp/spark-3090029c-06fc-41a8-a7f2-a86058916a55/spark-bd07d023-be3a-4035-89d9-8bb77761fd6f/pyspark_cassandra-0.1.0-py2.7.egg/pyspark_cassandra/rdd.py in select(self, *columns)\r\n     73 \r\n     74                 columns = as_java_array(self.ctx._gateway, \"String\", (str(c) for c in columns))\r\n---> 75                 new = copy(self)\r\n     76                 new._jrdd = new._jrdd.select(columns)\r\n     77                 return new\r\n\r\n/usr/lib/python2.7/copy.pyc in copy(x)\r\n     86         reductor = getattr(x, \"__reduce_ex__\", None)\r\n     87         if reductor:\r\n---> 88             rv = reductor(2)\r\n     89         else:\r\n     90             reductor = getattr(x, \"__reduce__\", None)\r\n\r\n/spark/python/pyspark/rdd.py in __getnewargs__(self)\r\n    145         # This method is called when attempting to pickle an RDD, which is always an error:\r\n    146         raise Exception(\r\n--> 147             \"It appears that you are attempting to broadcast an RDD or reference an RDD from an \"\r\n    148             \"action or transformation. RDD transformations and actions can only be invoked by the \"\r\n    149             \"driver, not inside of other transformations; for example, \"\r\n\r\nException: It appears that you are attempting to broadcast an RDD or reference an RDD from an action or transformation. RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(lambda x: rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.\r\n```"},"comment":{"url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/comments/82576264","html_url":"https://github.com/TargetHolding/pyspark-cassandra/issues/1#issuecomment-82576264","issue_url":"https://api.github.com/repos/TargetHolding/pyspark-cassandra/issues/1","id":82576264,"user":{"login":"frensjan","id":3985389,"avatar_url":"https://avatars.githubusercontent.com/u/3985389?v=3","gravatar_id":"","url":"https://api.github.com/users/frensjan","html_url":"https://github.com/frensjan","followers_url":"https://api.github.com/users/frensjan/followers","following_url":"https://api.github.com/users/frensjan/following{/other_user}","gists_url":"https://api.github.com/users/frensjan/gists{/gist_id}","starred_url":"https://api.github.com/users/frensjan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frensjan/subscriptions","organizations_url":"https://api.github.com/users/frensjan/orgs","repos_url":"https://api.github.com/users/frensjan/repos","events_url":"https://api.github.com/users/frensjan/events{/privacy}","received_events_url":"https://api.github.com/users/frensjan/received_events","type":"User","site_admin":false},"created_at":"2015-03-17T20:08:09Z","updated_at":"2015-03-17T20:08:09Z","body":"Sorry, didn't mean to close the issue just yet.\r\n\r\nIt seems that in Spark 1.2.1 a check has been added to protect users from shooting themselves in the foot by e.g. broadcasting an RDD or using a reference to an RDD. This 'check' is in my view a bit course because it is achieved by raising an exception in `__getnewargs__(...)`. This method is also invoked when using `copy.copy(...)`, and I used that to keep `CassandraRDD` immutable.\r\n\r\nI tried working around the issue by implementing the `__copy__(...)` method which ensures that `__getnewargs__(...)` isn't invoked on a (shallow) copy of `CassandraRDD`.\r\n\r\nI've tested this on Spark version 1.2.0 and it _still_ works. The question is, does this fix the issue on version 1.2.1? I haven't got the time to realy test this now. @lrhazi, let me know if you do :) Setting up unit / integration tests is on the to do list ..."}}