Iterating user-defined class objects inside a pyspark RDD
<p>I am reading data from csv and converting that data into a python class object. But when i try to iterate over that rdd with user-defined class objects, I get errors like, </p>

<pre><code>_pickle.PicklingError: Can't pickle &lt;class '__main__.User'&gt;: attribute lookup User on __main__ failed
</code></pre>

<p>I'm adding some part of the code here,</p>

<pre><code>class User:
    def __init__(self, line):
        self.user_id = line[0]
        self.location = line[1]
        self.age = line[2]

def create_user(line):
    user = User(line)
    return user

def print_user(line):
    user = line
    print(user.user_id)

conf = (SparkConf().setMaster("local").setAppName("exercise_set_2").set("spark.executor.memory", "1g"))
sc = SparkContext(conf = conf)
users = sc.textFile("BX-Users.csv").map(lambda line: line.split(";"))  
users_objs = users.map(lambda entry: create_user(entry))
users_objs.map(lambda entry: print_user(entry))
</code></pre>

<p>For the above code, I get results like,</p>

<pre><code>PythonRDD[93] at RDD at PythonRDD.scala:43
</code></pre>

<p>CSV data source URL(Needs a zip extraction): <a href="http://www2.informatik.uni-freiburg.de/~cziegler/BX/BX-CSV-Dump.zip" rel="nofollow">HERE</a></p>

<p><strong>UPDATE:</strong>
changing the code to include collect will result in error again, I still have to try with Pickle. I never tried that one before, If you anyone have a sample, I can do it easily.</p>

<pre><code>users_objs = users.map(lambda entry: create_user(entry)).collect()
</code></pre>