

Join them; it only takes a minute:

Sign up
Join the Stack Overflow community to:
  1. Ask programming questions
  2. Answer and help your peers
  3. Get recognized for your expertise

I am working on the parallelization an algorithm, which roughly does the following:

  1. Read several text documents with a total of 10k words.
  2. Create an objects for every word in the text corpus.
  3. Create a pair between all word-objects (yes, O(n)). And return the most frequent pairs.

I would like to parallelize the 3. step by creating the pairs between the first 1000 word-objects the rest on the fist machine, the second 1000 word-objects on the next machine, etc.

My question is how to pass the objects created in the 2. step to the Mapper? As far as I am aware I would require input files for this and hence would need to serialize the objects (though haven't worked with this before). Is there a direct way to pass the objects to the Mapper?

Thanks in advance for the help


UPDATE Thank you for reading my question before. Serialization seems to be the best way to solve this (see Furthermore, I have found this tutorial useful to read data from serialized objects into hadoop:

share|improve this question
Not by any means an expert in this area but I don't think there is. Serializing objects is not very hard. – Nick Feb 9 '11 at 3:35

How about parallelize all steps? Use your #1 text documents as input to your Mapper. Create the object for every word in the Mapper. In the Mapper your key-value pair will be the word-object pair (or object-word depending on what you are doing). The Reducer can then count the unique pairs.

Hadoop will take care of bringing all the same keys together into the same Reducer.

share|improve this answer
thanks for the suggestion; the algorithm is pretty massive and I intended to start with the most resource consuming part first, to see if I can improve the performance. – evgeni Feb 9 '11 at 4:35

Use twitter protobufs ( elephant-bird ) . Convert each word into a protobuf object and process it however you want. Also protobufs are much faster and light compared to default java serialization. Refer Kevin Weil's presentation on this.

share|improve this answer

Your Answer


