how to better process the huge history data in the kafka topic by using spark streaming
<p>I am experiencing an issue to start spark streaming on a really big kafka topic, there are around 150 million data in this topic already and the topic is growing super fast. </p>

<p>When I tried to start spark streaming and read data from the beginning of this topic by setting kafka parameter ("auto.offset.reset" -> "smallest"), it always try to finish all 150 million data processing in the first batch and return a "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. There isn't a lot calculation in this spark stream app though.</p>

<p>Can I have a way to process the history data in this topic in first several batches but not all in first batch? </p>

<p>Bunch of thanks in advance!
James</p>