java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream
<p>I getting NoClassDefFoundErrorerror while using Spark streaming API. Here is my Streaming code.</p>

<p>I know this is a problem with some mising jars and dependencies, but i couldnt figure out exactly what that is.</p>

<p>I am using kafka 0.9.0, spark 1.6.1 - Are these dependecies fine or do i need to change them? I have attached pom.xml below.</p>

<p>Here is the streaming API i am using.</p>

<p>JavaPairInputDStream directKafkaStream = KafkaUtils.createDirectStream(jsc, String.class,
            byte[].class, StringDecoder.class, DefaultDecoder.class, kafkaParams, topicSet);</p>

<p>here is my code piece. I am receiving error at while(itr.next())</p>

<pre><code>directKafkaStream.foreachRDD(rdd -&gt; {

    rdd.foreachPartition(itr -&gt; {


        try {

            while (itr.hasNext()) {
</code></pre>

<p>java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream</p>

<p>Here is my POM.xml</p>

<pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;

    &lt;parent&gt;
        &lt;groupId&gt;com.abcd.rep.xyz&lt;/groupId&gt;
        &lt;artifactId&gt;xyz&lt;/artifactId&gt;
        &lt;version&gt;1.0&lt;/version&gt;
        &lt;relativePath&gt;../pom.xml&lt;/relativePath&gt;
    &lt;/parent&gt;
        &lt;artifactId&gt;SparkPOC&lt;/artifactId&gt;
    &lt;version&gt;0.0.1-SNAPSHOT&lt;/version&gt;
    &lt;packaging&gt;jar&lt;/packaging&gt;
        &lt;name&gt;SparkPOCde&lt;/name&gt;
        &lt;url&gt;http://maven.apache.org&lt;/url&gt;
&lt;properties&gt;

&lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;


&lt;spark-version&gt;1.6.1&lt;/spark-version&gt;

&lt;kafka-version&gt;0.9.0.0&lt;/kafka-version&gt;

&lt;/properties&gt;


&lt;dependencies&gt;

&lt;!-- http://mvnrepository.com/artifact/org.springframework/spring-core --&gt;



&lt;!-- http://mvnrepository.com/artifact/org.springframework/spring-jdbc --&gt;
&lt;dependency&gt;

&lt;groupId&gt;log4j&lt;/groupId&gt;

&lt;artifactId&gt;log4j&lt;/artifactId&gt;

&lt;version&gt;1.2.17&lt;/version&gt;

&lt;/dependency&gt;

&lt;!-- http://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 --&gt;

&lt;dependency&gt;
&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;

&lt;artifactId&gt;spark-streaming_2.10&lt;/artifactId&gt;

&lt;version&gt;${spark-version}&lt;/version&gt;

&lt;/dependency&gt;

&lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 --&gt;



&lt;dependency&gt;

&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;

&lt;artifactId&gt;spark-streaming-kafka_2.10&lt;/artifactId&gt;

&lt;version&gt;1.6.2&lt;/version&gt;

 &lt;exclusions&gt;
        &lt;exclusion&gt;
            &lt;groupId&gt;io.netty&lt;/groupId&gt;
            &lt;artifactId&gt;netty&lt;/artifactId&gt;
        &lt;/exclusion&gt;

        &lt;exclusion&gt;
            &lt;groupId&gt;io.jboss.netty&lt;/groupId&gt;
            &lt;artifactId&gt;netty&lt;/artifactId&gt;
        &lt;/exclusion&gt;
    &lt;/exclusions&gt; 

&lt;/dependency&gt;

   &lt;dependency&gt;
            &lt;groupId&gt;com.abcd.rep.xyz&lt;/groupId&gt;
            &lt;artifactId&gt;xyzCommon&lt;/artifactId&gt;
            &lt;version&gt;1.0&lt;/version&gt;
            &lt;type&gt;jar&lt;/type&gt;
        &lt;/dependency&gt;


&lt;!-- http://mvnrepository.com/artifact/ojdbc/ojdbc --&gt;


&lt;!-- &lt;dependency&gt; &lt;groupId&gt;ojdbc&lt;/groupId&gt; &lt;artifactId&gt;ojdbc&lt;/artifactId&gt; &lt;version&gt;14&lt;/version&gt; &lt;/dependency&gt;--&gt;


&lt;!-- https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver --&gt;




&lt;!-- http://mvnrepository.com/artifact/org.springframework.data/spring-data-mongodb --&gt;




&lt;!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple --&gt;





&lt;/dependencies&gt;


&lt;build&gt;

&lt;finalName&gt;appname&lt;/finalName&gt;


&lt;resources&gt;


&lt;resource&gt;

&lt;directory&gt;src/main/resources&lt;/directory&gt;


&lt;excludes&gt;

&lt;exclude&gt;eventRules.json&lt;/exclude&gt;

&lt;exclude&gt;log4j.xml&lt;/exclude&gt;

&lt;exclude&gt;resources.properties&lt;/exclude&gt;

&lt;/excludes&gt;

&lt;/resource&gt;

&lt;/resources&gt;


&lt;plugins&gt;


&lt;plugin&gt;

&lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;

&lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;

&lt;version&gt;3.5.1&lt;/version&gt;


&lt;configuration&gt;

&lt;source&gt;1.8&lt;/source&gt;

&lt;target&gt;1.8&lt;/target&gt;

&lt;/configuration&gt;

&lt;/plugin&gt;


&lt;plugin&gt;

&lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;

&lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;

&lt;version&gt;2.4.3&lt;/version&gt;


&lt;executions&gt;


&lt;execution&gt;

&lt;phase&gt;package&lt;/phase&gt;


&lt;goals&gt;

&lt;goal&gt;shade&lt;/goal&gt;

&lt;/goals&gt;


&lt;configuration&gt;


&lt;filters&gt;


&lt;filter&gt;

&lt;artifact&gt;*:*&lt;/artifact&gt;


-&lt;excludes&gt;

&lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;

&lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;

&lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;

&lt;/excludes&gt;

&lt;/filter&gt;

&lt;/filters&gt;


&lt;transformers&gt;


-&lt;transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"&gt;

&lt;resource&gt;reference.conf&lt;/resource&gt;

&lt;/transformer&gt;


-&lt;transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"&gt;

&lt;mainClass&gt;com.abcd.rep.xyz.SparkPOCde.EventConsumerServiceImpl&lt;/mainClass&gt;

&lt;/transformer&gt;

&lt;/transformers&gt;

&lt;/configuration&gt;

&lt;/execution&gt;

&lt;/executions&gt;

&lt;/plugin&gt;

&lt;/plugins&gt;

&lt;/build&gt;

&lt;/project&gt;
</code></pre>