Apache spark Hive, executable JAR with maven shade
<p>I'm building apache-spark application with Apache Spark Hive. So far everything was ok - I've been running tests and whole application in Intellij IDEA and all tests together using maven.</p>

<p>Now I want to run whole application from bash and let it run with local single-node cluster. I'm using maven-shade-plugin to build single executable JAR.</p>

<p>Application crashes when it tries to create new HiveContext out of SparkContext. Thrown exception tells me that hive can't create metastore because there is some problem with datanucleus and its plugin system. I tried to follow several questions how to run datanucleus plugin system with shade but out of luck. For example: 
<a href="https://stackoverflow.com/questions/10096772/datanucleus-jdo-and-executable-jar-how-to-do-it">Datanucleus, JDO and executable jar - how to do it?</a></p>

<p>What is the best way to compose executable JAR of application using hive and run it from bash? Perhaps some settings of datanucleus and its plugin system?</p>

<p><strong>pom.xml</strong></p>

<pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;

    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;

    &lt;groupId&gt;test&lt;/groupId&gt;
    &lt;artifactId&gt;hive-test&lt;/artifactId&gt;
    &lt;version&gt;1.0.0&lt;/version&gt;

    &lt;dependencies&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
            &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
            &lt;version&gt;2.11.7&lt;/version&gt;
        &lt;/dependency&gt;

        &lt;!-- spark --&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
            &lt;artifactId&gt;spark-core_2.11&lt;/artifactId&gt;
            &lt;version&gt;1.6.0&lt;/version&gt;
        &lt;/dependency&gt;

        &lt;dependency&gt;
            &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
            &lt;artifactId&gt;spark-hive_2.11&lt;/artifactId&gt;
            &lt;version&gt;1.6.0&lt;/version&gt;
        &lt;/dependency&gt;
    &lt;/dependencies&gt;

    &lt;properties&gt;
        &lt;!-- To be specified in child pom:  &lt;main.class&gt;&lt;/main.class&gt; --&gt;
        &lt;final.jar.name&gt;${project.artifactId}-${project.version}&lt;/final.jar.name&gt;
        &lt;main.class&gt;com.test.HiveTest&lt;/main.class&gt;
    &lt;/properties&gt;

    &lt;build&gt;
        &lt;plugins&gt;
            &lt;!-- the Maven compiler plugin will compile Java source files --&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                &lt;version&gt;3.3&lt;/version&gt;
                &lt;configuration&gt;
                    &lt;source&gt;${java.version}&lt;/source&gt;
                    &lt;target&gt;${java.version}&lt;/target&gt;
                &lt;/configuration&gt;
            &lt;/plugin&gt;

            &lt;!-- the Maven Scala plugin will compile Scala source files --&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;org.scala-tools&lt;/groupId&gt;
                &lt;artifactId&gt;maven-scala-plugin&lt;/artifactId&gt;
                &lt;version&gt;2.15.2&lt;/version&gt;
                &lt;executions&gt;
                    &lt;execution&gt;
                        &lt;id&gt;scala-compile-first&lt;/id&gt;
                        &lt;phase&gt;process-resources&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;add-source&lt;/goal&gt;
                            &lt;goal&gt;compile&lt;/goal&gt;
                        &lt;/goals&gt;
                        &lt;configuration&gt;
                            &lt;args&gt;
                                &lt;arg&gt;-Xmax-classfile-name&lt;/arg&gt;
                                &lt;arg&gt;110&lt;/arg&gt;
                            &lt;/args&gt;
                        &lt;/configuration&gt;
                    &lt;/execution&gt;
                    &lt;execution&gt;
                        &lt;id&gt;scala-test-compile&lt;/id&gt;
                        &lt;phase&gt;process-test-resources&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;testCompile&lt;/goal&gt;
                        &lt;/goals&gt;
                        &lt;configuration&gt;
                            &lt;args&gt;
                                &lt;arg&gt;-Xmax-classfile-name&lt;/arg&gt;
                                &lt;arg&gt;110&lt;/arg&gt;
                            &lt;/args&gt;
                        &lt;/configuration&gt;
                    &lt;/execution&gt;
                &lt;/executions&gt;
            &lt;/plugin&gt;

            &lt;plugin&gt;
                &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
                &lt;executions&gt;
                    &lt;execution&gt;
                        &lt;phase&gt;package&lt;/phase&gt;
                        &lt;goals&gt;
                            &lt;goal&gt;shade&lt;/goal&gt;
                        &lt;/goals&gt;
                        &lt;configuration&gt;
                            &lt;transformers&gt;
                                &lt;transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"&gt;
                                    &lt;mainClass&gt;${main.class}&lt;/mainClass&gt;
                                &lt;/transformer&gt;
                                &lt;transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"&gt;
                                    &lt;resource&gt;reference.conf&lt;/resource&gt;
                                &lt;/transformer&gt;
                                &lt;transformer implementation="org.apache.maven.plugins.shade.resource.XmlAppendingTransformer"&gt;
                                    &lt;resource&gt;plugin.xml&lt;/resource&gt;
                                &lt;/transformer&gt;
                                &lt;transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/&gt;
                            &lt;/transformers&gt;
                            &lt;filters&gt;
                                &lt;filter&gt;
                                    &lt;artifact&gt;*:*&lt;/artifact&gt;
                                    &lt;excludes&gt;
                                        &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                                        &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                                        &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                                    &lt;/excludes&gt;
                                &lt;/filter&gt;
                            &lt;/filters&gt;
                        &lt;/configuration&gt;
                    &lt;/execution&gt;
                &lt;/executions&gt;
            &lt;/plugin&gt;
        &lt;/plugins&gt;
    &lt;/build&gt;
&lt;/project&gt;
</code></pre>

<p><strong>SampleCode</strong></p>

<pre><code>object HiveTest {
  def main(args: Array[String]) {
    // Set up Spark
    val conf = new SparkConf(true)
      .setMaster("local")
      .setAppName("hive-test")

    println("Initializing spark context")
    val sc = new SparkContext(conf)

    println("Initializing hive context")
    val hc = new HiveContext(sc)
  }
}
</code></pre>

<p><strong>Thrown exception</strong></p>

<pre><code>java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.spark.sql.hive.client.ClientWrapper.&lt;init&gt;(ClientWrapper.scala:194)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
        at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
        at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
        at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
        at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
        at org.apache.spark.sql.UDFRegistration.&lt;init&gt;(UDFRegistration.scala:40)
        at org.apache.spark.sql.SQLContext.&lt;init&gt;(SQLContext.scala:330)
        at org.apache.spark.sql.hive.HiveContext.&lt;init&gt;(HiveContext.scala:97)
        at org.apache.spark.sql.hive.HiveContext.&lt;init&gt;(HiveContext.scala:101)
        at com.test.HiveTest$.main(HiveTest.scala:21)
        at com.test.HiveTest.main(HiveTest.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
        ... 12 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
        ... 18 more
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1193)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:365)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:394)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:291)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:258)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:57)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:66)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:624)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:66)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:199)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.&lt;init&gt;(SessionHiveMetaStoreClient.java:74)
        ... 23 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
        ... 42 more
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
        at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:283)
        at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:247)
        at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:225)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.&lt;init&gt;(JDOPersistenceManagerFactory.java:416)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
        ... 50 more
</code></pre>

<p>Thank you in advance</p>