{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/449","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/449/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/449/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/449/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/449","id":75753531,"number":449,"title":"Spark SQL 1.3 - Exception in Elasticsearch when executing JOIN with DataFrame created from Oracle's table","user":{"login":"fdmitriy","id":11669549,"avatar_url":"https://avatars.githubusercontent.com/u/11669549?v=3","gravatar_id":"","url":"https://api.github.com/users/fdmitriy","html_url":"https://github.com/fdmitriy","followers_url":"https://api.github.com/users/fdmitriy/followers","following_url":"https://api.github.com/users/fdmitriy/following{/other_user}","gists_url":"https://api.github.com/users/fdmitriy/gists{/gist_id}","starred_url":"https://api.github.com/users/fdmitriy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fdmitriy/subscriptions","organizations_url":"https://api.github.com/users/fdmitriy/orgs","repos_url":"https://api.github.com/users/fdmitriy/repos","events_url":"https://api.github.com/users/fdmitriy/events{/privacy}","received_events_url":"https://api.github.com/users/fdmitriy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.1.0.Beta5","name":"v2.1.0.Beta5","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-05-12T21:57:21Z","updated_at":"2015-05-17T08:46:04Z","closed_at":null,"body":"Hi,\r\n\r\nEnvironment: Spark 1.3.0\r\nElasticsearch: 1.4.4\r\nElasticsearch-Hadoop: 2.1.0.Beta4\r\nOracle Express 11g XE\r\n\r\nI am trying to run a join between 2 DataFrames, one from Elasticsearch and another one from Oracle and getting exception as below.\r\nThis issue only happens when loading Elasticsearch data using Spark SQL \"load\" function. It doesn't happen when loading using esDF function. However, when using esDF function, mappings of columns to column names is broken (issue: https://github.com/elastic/elasticsearch-hadoop/issues/451)\r\n\r\ncom.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.elasticsearch.spark.sql.ScalaEsRow\r\n\tat com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1050)\r\n\tat com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1062)\r\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:228)\r\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:217)\r\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)\r\n\tat com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)\r\n\tat com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)\r\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)\r\n\tat org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:138)\r\n\tat org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)\r\n\tat org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)\r\n\tat org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)\r\n\tat scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)\r\n\tat org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)\r\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat org.apache.spark.sql.execution.joins.HashJoin$$anon$1.hasNext(HashJoin.scala:66)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:798)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:798)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)\r\n\tat org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:64)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\nCode:\r\n\r\n    //=======================================================================\r\n    // Preparing Elasticsearch DataFrame\r\n    //=======================================================================\r\n\r\n    val sqlContext = new SQLContext(sc)\r\n    val hours = sqlContext.load(\"summary/hours\", \"org.elasticsearch.spark.sql\")\r\n    hours.foreach (x => println(x))\r\n\r\n    //=======================================================================\r\n    // Preparing Oracle DataFrame\r\n    //=======================================================================\r\n    \r\n    val users = sqlContext.load(\"jdbc\", Map(\"url\" -> dbConnectionString,\r\n                                        \"dbtable\" -> \"sats.users\",\r\n                                        \"driver\"  -> \"oracle.jdbc.driver.OracleDriver\"))\r\n    users.foreach (x => println(x))\r\n\r\n    //=======================================================================\r\n    // Joining ES and Oracle DataFrames\r\n    //=======================================================================\r\n    \r\n    val hoursAug = hours.join(users, hours(\"User\") === users(\"USERNAME\"))\r\n    hoursAug.foreach (x => println(x))\r\n\r\nThe elements of Elasticsearch and Oracle DataFrames are printed successfully on the screen (first two foreach{...}) and the exceptions happens when join is executed.\r\n\r\nCan you please advise if this is a bug and if yes, is there any workaround?\r\n\r\nThanks,\r\nDmitriy Fingerman"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/102769063","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/449#issuecomment-102769063","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/449","id":102769063,"user":{"login":"costin","id":76245,"avatar_url":"https://avatars.githubusercontent.com/u/76245?v=3","gravatar_id":"","url":"https://api.github.com/users/costin","html_url":"https://github.com/costin","followers_url":"https://api.github.com/users/costin/followers","following_url":"https://api.github.com/users/costin/following{/other_user}","gists_url":"https://api.github.com/users/costin/gists{/gist_id}","starred_url":"https://api.github.com/users/costin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costin/subscriptions","organizations_url":"https://api.github.com/users/costin/orgs","repos_url":"https://api.github.com/users/costin/repos","events_url":"https://api.github.com/users/costin/events{/privacy}","received_events_url":"https://api.github.com/users/costin/received_events","type":"User","site_admin":false},"created_at":"2015-05-17T08:46:04Z","updated_at":"2015-05-17T08:46:04Z","body":"Hi @fdmitriy ,\r\n\r\nThis looks indeed like a bug - I've pushed a fix (and the new snapshot builds) to address it - can you please give a try and report back? it might address #451 or not - haven't had time to look into it yet."}}