{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/529","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/529/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/529/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/529/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/529","id":98413587,"number":529,"title":"Spark 1.3 -- Error in retrieving an array element from Elastic Search","user":{"login":"amajumdarinfa","id":13589648,"avatar_url":"https://avatars.githubusercontent.com/u/13589648?v=3","gravatar_id":"","url":"https://api.github.com/users/amajumdarinfa","html_url":"https://github.com/amajumdarinfa","followers_url":"https://api.github.com/users/amajumdarinfa/followers","following_url":"https://api.github.com/users/amajumdarinfa/following{/other_user}","gists_url":"https://api.github.com/users/amajumdarinfa/gists{/gist_id}","starred_url":"https://api.github.com/users/amajumdarinfa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amajumdarinfa/subscriptions","organizations_url":"https://api.github.com/users/amajumdarinfa/orgs","repos_url":"https://api.github.com/users/amajumdarinfa/repos","events_url":"https://api.github.com/users/amajumdarinfa/events{/privacy}","received_events_url":"https://api.github.com/users/amajumdarinfa/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/hive","name":"hive","color":"02d7e1"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.1.1","name":"v2.1.1","color":"bfdadc"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.2.0.m1","name":"v2.2.0.m1","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-31T15:29:37Z","updated_at":"2015-07-31T17:12:43Z","closed_at":null,"body":"Hi\r\nEnvironment: Spark 1.3.0\r\n Elasticsearch: 1.4.4\r\n\r\nInside ElasticSearch I have a JSON document with a JSON array. I run a spark application that loads the ElasticSearch Type as a temp table and runs Hive Thrift Server 2. In the client side I have a simple JDBC client that executes a simple SQL statement like \"select * from X\". The spark application fails to verify the resultset due to the existence of the JSON array and throws the following exception.\r\n\r\n```\r\nException in thread \"main\" org.apache.hive.service.cli.HiveSQLException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 10, localhost): java.lang.ClassCastException: scala.collection.convert.Wrappers$JListWrapper cannot be cast to java.lang.String\r\n       at org.elasticsearch.spark.sql.ScalaEsRow.getString(ScalaEsRow.scala:58)\r\n       at org.apache.spark.sql.columnar.StringColumnStats.gatherStats(ColumnStats.scala:208)\r\n       at org.apache.spark.sql.columnar.NullableColumnBuilder$class.appendFrom(NullableColumnBuilder.scala:56)\r\n       at org.apache.spark.sql.columnar.NativeColumnBuilder.org$apache$spark$sql$columnar$compression$CompressibleColumnBuilder$$super$appendFrom(ColumnBuilder.scala:87)\r\n       at org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder$class.appendFrom(CompressibleColumnBuilder.scala:78)\r\n       at org.apache.spark.sql.columnar.NativeColumnBuilder.appendFrom(ColumnBuilder.scala:87)\r\n       at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:135)\r\n       at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:111)\r\n       at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249)\r\n       at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:172)\r\n       at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:79)\r\n       at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)\r\n       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)\r\n       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)\r\n       at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)\r\n       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)\r\n       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)\r\n       at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)\r\n       at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)\r\n       at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\r\n       at org.apache.spark.scheduler.Task.run(Task.scala:64)\r\n       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)\r\n       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       at java.lang.Thread.run(Thread.java:745)\r\n\r\nDriver stacktrace:\r\n       at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)\r\n       at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)\r\n       at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)\r\n       at com.infa.sats.daoTest.Test.main(Test.java:34)\r\n```\r\n\r\nThe JSON document looks like this -->\r\n```json\r\n{\r\n    \"username\": \"CCC\",\r\n    \"firstName\" : \"CC\",\r\n    \"lastName\" : \"C\",\r\n    \"fullName\" : \"CCC\",\r\n    \"email\" : \"CCC2@infa.com\",\r\n    \"groups\" : [\"Group_111\"] \r\n}\r\n```\r\nNote that if I do not have \"groups\" as an array the issue goes away. \r\n\r\nIs it a known issue? Do you suggest any work-around?\r\n\r\nThanks\r\nAbhijit"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/126755363","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/529#issuecomment-126755363","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/529","id":126755363,"user":{"login":"amajumdarinfa","id":13589648,"avatar_url":"https://avatars.githubusercontent.com/u/13589648?v=3","gravatar_id":"","url":"https://api.github.com/users/amajumdarinfa","html_url":"https://github.com/amajumdarinfa","followers_url":"https://api.github.com/users/amajumdarinfa/followers","following_url":"https://api.github.com/users/amajumdarinfa/following{/other_user}","gists_url":"https://api.github.com/users/amajumdarinfa/gists{/gist_id}","starred_url":"https://api.github.com/users/amajumdarinfa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amajumdarinfa/subscriptions","organizations_url":"https://api.github.com/users/amajumdarinfa/orgs","repos_url":"https://api.github.com/users/amajumdarinfa/repos","events_url":"https://api.github.com/users/amajumdarinfa/events{/privacy}","received_events_url":"https://api.github.com/users/amajumdarinfa/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T17:12:42Z","updated_at":"2015-07-31T17:12:42Z","body":"Thanks Costin!\r\n\r\nTHe es-hadoop version is 2.1.0.\r\n\r\nOk, that means the ES Type must have a schema in order to successfully integrate an array field with Spark SQL."}}