{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/484","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/484/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/484/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/484/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/484","id":90724585,"number":484,"title":"java.util.NoSuchElementException: None.get on 2.1.0.rc1 when Array of objects in mapping","user":{"login":"rhernando","id":616969,"avatar_url":"https://avatars.githubusercontent.com/u/616969?v=3","gravatar_id":"","url":"https://api.github.com/users/rhernando","html_url":"https://github.com/rhernando","followers_url":"https://api.github.com/users/rhernando/followers","following_url":"https://api.github.com/users/rhernando/following{/other_user}","gists_url":"https://api.github.com/users/rhernando/gists{/gist_id}","starred_url":"https://api.github.com/users/rhernando/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhernando/subscriptions","organizations_url":"https://api.github.com/users/rhernando/orgs","repos_url":"https://api.github.com/users/rhernando/repos","events_url":"https://api.github.com/users/rhernando/events{/privacy}","received_events_url":"https://api.github.com/users/rhernando/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-24T16:24:09Z","updated_at":"2015-06-27T14:25:16Z","closed_at":null,"body":"Using;\r\n* Elasticsearch 1.6.0\r\n* Spark 1.4.0\r\n* Elasticsearch-hadoop 2.1.0-RC1\r\n\r\nWhen we query with SparkSql we get an error:\r\n\r\n```\r\n    val df = sqlc.read.format(\"org.elasticsearch.spark.sql\").load(\"idxcli/exploits\")\r\n    println(df.count())\r\n```\r\n\r\n```\r\njava.util.NoSuchElementException: None.get\r\n\tat scala.None$.get(Option.scala:313)\r\n\tat scala.None$.get(Option.scala:311)\r\n\tat org.elasticsearch.spark.sql.RowValueReader$class.rowOrder(RowValueReader.scala:24)\r\n\tat org.elasticsearch.spark.sql.ScalaRowValueReader.rowOrder(ScalaEsRowValueReader.scala:13)\r\n\tat org.elasticsearch.spark.sql.ScalaRowValueReader.createMap(ScalaEsRowValueReader.scala:32)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:620)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:559)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:636)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:559)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:636)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:559)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:358)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:293)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:188)\r\n\tat org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:167)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:403)\r\n\tat org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:76)\r\n\tat org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:46)\r\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)\r\n\tat org.apache.spark.sql.execution.Aggregate$$anonfun$doExecute$1$$anonfun$6.apply(Aggregate.scala:129)\r\n\tat org.apache.spark.sql.execution.Aggregate$$anonfun$doExecute$1$$anonfun$6.apply(Aggregate.scala:126)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:244)\r\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:244)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:70)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nI think the problem occur because we have an Array of objects in our index\r\n\r\nOur index:\r\n\r\n```\r\n\"exploits\" : {\r\n        \"properties\" : {\r\n          \"actions\" : {\r\n            \"properties\" : {\r\n              \"action\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_author\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_event\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_id\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_section\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_subsection\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"logobj_tags\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"relation_progress\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"subject_geoip_countrycode\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"subject_origin\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"subject_referrer\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"subject_ua_name\" : {\r\n                \"type\" : \"string\"\r\n              },\r\n              \"subject_ua_os\" : {\r\n                \"type\" : \"string\"\r\n              }\r\n            }\r\n          },\r\n          \"category\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"description\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"endDate\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"evtType\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"id\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"name\" : {\r\n            \"type\" : \"string\",\r\n            \"store\" : true\r\n          },\r\n          \"points\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"startDate\" : {\r\n            \"type\" : \"string\"\r\n          },\r\n          \"status\" : {\r\n            \"type\" : \"boolean\"\r\n          },\r\n          \"type\" : {\r\n            \"type\" : \"string\"\r\n          }\r\n        }\r\n      }\r\n    }\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/116067581","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/484#issuecomment-116067581","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/484","id":116067581,"user":{"login":"skurfuerst","id":190777,"avatar_url":"https://avatars.githubusercontent.com/u/190777?v=3","gravatar_id":"","url":"https://api.github.com/users/skurfuerst","html_url":"https://github.com/skurfuerst","followers_url":"https://api.github.com/users/skurfuerst/followers","following_url":"https://api.github.com/users/skurfuerst/following{/other_user}","gists_url":"https://api.github.com/users/skurfuerst/gists{/gist_id}","starred_url":"https://api.github.com/users/skurfuerst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skurfuerst/subscriptions","organizations_url":"https://api.github.com/users/skurfuerst/orgs","repos_url":"https://api.github.com/users/skurfuerst/repos","events_url":"https://api.github.com/users/skurfuerst/events{/privacy}","received_events_url":"https://api.github.com/users/skurfuerst/received_events","type":"User","site_admin":false},"created_at":"2015-06-27T14:25:16Z","updated_at":"2015-06-27T14:25:16Z","body":"Hey,\r\n\r\nseems I have the same error; also having arrays of objects in the index.\r\n\r\nAll the best,\r\nSebastian"}}