{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/497","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/497/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/497/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/497/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/497","id":94416880,"number":497,"title":"DataFrame contains scala buffers only when query is specified","user":{"login":"analyticswarescott","id":10065058,"avatar_url":"https://avatars.githubusercontent.com/u/10065058?v=3","gravatar_id":"","url":"https://api.github.com/users/analyticswarescott","html_url":"https://github.com/analyticswarescott","followers_url":"https://api.github.com/users/analyticswarescott/followers","following_url":"https://api.github.com/users/analyticswarescott/following{/other_user}","gists_url":"https://api.github.com/users/analyticswarescott/gists{/gist_id}","starred_url":"https://api.github.com/users/analyticswarescott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/analyticswarescott/subscriptions","organizations_url":"https://api.github.com/users/analyticswarescott/orgs","repos_url":"https://api.github.com/users/analyticswarescott/repos","events_url":"https://api.github.com/users/analyticswarescott/events{/privacy}","received_events_url":"https://api.github.com/users/analyticswarescott/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/enhancement","name":"enhancement","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.1.1","name":"v2.1.1","color":"bfdadc"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.2.0.m1","name":"v2.2.0.m1","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-11T00:11:38Z","updated_at":"2015-08-04T17:27:40Z","closed_at":null,"body":"The DataFrame returned by JavaEsSparkSQL.esDF contains Scala Buffers when a query string is specified, but not when the simpler overload is used.  Code snippet and log output is below.\r\n\r\n```java\r\n    DataFrame rdd = null;\r\n            SourceLoaderES.logger.info(\" loading \" + indexName + \"/\" + docType + \" using REST to get column list \");\r\n            \r\n            String query = getQueryString(indexName, docType);\r\n            SourceLoaderES.logger.debug(\"ES query string\" + query);          \t\t\r\n           // rdd = JavaEsSparkSQL.esDF(_handler._sc, indexName + \"/\" + docType, query);\r\n            \r\n           rdd = JavaEsSparkSQL.esDF(_handler._sc, indexName + \"/\" + docType);\r\n           DataFrame rdd2 = JavaEsSparkSQL.esDF(_handler._sc, indexName + \"/\" + docType, query);\r\n           \r\n        \r\n        SourceLoaderES.logger.debug(\"inferred SCHEMA \" + rdd.schema().toString());\r\n\r\n\r\n        if (AdminMgr.getConfig(SyncRunner.DEBUG_ENABLE_DF_COUNTS) != null) {\r\n\t    \tif (AdminMgr.getConfig(SyncRunner.DEBUG_ENABLE_DF_COUNTS).equals( \"true\")) {\r\n\t           logger.warn(\" DataFrame count: \" + rdd.count());\r\n\t           logger.warn(\" DataFrame first row: \" + rdd.showString(1));\r\n\t    \t}\r\n        }\r\n        logger.warn(\" DataFrame first row: \" + rdd.showString(1));\r\n        logger.warn(\" DataFrame2 first row: \" + rdd2.showString(1));\r\n\t\t//SourceLoaderES.logger.info(\"ES SOURCE \" + sourceName + \"  SCHEMA \" + rdd.schema().toString());\r\n\r\n```\r\n\r\nLog output from this snippet:\r\n\r\n```\r\n2015-07-10 17:09:16,281 [ (Sync) 15 - Sync 1] INFO  com.dg.data.sync.SourceLoaderES -  loading g5b778bb6-faa0-41a0-bb36-8a4c54b13774_dg_dim1/dim_lookups using REST to get column list \r\n2015-07-10 17:09:16,297 [ (Sync) 15 - Sync 1] DEBUG com.dg.data.sync.SourceLoaderES -  returned mapping for query string: {\"properties\":{\"xid\":{\"type\":\"string\"},\"ApplicationLanguageId\":{\"type\":\"long\"},\"LookupKeyActive\":{\"type\":\"string\"},\"LookupKey\":{\"type\":\"long\"},\"LookupLevel\":{\"type\":\"long\"},\"LookupModule\":{\"type\":\"long\"},\"LookupKeyName\":{\"type\":\"string\"}}}\r\n2015-07-10 17:09:16,297 [ (Sync) 15 - Sync 1] DEBUG com.dg.data.sync.SourceLoaderES - ES query string{\"query\":{\"match_all\":{}},\"fields\":[\"xid\",\"ApplicationLanguageId\",\"LookupKeyActive\",\"LookupKey\",\"LookupLevel\",\"LookupModule\",\"LookupKeyName\"]}\r\n2015-07-10 17:09:16,359 [ (Sync) 15 - Sync 1] DEBUG com.dg.data.sync.SourceLoaderES - inferred SCHEMA StructType(StructField(ApplicationLanguageId,LongType,true), StructField(LookupKey,LongType,true), StructField(LookupKeyActive,StringType,true), StructField(LookupKeyName,StringType,true), StructField(LookupLevel,LongType,true), StructField(LookupModule,LongType,true), StructField(xid,StringType,true))\r\n2015-07-10 17:09:16,641 [ (Sync) 15 - Sync 1] WARN  com.dg.data.sync.SourceLoaderES -  DataFrame first row: ApplicationLanguageId LookupKey LookupKeyActive LookupKeyName LookupLevel LookupModule xid           \r\n1                     11        true            11 am         2           61           61-2-true-1-11\r\n2015-07-10 17:09:16,797 [ (Sync) 15 - Sync 1] WARN  com.dg.data.sync.SourceLoaderES -  DataFrame2 first row: ApplicationLanguageId LookupKey  LookupKeyActive LookupKeyName LookupLevel LookupModule xid                 \r\nBuffer(1)             Buffer(11) Buffer(true)    Buffer(11 am) Buffer(2)   Buffer(61)   Buffer(61-2-true-...\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/127685154","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/497#issuecomment-127685154","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/497","id":127685154,"user":{"login":"analyticswarescott","id":10065058,"avatar_url":"https://avatars.githubusercontent.com/u/10065058?v=3","gravatar_id":"","url":"https://api.github.com/users/analyticswarescott","html_url":"https://github.com/analyticswarescott","followers_url":"https://api.github.com/users/analyticswarescott/followers","following_url":"https://api.github.com/users/analyticswarescott/following{/other_user}","gists_url":"https://api.github.com/users/analyticswarescott/gists{/gist_id}","starred_url":"https://api.github.com/users/analyticswarescott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/analyticswarescott/subscriptions","organizations_url":"https://api.github.com/users/analyticswarescott/orgs","repos_url":"https://api.github.com/users/analyticswarescott/repos","events_url":"https://api.github.com/users/analyticswarescott/events{/privacy}","received_events_url":"https://api.github.com/users/analyticswarescott/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T17:27:40Z","updated_at":"2015-08-04T17:27:40Z","body":"Your example test query string does not specify a set of fields to return,\nonly a constraint.  The \"fields\" query element is what causes the issue in\nmy tests, including in Spark 1.4.1.\n\nI will provide more information as soon as I am able.\n\n--Scott\n\n\n\n\nOn Tue, Jul 28, 2015 at 11:49 AM, Costin Leau <notifications@github.com>\nwrote:\n\n> @analyticswarescott <https://github.com/analyticswarescott> Unfortunately\n> I'm unable to reproduce the problem. What version of Es-hadoop and Spark\n> are you using? Are you by any chance on Spark 1.3?\n> On 2.1.x and master, everything gets properly printed - maybe your query\n> is special (can you share it)?\n> I've also noticed you are using showString which is not available in\n> Spark 1.4 (in fact it is private).\n> Either way, I have tried the following snippets (which is part of the test\n> suite) and I get no reference to Buffer:\n>\n> println(JavaEsSparkSQL.esDF(sqc, target, \"?q=name:me*\", cfg.asJava).head())\n>\n> [170,Mew,http://userserve-ak.last.fm/serve/252/42247291.jpg,6146220025000,http://www.last.fm/music/Mew]``` <http://userserve-ak.last.fm/serve/252/42247291.jpg,6146220025000,http://www.last.fm/music/Mew%5D>\n>\n> println(JavaEsSparkSQL.esDF(sqc, target, \"?q=name:me*\", cfg.asJava).show(3)\n>\n>\n> +---+--------------------+--------------------+--------------+--------------------+\n> | id| name| pictures| time| url|\n>\n> +---+--------------------+--------------------+--------------+--------------------+\n> |170| Mew|http://userserve-...| 6146220025000|http://www.last.f...|\n> |918| Megadeth|http://userserve-...|29656092025000|http://www.last.f...|\n> |996|Mike & The Mechanics|\n> http://userserve-...|32117541625000|http://www.last.f...|\n>\n> +---+--------------------+--------------------+--------------+--------------------+\n>\n>\n> Can you post your entire log and potentially turn on logging on the rest package (`org.elasticsearch.hadoop.rest`) all the way to `TRACE` and upload the result as a gist?\n>\n> Thanks\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/elastic/elasticsearch-hadoop/issues/497#issuecomment-125526221>\n> .\n>\n"}}