{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9517","labels_url":"https://api.github.com/repos/apache/spark/issues/9517/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9517/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9517/events","html_url":"https://github.com/apache/spark/pull/9517","id":115455734,"number":9517,"title":"[SPARK-11500][SQL] Not deterministic order of columns when using merging schemas.","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-11-06T07:51:46Z","updated_at":"2015-11-10T04:20:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9517","html_url":"https://github.com/apache/spark/pull/9517","diff_url":"https://github.com/apache/spark/pull/9517.diff","patch_url":"https://github.com/apache/spark/pull/9517.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-11500\r\n\r\nAs filed in SPARK-11500, if merging schemas is enabled, the order of files to touch is a matter which might affect the ordering of the output columns. \r\n\r\nThis was mostly because of the use of `Set` and `Map` so I replaced them to `LinkedHashSet` and `LinkedHashMap` to keep the insertion order.\r\n\r\nAlso, I changed `reduceOption` to `reduceLeftOption`, and replaced the order of `filesToTouch` from `metadataStatuses ++ commonMetadataStatuses ++ needMerged` to  `needMerged ++ metadataStatuses ++ commonMetadataStatuses` in order to touch the part-files first which always have the schema in footers whereas the others might not exist.\r\n\r\nOne nit is, If merging schemas is not enabled, but when multiple files are given, there is no guarantee of the output order, since there might not be a summary file for the first file, which ends up putting ahead the columns of the other files. \r\n\r\nHowever, I thought this should be okay since disabling merging schemas means (assumes) all the files have the same schemas.\r\n\r\nIn addition, in the test code for this, I only checked the names of fields."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/155281726","html_url":"https://github.com/apache/spark/pull/9517#issuecomment-155281726","issue_url":"https://api.github.com/repos/apache/spark/issues/9517","id":155281726,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T04:20:08Z","updated_at":"2015-11-10T04:20:08Z","body":"**[Test build #45489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45489/consoleFull)** for PR 9517 at commit [`32dfb87`](https://github.com/apache/spark/commit/32dfb87ce36a093c54d4a3dfd39ccbc00c417af9)."}}