{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/9419","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9419/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9419/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9419/events","html_url":"https://github.com/elastic/elasticsearch/issues/9419","id":55492990,"number":9419,"title":"Doc_values are ignored after upgrade to 1.3.7","user":{"login":"dimzak","id":4560173,"avatar_url":"https://avatars.githubusercontent.com/u/4560173?v=3","gravatar_id":"","url":"https://api.github.com/users/dimzak","html_url":"https://github.com/dimzak","followers_url":"https://api.github.com/users/dimzak/followers","following_url":"https://api.github.com/users/dimzak/following{/other_user}","gists_url":"https://api.github.com/users/dimzak/gists{/gist_id}","starred_url":"https://api.github.com/users/dimzak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dimzak/subscriptions","organizations_url":"https://api.github.com/users/dimzak/orgs","repos_url":"https://api.github.com/users/dimzak/repos","events_url":"https://api.github.com/users/dimzak/events{/privacy}","received_events_url":"https://api.github.com/users/dimzak/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-01-26T15:10:30Z","updated_at":"2015-04-20T12:27:46Z","closed_at":null,"body":"We have daily indexes with a mapping for field `ts as doc_value`. \r\nMapping of ts field before and after upgrade:\r\n`GET /delivery-logs-index.prod-20150126/_mapping/`\r\nts part response:\r\n```json\r\n \"ts\": {\r\n                  \"type\": \"date\",\r\n                  \"doc_values\": true,\r\n                  \"format\": \"dateOptionalTime\"\r\n               }\r\n```\r\n\r\nBut then when observing fielddata:\r\n`GET /delivery-logs-index.prod-20150126/_stats/fielddata?fields=ts`\r\nGot this:\r\n```json\r\n{\r\n   \"_shards\": {\r\n      \"total\": 24,\r\n      \"successful\": 24,\r\n      \"failed\": 0\r\n   },\r\n   \"_all\": {\r\n      \"primaries\": {\r\n         \"fielddata\": {\r\n            \"memory_size_in_bytes\": 2407666208,\r\n            \"evictions\": 642,\r\n            \"fields\": {\r\n               \"ts\": {\r\n                  \"memory_size_in_bytes\": 767245688\r\n               }\r\n            }\r\n         }\r\n      },\r\n      \"total\": {\r\n         \"fielddata\": {\r\n            \"memory_size_in_bytes\": 4994980234,\r\n            \"evictions\": 1250,\r\n            \"fields\": {\r\n               \"ts\": {\r\n                  \"memory_size_in_bytes\": 1532342416\r\n               }\r\n            }\r\n         }\r\n      }\r\n   },\r\n   \"indices\": {\r\n      \"delivery-logs-index.prod-20150126\": {\r\n         \"primaries\": {\r\n            \"fielddata\": {\r\n               \"memory_size_in_bytes\": 2407666208,\r\n               \"evictions\": 642,\r\n               \"fields\": {\r\n                  \"ts\": {\r\n                     \"memory_size_in_bytes\": 767245688\r\n                  }\r\n               }\r\n            }\r\n         },\r\n         \"total\": {\r\n            \"fielddata\": {\r\n               \"memory_size_in_bytes\": 4994980234,\r\n               \"evictions\": 1250,\r\n               \"fields\": {\r\n                  \"ts\": {\r\n                     \"memory_size_in_bytes\": 1532342416\r\n                  }\r\n               }\r\n            }\r\n         }\r\n      }\r\n   }\r\n}\r\n```\r\n\r\nShouldn't it be always `0`???\r\n\r\nAnother strange thing is that when I try the same fielddata query for indexes BEFORE upgrade:\r\n`GET /delivery-logs-index.prod-20150121/_stats/fielddata?fields=ts`\r\n I got this:\r\n```json\r\n{\r\n    \"_shards\": {\r\n        \"total\": 24,\r\n        \"successful\": 24,\r\n        \"failed\": 0\r\n    },\r\n    \"_all\": {\r\n        \"primaries\": {\r\n            \"fielddata\": {\r\n                \"memory_size_in_bytes\": 3001763154,\r\n                \"evictions\": 2655,\r\n                \"fields\": {}\r\n            }\r\n        },\r\n        \"total\": {\r\n            \"fielddata\": {\r\n                \"memory_size_in_bytes\": 5979477692,\r\n                \"evictions\": 4546,\r\n                \"fields\": {}\r\n            }\r\n        }\r\n    },\r\n    \"indices\": {\r\n        \"delivery-logs-index.prod-20150121\": {\r\n            \"primaries\": {\r\n                \"fielddata\": {\r\n                    \"memory_size_in_bytes\": 3001763154,\r\n                    \"evictions\": 2655,\r\n                    \"fields\": {}\r\n                }\r\n            },\r\n            \"total\": {\r\n                \"fielddata\": {\r\n                    \"memory_size_in_bytes\": 5979477692,\r\n                    \"evictions\": 4546,\r\n                    \"fields\": {}\r\n                }\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\nField `ts` is entirely missing(seems correct to me)!\r\n\r\nMapping for each daily index is used by a template which didn't change during upgrade.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/94438652","html_url":"https://github.com/elastic/elasticsearch/issues/9419#issuecomment-94438652","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9419","id":94438652,"user":{"login":"dimzak","id":4560173,"avatar_url":"https://avatars.githubusercontent.com/u/4560173?v=3","gravatar_id":"","url":"https://api.github.com/users/dimzak","html_url":"https://github.com/dimzak","followers_url":"https://api.github.com/users/dimzak/followers","following_url":"https://api.github.com/users/dimzak/following{/other_user}","gists_url":"https://api.github.com/users/dimzak/gists{/gist_id}","starred_url":"https://api.github.com/users/dimzak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dimzak/subscriptions","organizations_url":"https://api.github.com/users/dimzak/orgs","repos_url":"https://api.github.com/users/dimzak/repos","events_url":"https://api.github.com/users/dimzak/events{/privacy}","received_events_url":"https://api.github.com/users/dimzak/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T12:27:46Z","updated_at":"2015-04-20T12:27:46Z","body":"Hi @jpountz, sorry for the delay...\r\nMigration was 1.3.5 -> 1.3.7\r\n\r\nUnfortunately it was not a double mapping thing, but I managed to reproduce the exact error in an amazon environment( pure v1.3.7 without migration from 1.3.5).\r\n\r\nI believe the problem is that `ts` field exists also in nested documents(sorry if I forgot to mention it earlier)\r\n\r\nand so with the below mapping:\r\n```json\r\n\"mappings\": {\r\n   \"track-event\": {\r\n      \"dynamic\": \"strict\",\r\n      \"_all\": {\r\n         \"enabled\": false\r\n      },\r\n      \"_parent\": {\r\n         \"type\": \"ad-request\"\r\n      },\r\n      \"_routing\": {\r\n         \"required\": true\r\n      },\r\n      \"_timestamp\": {\r\n         \"enabled\": true,\r\n         \"store\": true\r\n      },\r\n      \"properties\": {\r\n         \"chid\": {\r\n            \"type\": \"string\",\r\n            \"index\": \"not_analyzed\"\r\n         },\r\n         \"event\": {\r\n            \"type\": \"string\",\r\n            \"index\": \"not_analyzed\"\r\n         },\r\n         \"ts\": {\r\n            \"type\": \"date\",\r\n            \"format\": \"dateOptionalTime\"\r\n         }\r\n      }\r\n   },\r\n   \"ad-request\": {\r\n      \"dynamic\": \"strict\",\r\n      \"_all\": {\r\n         \"enabled\": false\r\n      },\r\n      \"_id\": {\r\n         \"path\": \"rid\"\r\n      },\r\n      \"_timestamp\": {\r\n         \"enabled\": true,\r\n         \"store\": true\r\n      },\r\n      \"properties\": {\r\n         \"chid\": {\r\n            \"type\": \"string\",\r\n            \"index\": \"not_analyzed\"\r\n         }\r\n         \"track_events\": {\r\n            \"type\": \"nested\",\r\n            \"include_in_parent\": true,\r\n            \"dynamic\": \"strict\",\r\n            \"properties\": {\r\n               \"chid\": {\r\n                  \"type\": \"string\",\r\n                  \"index\": \"not_analyzed\"\r\n               },\r\n               \"event\": {\r\n                  \"type\": \"string\",\r\n                  \"index\": \"not_analyzed\"\r\n               },\r\n               \"ts\": {\r\n                  \"type\": \"date\",\r\n                  \"format\": \"dateOptionalTime\"\r\n               }\r\n            }\r\n         },\r\n         \"ts\": {\r\n            \"type\": \"date\",\r\n            \"doc_values\": true,\r\n            \"format\": \"dateOptionalTime\"\r\n         },\r\n\t\t \"event\": {\r\n                  \"type\": \"string\",\r\n                  \"index\": \"not_analyzed\",\r\n                  \"doc_values\": true\r\n         },\r\n         \"uid\": {\r\n            \"type\": \"string\",\r\n            \"index\": \"not_analyzed\",\r\n            \"doc_values\": true\r\n         }\r\n      }\r\n   }\r\n}\r\n```\r\n\r\nboth `ts` and `event` even though they are stored as `doc_values`, when searching `fielddata` result is:\r\n`GET /delivery-logs-index.prod-*/_stats/fielddata?fields=ts,uid,event`\r\n\r\nResult:\r\n\r\n```json\r\n{\r\n   \"_shards\": {\r\n      \"total\": 600,\r\n      \"successful\": 600,\r\n      \"failed\": 0\r\n   },\r\n   \"_all\": {\r\n      \"primaries\": {\r\n         \"fielddata\": {\r\n            \"memory_size_in_bytes\": 29238124,\r\n            \"evictions\": 0,\r\n            \"fields\": {\r\n               \"ts\": {\r\n                  \"memory_size_in_bytes\": 17727368\r\n               },\r\n               \"event\": {\r\n                  \"memory_size_in_bytes\": 4929652\r\n               }\r\n            }\r\n         }\r\n      },\r\n      \"total\": {\r\n         \"fielddata\": {\r\n            \"memory_size_in_bytes\": 52637682,\r\n            \"evictions\": 0,\r\n            \"fields\": {\r\n               \"ts\": {\r\n                  \"memory_size_in_bytes\": 35516768\r\n               },\r\n               \"event\": {\r\n                  \"memory_size_in_bytes\": 7405266\r\n               }\r\n            }\r\n         }\r\n      }\r\n   },\r\n```\r\n\r\nSo this has something to do with the nested object ? \r\n`uid` which is only available on `parent` seems to be working fine."}}