{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/8871","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/8871/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/8871/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/8871/events","html_url":"https://github.com/elastic/elasticsearch/issues/8871","id":51553509,"number":8871,"title":"Move field mappings to the index level internally","user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Mapping","name":":Mapping","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/breaking","name":"breaking","color":"ff0000"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.0.0","name":"v2.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2014-12-10T12:38:34Z","updated_at":"2015-03-12T22:13:16Z","closed_at":null,"body":"Internally, field mappings should be stored at the index level, rather than at the type level.  Types are essentially a way of grouping fields. While the mapping APIs will not change, an exception will be thrown if an attempt is made to map a field in one type in a way that conflicts with the mapping of a field with same name in another type.\r\n\r\nThe `_parent` field is the one exception - this field will remain at the type level.\r\n\r\nRelates to #8870"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/78649842","html_url":"https://github.com/elastic/elasticsearch/issues/8871#issuecomment-78649842","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/8871","id":78649842,"user":{"login":"kiwigaffa","id":1020867,"avatar_url":"https://avatars.githubusercontent.com/u/1020867?v=3","gravatar_id":"","url":"https://api.github.com/users/kiwigaffa","html_url":"https://github.com/kiwigaffa","followers_url":"https://api.github.com/users/kiwigaffa/followers","following_url":"https://api.github.com/users/kiwigaffa/following{/other_user}","gists_url":"https://api.github.com/users/kiwigaffa/gists{/gist_id}","starred_url":"https://api.github.com/users/kiwigaffa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiwigaffa/subscriptions","organizations_url":"https://api.github.com/users/kiwigaffa/orgs","repos_url":"https://api.github.com/users/kiwigaffa/repos","events_url":"https://api.github.com/users/kiwigaffa/events{/privacy}","received_events_url":"https://api.github.com/users/kiwigaffa/received_events","type":"User","site_admin":false},"created_at":"2015-03-12T22:13:16Z","updated_at":"2015-03-12T22:13:16Z","body":"We have a lot of indices, each of which has a large set of mappings. While no two sets of mappings are identical, for most of our indices there is considerable overlap. Being able to share mappings between indices would help us reduce the total cluster state size.\r\n\r\nMore concretely, imagine that we have 10 indices (a-j) all being written to, and we cycle each of those  indices every hour (i.e. create a' to replace a, b' to replace b, and so on). If the 10 new indices (a'-j') could simply use the mappings from their predecessors, then we'd avoid the mappings storm as data is added to the empty indices.\r\n\r\nThis is possible today by simply setting the mappings when the new index is created, but every time that happens, we're increasing our total cluster state by the size of those mappings. We also have mappings \"drift\", where some percentage of all mappings are used in one index, but not in the next. Simply copying the mappings means we accumulate all of those \"dead\" mappings in each new index. If, instead, we could re-use an existing set of mappings, then we minimize total cluster state size. \"Drift\" is still an issue, but the cost is effectively zero, since those mappings are still required for the indices they were used in.\r\n\r\nI guess what I'm asking for is to move the mappings to one level higher than the index, and allow that higher level mapping to be shared by multiple indices."}}