{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/10318","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10318/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10318/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10318/events","html_url":"https://github.com/elastic/elasticsearch/issues/10318","id":65210200,"number":10318,"title":"Endless mapping refresh","user":{"login":"EikeDehling","id":11720247,"avatar_url":"https://avatars.githubusercontent.com/u/11720247?v=3","gravatar_id":"","url":"https://api.github.com/users/EikeDehling","html_url":"https://github.com/EikeDehling","followers_url":"https://api.github.com/users/EikeDehling/followers","following_url":"https://api.github.com/users/EikeDehling/following{/other_user}","gists_url":"https://api.github.com/users/EikeDehling/gists{/gist_id}","starred_url":"https://api.github.com/users/EikeDehling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EikeDehling/subscriptions","organizations_url":"https://api.github.com/users/EikeDehling/orgs","repos_url":"https://api.github.com/users/EikeDehling/repos","events_url":"https://api.github.com/users/EikeDehling/events{/privacy}","received_events_url":"https://api.github.com/users/EikeDehling/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-30T13:22:04Z","updated_at":"2015-03-31T19:27:29Z","closed_at":null,"body":"- We're running ES 1.4.2\r\n- Can sometimes be solved by closing/re-opening affected indices, but the issue usually returns prettys soon\r\n- This \"clogs\" up pending tasks and some tasks get stuck / cluster actions will now always timeout (e.g. loading/removing warmers)\r\n- This consumes all network capacity on the ES nodes\r\n\r\nhttps://gist.github.com/EikeDehling/a015a5137ac5d99dc850\r\n\r\n[buzzcapture@hermes ~]$ curl http://artemis3:9200/_cat/pending_tasks\r\n18149502    1s HIGH   refresh-mapping [postings-5360000000][[posting]] \r\n18149503 745ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149509 736ms HIGH   refresh-mapping [postings-5360000000][[posting]] \r\n18149504 737ms HIGH   refresh-mapping [postings-4180000000][[posting]] \r\n18149510 735ms HIGH   refresh-mapping [postings-5190000000][[posting]] \r\n18149512 735ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149506 736ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149505 736ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149511 735ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149515 732ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149519 290ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149521 289ms HIGH   refresh-mapping [postings-5190000000][[posting]] \r\n18149513 734ms HIGH   refresh-mapping [postings-5100000000][[posting]] \r\n18149525 287ms HIGH   refresh-mapping [postings-5100000000][[posting]] \r\n18149507 736ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149508 736ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149514 733ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149516 299ms HIGH   refresh-mapping [postings-4180000000][[posting]] \r\n18149517 298ms HIGH   refresh-mapping [postings-5360000000][[posting]] \r\n18149518 291ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n12674966  1.7d NORMAL master ping (from: [Y8LnaPqjTv-4Vn4CWXWWlQ])     \r\n18149520 290ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n12676681  1.7d NORMAL master ping (from: [Y8LnaPqjTv-4Vn4CWXWWlQ])     \r\n18149522 289ms HIGH   refresh-mapping [postings-5190000000][[posting]] \r\n18149523 288ms HIGH   refresh-mapping [postings-5100000000][[posting]] \r\n18149524 288ms HIGH   refresh-mapping [postings-4180000000][[posting]] \r\n12678378  1.7d NORMAL master ping (from: [Y8LnaPqjTv-4Vn4CWXWWlQ])     \r\n18149526 286ms HIGH   refresh-mapping [postings-5430000000][[posting]] \r\n18149527 286ms HIGH   refresh-mapping [postings-5190000000][[posting]] \r\n18149528 286ms HIGH   refresh-mapping [postings-4180000000][[posting]] \r\n18149529 286ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149530 284ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149531 284ms HIGH   refresh-mapping [postings-5360000000][[posting]] \r\n18149532 284ms HIGH   refresh-mapping [postings-5100000000][[posting]] \r\n18149533 284ms HIGH   refresh-mapping [postings-5500000000][[posting]] \r\n18149534 281ms HIGH   refresh-mapping [postings-5360000000][[posting]]"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/88218410","html_url":"https://github.com/elastic/elasticsearch/issues/10318#issuecomment-88218410","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10318","id":88218410,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2015-03-31T19:27:29Z","updated_at":"2015-03-31T19:27:29Z","body":"@EikeDehling thx for that. It's accurate. The problem lies in the way the field data settings are rendered:\r\n\r\nhttps://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/index/mapper/core/AbstractFieldMapper.java#L756\r\n\r\n```\r\nbuilder.field(\"fielddata\", (Map) fieldDataType.getSettings().getAsMap());\r\n```\r\n\r\nThe order of the keys in that map is arbitrary (practically). It may be different between master and nodes causing this endless loop.\r\n\r\nTo work around this, you can set `indices.cluster.send_refresh_mapping` to false (requires node restart). This will disable the sending of mapping refresh instructions. You must remember to remove this settings before you upgrade to the next ES version, which will have a fix for this."}}