{"action":"created","issue":{"url":"https://api.github.com/repos/ruflin/Elastica/issues/830","labels_url":"https://api.github.com/repos/ruflin/Elastica/issues/830/labels{/name}","comments_url":"https://api.github.com/repos/ruflin/Elastica/issues/830/comments","events_url":"https://api.github.com/repos/ruflin/Elastica/issues/830/events","html_url":"https://github.com/ruflin/Elastica/issues/830","id":74169430,"number":830,"title":"Util method - is this an issue or am i missing something?","user":{"login":"virtuman","id":1087996,"avatar_url":"https://avatars.githubusercontent.com/u/1087996?v=3","gravatar_id":"","url":"https://api.github.com/users/virtuman","html_url":"https://github.com/virtuman","followers_url":"https://api.github.com/users/virtuman/followers","following_url":"https://api.github.com/users/virtuman/following{/other_user}","gists_url":"https://api.github.com/users/virtuman/gists{/gist_id}","starred_url":"https://api.github.com/users/virtuman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/virtuman/subscriptions","organizations_url":"https://api.github.com/users/virtuman/orgs","repos_url":"https://api.github.com/users/virtuman/repos","events_url":"https://api.github.com/users/virtuman/events{/privacy}","received_events_url":"https://api.github.com/users/virtuman/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-05-08T00:02:01Z","updated_at":"2015-05-11T08:39:03Z","closed_at":"2015-05-11T06:36:26Z","body":"Here's a method in Util.php:\r\n```\r\n    public static function reindex(Client $client, $newIndex, $oldIndex, $expiryTime = '1m', $sizePerShard = 1000)\r\n    {\r\n        $search = new Search($client);\r\n        $oldIndex = new Index($client, $oldIndex);\r\n        $newIndex = new Index($client, $newIndex);\r\n        \r\n        $bulk = new Bulk($client);\r\n        $bulk->setIndex($newIndex);\r\n                \r\n        $search->addIndex($oldIndex);\r\n        $search->setOption(Search::OPTION_SEARCH_TYPE, Search::OPTION_SEARCH_TYPE_SCAN);\r\n        $scanAndScroll = new ScanAndScroll($search, $expiryTime, $sizePerShard);\r\n        foreach ($scanAndScroll as $resultSet) {\r\n            $data = $resultSet->getResponse()->getData();\r\n            foreach ($data['hits']['hits'] as $d) {\r\n                $meta = array('_index' => $newIndex->getName() , '_type' => $d['_type'], '_id' => $d['_id']);\r\n                $actions[] = new Action(Action::OP_TYPE_INDEX, $meta ,$d['_source']);\r\n            }\r\n            //$actions = new Bulk\\Action();\r\n            $bulk->addActions($actions);\r\n            $bulk->send();\r\n        }\r\n        $newIndex->refresh();\r\n        return \"reindex complete\";\r\n    }\r\n```\r\n\r\n\r\nLines of interest here are:\r\n```\r\n        foreach ($scanAndScroll as $resultSet) {\r\n            $data = $resultSet->getResponse()->getData();\r\n            foreach ($data['hits']['hits'] as $d) {\r\n                $meta = array('_index' => $newIndex->getName() , '_type' => $d['_type'], '_id' => $d['_id']);\r\n                $actions[] = new Action(Action::OP_TYPE_INDEX, $meta ,$d['_source']);\r\n            }\r\n            //$actions = new Bulk\\Action();\r\n            $bulk->addActions($actions);\r\n            $bulk->send();\r\n        }\r\n```\r\n\r\n$actions container here seems to be constantly appended to during all scanandscroll + hits iteration.\r\nWouldn't this result in results doubling up on each iteration, ie:\r\n1. scanAndScroll is set to select 10 entries from each of 5 shards\r\n2. during first iteration $actions will become container for 50 docs\r\n3. addActions and send() will push these products to newIndex\r\n4. during second iteration $actions will get another 50 added to it and will now become a container for 100 docs\r\n5. addActions and send() will now push all 100 products\r\n6. ... same repeats until the end resulting in $actions to hold entire dataset that was being selected by scanAndScroll to avoid out of memory issues\r\n\r\ngranted it's not an issue with documents doubling up because _id is set explicitly on each iteration, but to my understanding $actions should be reset within scanAndScroll foreach loop?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ruflin/Elastica/issues/comments/100816740","html_url":"https://github.com/ruflin/Elastica/issues/830#issuecomment-100816740","issue_url":"https://api.github.com/repos/ruflin/Elastica/issues/830","id":100816740,"user":{"login":"webdevsHub","id":1021820,"avatar_url":"https://avatars.githubusercontent.com/u/1021820?v=3","gravatar_id":"","url":"https://api.github.com/users/webdevsHub","html_url":"https://github.com/webdevsHub","followers_url":"https://api.github.com/users/webdevsHub/followers","following_url":"https://api.github.com/users/webdevsHub/following{/other_user}","gists_url":"https://api.github.com/users/webdevsHub/gists{/gist_id}","starred_url":"https://api.github.com/users/webdevsHub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/webdevsHub/subscriptions","organizations_url":"https://api.github.com/users/webdevsHub/orgs","repos_url":"https://api.github.com/users/webdevsHub/repos","events_url":"https://api.github.com/users/webdevsHub/events{/privacy}","received_events_url":"https://api.github.com/users/webdevsHub/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T08:39:03Z","updated_at":"2015-05-11T08:39:03Z","body":"> old index first and then the new index\r\n\r\n:+1:\r\n\r\n> Is this really reindex? Isn't this copy?\r\n\r\nThe current implementation is more copy then reindex. Maybe rename that function to `copy` and create a new `reindex` function that checks and creates the mapping in new index before copying the data."}}