{"action":"opened","issue":{"url":"https://api.github.com/repos/mblakele/taskbot/issues/3","labels_url":"https://api.github.com/repos/mblakele/taskbot/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/mblakele/taskbot/issues/3/comments","events_url":"https://api.github.com/repos/mblakele/taskbot/issues/3/events","html_url":"https://github.com/mblakele/taskbot/issues/3","id":123662866,"number":3,"title":"Issue with deletion of documents.","user":{"login":"peetkes","id":550788,"avatar_url":"https://avatars.githubusercontent.com/u/550788?v=3","gravatar_id":"","url":"https://api.github.com/users/peetkes","html_url":"https://github.com/peetkes","followers_url":"https://api.github.com/users/peetkes/followers","following_url":"https://api.github.com/users/peetkes/following{/other_user}","gists_url":"https://api.github.com/users/peetkes/gists{/gist_id}","starred_url":"https://api.github.com/users/peetkes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peetkes/subscriptions","organizations_url":"https://api.github.com/users/peetkes/orgs","repos_url":"https://api.github.com/users/peetkes/repos","events_url":"https://api.github.com/users/peetkes/events{/privacy}","received_events_url":"https://api.github.com/users/peetkes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-23T13:47:40Z","updated_at":"2015-12-23T13:47:40Z","closed_at":null,"body":"Hi,\r\n\r\nI am testrunning the taskbot to be able to delete documents wit taskbot.\r\nI used the sample from the docs to create 1 million docs. I can see this filling up the tskqueue.\r\n\r\nInsert documents:\r\n\r\nxquery version \"1.0-ml\";\r\n\r\nimport module namespace tb = \"ns://blakeley.com/taskbot\" at \"/taskbot.xqy\";\r\n\r\n(: This will create 1 million docs in block sizes of 500, so it will create 2000 tasks on the taskserver queue :)\r\ntb:list-segment-process(\r\n  (: Total size of the job. :)\r\n  1 to 1000 * 1000,\r\n  (: Size of each segment of work. :)\r\n  500,\r\n  \"/test/asset\",\r\n  (: This anonymous function will be called for each segment. :)\r\n  function($list as item()+, $opts as map:map?) {\r\n    (: Any chainsaw should have a safety. Check it here. :)\r\n    tb:maybe-fatal(),\r\n    for $i in $list\r\n    return xdmp:document-insert(\r\n      \"/test/asset/\"||$i,\r\n      element asset {\r\n        attribute id { 'asset'||$i },\r\n        element asset-org { 1 + xdmp:random(99) },\r\n        element asset-person { 1 + xdmp:random(999) },\r\n        (1 to xdmp:random(9))\r\n        ! element asset-ref { xdmp:random(1000) } }),\r\n      xdmp:commit()\r\n    },\r\n  (: options - not used in this example. :)\r\n  map:new(map:entry('testing', '123...')),\r\n  (: This is an update, so be sure to say so. :)\r\n  <options xmlns=\"xdmp:eval\">\r\n    <transaction-mode>update</transaction-mode>\r\n  </options>\r\n)\r\n\r\nDelete documents:\r\nxquery version \"1.0-ml\";\r\nimport module namespace tb = \"ns://blakeley.com/taskbot\" at \"/taskbot.xqy\";\r\nlet $uris := cts:uris()\r\n    tb:list-segment-process(\r\n      (: Total size of the job. :)\r\n      $uris,\r\n      (: Size of each segment of work. :)\r\n      500,\r\n      \"Delete Documents\",\r\n      (: This anonymous function will be called for each segment. :)\r\n      function($list as item()+, $opts as map:map?) {\r\n        (: Any chainsaw should have a safety. Check it here. :)\r\n        tb:maybe-fatal(),\r\n        for $uri in $list\r\n        return xdmp:document-delete($uri),\r\n        xdmp:sleep(500),\r\n        xdmp:commit()\r\n      },\r\n      (: options - not used in this example. :)\r\n      map:new(map:entry('testing', '123...')),\r\n      (: This is an update, so be sure to say so. :)\r\n      $tb:OPTIONS-UPDATE\r\n    )\r\n\r\nWhen I execute the deletion, I can see about half of the documents get deleted!!\r\nI also get errors stating \r\n2015-12-23 12:55:16.183 Notice: TaskServer: XDMP-AS: (err:XPTY0004) $list as item()+ -- Invalid coercion: () as item()+\r\n2015-12-23 12:55:16.183 Notice: TaskServer: in /taskbot.xqy, at 294:37,\r\n2015-12-23 12:55:16.183 Notice: TaskServer: in function() as item()*() [1.0-ml]\r\n2015-12-23 12:55:16.183 Notice: TaskServer: in /taskbot.xqy [1.0-ml]\r\n2015-12-23 12:55:16.279 Info: App-Services: ()\r\n\r\nThis would mean that during  execution the list of uris will shrink according to the nr of documents that get deleted. Even surrounding the cts:uris() call with cts:eager will not resolve this.\r\n\r\nI got this resolved with the help of Geert Josten who suggested the following:\r\nwrap the $uris (first parameter of the list-segment-process) in json:array-values(json:to-array($uris))\r\n\r\nI also noted the following when I tried the $OPTIONS-SYNC-UPDATE parameter.\r\nWhen I use the $OPTIONS-SYNC-UPDATE parameter, the taskservers queue will never fill up like it does when I tried this with OPTIONS-UPDATE. It even does not make use of all available threads on the taskserver.\r\n\r\nWhy is that?\r\n"}}