{"action":"created","issue":{"url":"https://api.github.com/repos/jeroenooms/curl/issues/51","labels_url":"https://api.github.com/repos/jeroenooms/curl/issues/51/labels{/name}","comments_url":"https://api.github.com/repos/jeroenooms/curl/issues/51/comments","events_url":"https://api.github.com/repos/jeroenooms/curl/issues/51/events","html_url":"https://github.com/jeroenooms/curl/issues/51","id":122783307,"number":51,"title":"Possible to implement asynchronous (i.e. non-blocking io) in curl?","user":{"login":"zachmayer","id":581590,"avatar_url":"https://avatars.githubusercontent.com/u/581590?v=3","gravatar_id":"","url":"https://api.github.com/users/zachmayer","html_url":"https://github.com/zachmayer","followers_url":"https://api.github.com/users/zachmayer/followers","following_url":"https://api.github.com/users/zachmayer/following{/other_user}","gists_url":"https://api.github.com/users/zachmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/zachmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zachmayer/subscriptions","organizations_url":"https://api.github.com/users/zachmayer/orgs","repos_url":"https://api.github.com/users/zachmayer/repos","events_url":"https://api.github.com/users/zachmayer/events{/privacy}","received_events_url":"https://api.github.com/users/zachmayer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-17T17:29:23Z","updated_at":"2015-12-17T17:29:52Z","closed_at":null,"body":"From https://github.com/hadley/httr/issues/271:\r\n>This is not an issue, more of a question that I wanted to pose to the community. If there is already a method for achieving this, I apologize for the repeat and would appreciate being pointed in the right direction.\r\n\r\n>I use httr extensively and often find myself in situations where the needed aggregation of data requires hundreds of thousands of REST calls. This becomes performance limiting in R because it seems like every http call made using httr is a blocking call.\r\n\r\n>Is there any plan or path forward for enabling asynchronous io within curl similar to what exists in Python via aiohttp or Scala via Akka-http?"},"comment":{"url":"https://api.github.com/repos/jeroenooms/curl/issues/comments/165522013","html_url":"https://github.com/jeroenooms/curl/issues/51#issuecomment-165522013","issue_url":"https://api.github.com/repos/jeroenooms/curl/issues/51","id":165522013,"user":{"login":"zachmayer","id":581590,"avatar_url":"https://avatars.githubusercontent.com/u/581590?v=3","gravatar_id":"","url":"https://api.github.com/users/zachmayer","html_url":"https://github.com/zachmayer","followers_url":"https://api.github.com/users/zachmayer/followers","following_url":"https://api.github.com/users/zachmayer/following{/other_user}","gists_url":"https://api.github.com/users/zachmayer/gists{/gist_id}","starred_url":"https://api.github.com/users/zachmayer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zachmayer/subscriptions","organizations_url":"https://api.github.com/users/zachmayer/orgs","repos_url":"https://api.github.com/users/zachmayer/repos","events_url":"https://api.github.com/users/zachmayer/events{/privacy}","received_events_url":"https://api.github.com/users/zachmayer/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T17:29:52Z","updated_at":"2015-12-17T17:29:52Z","body":"And a follow up:\r\n[Here's an example from Rcurl](http://www.omegahat.org/RCurl/concurrent.html):\r\n```{R}\r\ngetURIs =\r\nfunction(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)\r\n{\r\n  content = list()\r\n  curls = list()\r\n\r\n  for(i in uris) {\r\n    curl = getCurlHandle()\r\n    content[[i]] = basicTextGatherer()\r\n    opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)    \r\n    curlSetOpt(.opts = opts, curl = curl)\r\n    multiHandle = push(multiHandle, curl)\r\n  }\r\n\r\n  if(.perform) {\r\n     complete(multiHandle)\r\n     lapply(content, function(x) x$value())\r\n   } else {\r\n     return(list(multiHandle = multiHandle, content = content))\r\n   }\r\n}\r\n```\r\n\r\nThere is also [getURIAsynchronous](http://www.inside-r.org/packages/cran/RCurl/docs/getURIAsynchronous)\r\n\r\nHere's an example use case where this would be helpful.  I want to submit 1,000 requests to the server, and each request takes 10 minutes to process (the server has to lookup some data and do some math that takes a long time).  However, the sever can handle many thousands of simultaneous requests.  \r\n\r\nCurrently, I'm looping through something like this:\r\n```{R}\r\nrequests <- lapply(urls, POST, ...)\r\n```\r\nWhich blocks on each request and takes 1,000 x 10 minutes to complete.  It'd be really nice to be able to send each request off to the server, without blocking on the request being completed.  Then after they have all been submitted, we can block on collecting the results with a loop like this:\r\n\r\n```{R}\r\nrequests <- lapply(urls, POST, ..., async=TRUE)\r\nresults  <- lapply(results, httr::complete)\r\n```\r\n\r\nWhere `httr::complete` would be similar to [`RCurl::complete`](http://www.inside-r.org/packages/cran/RCurl/docs/complete).  The second example is nice because we can submit all the requests at once and let the server start processing them before blocking on gathering the results.  In theory, this loop would take ~10 minutes to complete, plus the overhead of the 2 `lapply` loops.\r\n"}}