{"action":"created","issue":{"url":"https://api.github.com/repos/voryx/Thruway/issues/135","labels_url":"https://api.github.com/repos/voryx/Thruway/issues/135/labels{/name}","comments_url":"https://api.github.com/repos/voryx/Thruway/issues/135/comments","events_url":"https://api.github.com/repos/voryx/Thruway/issues/135/events","html_url":"https://github.com/voryx/Thruway/issues/135","id":105814143,"number":135,"title":"MapReduce","user":{"login":"MikeSWelch","id":1283967,"avatar_url":"https://avatars.githubusercontent.com/u/1283967?v=3","gravatar_id":"","url":"https://api.github.com/users/MikeSWelch","html_url":"https://github.com/MikeSWelch","followers_url":"https://api.github.com/users/MikeSWelch/followers","following_url":"https://api.github.com/users/MikeSWelch/following{/other_user}","gists_url":"https://api.github.com/users/MikeSWelch/gists{/gist_id}","starred_url":"https://api.github.com/users/MikeSWelch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MikeSWelch/subscriptions","organizations_url":"https://api.github.com/users/MikeSWelch/orgs","repos_url":"https://api.github.com/users/MikeSWelch/repos","events_url":"https://api.github.com/users/MikeSWelch/events{/privacy}","received_events_url":"https://api.github.com/users/MikeSWelch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-09-10T14:09:50Z","updated_at":"2015-09-12T00:55:37Z","closed_at":null,"body":"I'm having trouble trying to implement MapReduce with Thruway.\r\n\r\nFrom a high level, I'm looking to issue 100 requests and I want 100 responses.\r\n\r\nI'm trying to create 100 clients on the same loop that each subscribe on their own topic.  When I start the loop, the clients will make a post to one of my web servers.  The web server then does the requested work, fires up a client and publishes the completed work on the same topic that the requesting client is listening on.\r\n\r\nWhen I do this with 100 clients, about 90 of the clients get the desired responses and the rest just hang.  I can see that 100 web requests were made, 100 jobs completed, and 100 new clients started up to publish their completed work.  However, about 10 of the requesting clients do not receive responses on their topic.\r\n\r\nWhen I run this with 10 clients, it works.  I have tested running it over 1000 times with 10 clients, and every single client gets a response every time.\r\n\r\nHow do I figure out why some clients are not receiving their responses?  Is there a TTL so I can set the clients to retry?  Is there a maximum amount of topics that a router can handle concurrently?  Is there a maximum amount of messages a router can handle conccurrently?  Is this a memory issue?"},"comment":{"url":"https://api.github.com/repos/voryx/Thruway/issues/comments/139693827","html_url":"https://github.com/voryx/Thruway/issues/135#issuecomment-139693827","issue_url":"https://api.github.com/repos/voryx/Thruway/issues/135","id":139693827,"user":{"login":"davidwdan","id":4969183,"avatar_url":"https://avatars.githubusercontent.com/u/4969183?v=3","gravatar_id":"","url":"https://api.github.com/users/davidwdan","html_url":"https://github.com/davidwdan","followers_url":"https://api.github.com/users/davidwdan/followers","following_url":"https://api.github.com/users/davidwdan/following{/other_user}","gists_url":"https://api.github.com/users/davidwdan/gists{/gist_id}","starred_url":"https://api.github.com/users/davidwdan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidwdan/subscriptions","organizations_url":"https://api.github.com/users/davidwdan/orgs","repos_url":"https://api.github.com/users/davidwdan/repos","events_url":"https://api.github.com/users/davidwdan/events{/privacy}","received_events_url":"https://api.github.com/users/davidwdan/received_events","type":"User","site_admin":false},"created_at":"2015-09-12T00:55:37Z","updated_at":"2015-09-12T00:55:37Z","body":"We played around with this for a while and tried to make it work the way you're doing it, but we ran into two problems:\r\n\r\n1.  When you flood the router with a lot of connections at once, you run into websocket backlog limits.  Once the limit is reach, the router returns a connection refused, which requires you to properly handle the reconnection attempt and clean up the subscription.\r\n2.  I tested apache and nginx and they both quickly hit connection limits as I tried to scale up, which also causes errors that need to be handled properly.\r\n\r\nI think a much better approach to this problem would be to use sub processes and RPC calls.  This way you remove the web server overhead and you have some response guarantees with RPC. \r\n\r\nI created a simple example that can handle over 1000 RPC calls.  To try it out, you'll want to make sure that you're using `libevent` and have the following packages installed:\r\n\r\n```JSON\r\n \"react/child-process\": \"0.4.*\",\r\n \"react/stream\": \"dev-master as 0.4.2\"\r\n```\r\n\r\nstart.php:\r\n```PHP\r\n<?php\r\nrequire_once __DIR__ . '/vendor/autoload.php';\r\n\r\n$startTime = microtime(true);\r\n$loop      = \\React\\EventLoop\\Factory::create();\r\n$processes = [];\r\n\r\n//Create and start 150 workers\r\nfor ($i = 0; $i < 150; $i++) {\r\n    $process = new React\\ChildProcess\\Process(PHP_BINARY . ' worker.php');\r\n    $process->on('exit', function ($exitCode, $termSignal) {\r\n        echo json_encode($exitCode) . \"\\n\";\r\n    });\r\n    $processes[] = $process;\r\n    $process->start($loop);\r\n}\r\n\r\n$client = new \\Thruway\\Peer\\Client('realm2', $loop);\r\n$client->addTransportProvider(new \\Thruway\\Transport\\PawlTransportProvider('ws://127.0.0.1:9090/'));\r\n$client->setAttemptRetry(false);\r\n\r\n$client->on('open', function (\\Thruway\\ClientSession $session) {\r\n    $promises = [];\r\n\r\n    for ($i = 0; $i < 1000; $i++) {\r\n        $promises[] = $session->call('do.some.work', [$i]);\r\n    }\r\n\r\n    \\React\\Promise\\all($promises)->then(\r\n      function ($results) use ($session) {\r\n          $session->close();\r\n          $session->getLoop()->stop();\r\n          echo \"result\" . json_encode($results) . \" \\n\" . count($results) . \"\\n\";\r\n      },\r\n      function ($error) use ($session) {\r\n          $session->close();\r\n          $session->getLoop()->stop();\r\n          echo \"ERROR\" . json_encode($error) . \"\\n\";\r\n      });\r\n});\r\n\r\n$client->start($loop);\r\n\r\n//End all sub processes\r\nforeach ($processes as $process) {\r\n    $process->terminate();\r\n}\r\n\r\necho microtime(true) - $startTime . \"\\n\";\r\n```\r\n\r\nworker.php (sub process):\r\n```PHP\r\n<?php\r\nuse Thruway\\Peer\\Client;\r\n\r\nrequire_once __DIR__ . '/vendor/autoload.php';\r\n\r\n$loop   = \\React\\EventLoop\\Factory::create();\r\n$client = new Client(\"realm2\", $loop);\r\n$client->addTransportProvider(new Thruway\\Transport\\PawlTransportProvider('ws://127.0.0.1:9090'));\r\n$client->on('open', function (Thruway\\ClientSession $session) {\r\n\r\n    $session->register('do.some.work',\r\n      function ($args) {\r\n          //Simulate blocking web request\r\n          sleep(1);\r\n\r\n          return $args[0];\r\n      },\r\n      //This option will tell the router to round robin the requests to all clients that have registered this RPC\r\n      [\"thruway_multiregister\" => true]);\r\n});\r\n\r\n//End this process if it's around for more than 15 seconds.  This is just a fail safe if the parent process doesn't exit properly.\r\n$loop->addTimer(15, function ($timer) {\r\n    $timer->cancel();\r\n});\r\n\r\n$client->start();\r\n```\r\n\r\nThen run:\r\n```bash\r\nphp start.php\r\n```\r\n\r\nLet me know if that works for what you're trying to do.\r\n\r\n\r\n "}}