{"action":"created","issue":{"url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/2364","labels_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/2364/labels{/name}","comments_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/2364/comments","events_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/2364/events","html_url":"https://github.com/ow2-proactive/scheduling/issues/2364","id":112825752,"number":2364,"title":"\"java.lang.OutOfMemoryError: Unable to create new native thread\" using many nodes per host","user":{"login":"lpellegr","id":128898,"avatar_url":"https://avatars.githubusercontent.com/u/128898?v=3","gravatar_id":"","url":"https://api.github.com/users/lpellegr","html_url":"https://github.com/lpellegr","followers_url":"https://api.github.com/users/lpellegr/followers","following_url":"https://api.github.com/users/lpellegr/following{/other_user}","gists_url":"https://api.github.com/users/lpellegr/gists{/gist_id}","starred_url":"https://api.github.com/users/lpellegr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lpellegr/subscriptions","organizations_url":"https://api.github.com/users/lpellegr/orgs","repos_url":"https://api.github.com/users/lpellegr/repos","events_url":"https://api.github.com/users/lpellegr/events{/privacy}","received_events_url":"https://api.github.com/users/lpellegr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ow2-proactive/scheduling/labels/type:task","name":"type:task","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/ow2-proactive/scheduling/milestones/38","html_url":"https://github.com/ow2-proactive/scheduling/milestones/7.X","labels_url":"https://api.github.com/repos/ow2-proactive/scheduling/milestones/38/labels","id":1326898,"number":38,"title":"7.X","description":"","creator":{"login":"bamedro","id":1952557,"avatar_url":"https://avatars.githubusercontent.com/u/1952557?v=3","gravatar_id":"","url":"https://api.github.com/users/bamedro","html_url":"https://github.com/bamedro","followers_url":"https://api.github.com/users/bamedro/followers","following_url":"https://api.github.com/users/bamedro/following{/other_user}","gists_url":"https://api.github.com/users/bamedro/gists{/gist_id}","starred_url":"https://api.github.com/users/bamedro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bamedro/subscriptions","organizations_url":"https://api.github.com/users/bamedro/orgs","repos_url":"https://api.github.com/users/bamedro/repos","events_url":"https://api.github.com/users/bamedro/events{/privacy}","received_events_url":"https://api.github.com/users/bamedro/received_events","type":"User","site_admin":false},"open_issues":117,"closed_issues":4,"state":"open","created_at":"2015-09-28T14:04:06Z","updated_at":"2015-10-27T16:10:16Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-10-22T15:15:32Z","updated_at":"2015-10-27T20:27:36Z","closed_at":null,"body":"While running benchmarks on EC2 it was noticed that having too many ProActive nodes running tasks may cause [java.lang.OutOfMemoryError: Unable to create new native thread](https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread) on nodes.\r\n\r\nEach machine running ProActive nodes had the following configuration:\r\n\r\n    - CPU with 36 cores\r\n    - 180 PA nodes\r\n    - Nodes deployed using property\r\n    - Property `-Dproactive.node.workers=180` for using single JVM feature\r\n\r\nWhen getting the error, each node was in the following state:\r\n\r\n    - Running 180 tasks\r\n    - Each task was a Java task\r\n    - Java task content is `Thread.sleep(600000)` (10 minutes)\r\n\r\nIt was possible to reproduce the issue with a single machine hosting _180_ PA nodes. However, when the issue occurs all machine resources are exhausted and it not even possible to connect to it in order to retrieve logs, analyze load, count number of processes, etc.:\r\n\r\n```\r\nlpellegr@oops:benchmarks-ec2$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ~/.ssh/activeeon-ec2.pem ec2-user@52.29.86.126\r\nWarning: Permanently added '52.29.86.126' (ECDSA) to the list of known hosts.\r\nLast login: Wed Oct 21 12:12:12 2015 from activeeon.pck.nerim.net\r\n-bash: fork: retry: Aucun processus enfant\r\n-bash: fork: retry: Aucun processus enfant\r\n...\r\n```\r\n\r\nThe issue is related to the number of threads which are created by nodes. It is important to notice that the exact same configuration as before using a sleep timeout of _25_ seconds for Java tasks prevents the issue to occur. \r\n\r\nThe previous observation let me think the issue could be related with _pings_ which are performed periodically (each 20 seconds). Although the single JVM feature spawns only one JVM for multiple PA nodes, then one ping is done for each PA node running on the host (more exactly each _TaskLauncher_ active object per node)\r\n\r\nBelow are some pointers to related classes:\r\n\r\n  - [NodePingThread](https://github.com/ow2-proactive/scheduling/blob/master/scheduler/scheduler-server/src/main/java/org/ow2/proactive/scheduler/core/NodePingThread.java#L25)\r\n  - [SchedulingService#pingTaskNode](https://github.com/ow2-proactive/scheduling/blob/master/scheduler/scheduler-server/src/main/java/org/ow2/proactive/scheduler/core/SchedulingService.java#L788)\r\n  -  [TaskLauncher#getProgress](https://github.com/ow2-proactive/scheduling/blob/master/scheduler/scheduler-node/src/main/java/org/ow2/proactive/scheduler/task/TaskLauncher.java#L369)\r\n\r\nRegarding the item _TaskLauncher#getProgress_, the method is defined with _@ImmediateService_ which implies that a new thread will be created for each invokation, which may explain threads exhaustion since there is no upper bound for immediate services.\r\n\r\nI think it could be interesting to investigate deeper the issue. One possibility is to increase the ping period to see if the issue comes from that. If it is the case, a solution or even general improvement could be to see if the scheduler behaviour could be changed in order to send a single ping request to an host even if it contains several ProActive nodes."},"comment":{"url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/comments/151634013","html_url":"https://github.com/ow2-proactive/scheduling/issues/2364#issuecomment-151634013","issue_url":"https://api.github.com/repos/ow2-proactive/scheduling/issues/2364","id":151634013,"user":{"login":"lpellegr","id":128898,"avatar_url":"https://avatars.githubusercontent.com/u/128898?v=3","gravatar_id":"","url":"https://api.github.com/users/lpellegr","html_url":"https://github.com/lpellegr","followers_url":"https://api.github.com/users/lpellegr/followers","following_url":"https://api.github.com/users/lpellegr/following{/other_user}","gists_url":"https://api.github.com/users/lpellegr/gists{/gist_id}","starred_url":"https://api.github.com/users/lpellegr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lpellegr/subscriptions","organizations_url":"https://api.github.com/users/lpellegr/orgs","repos_url":"https://api.github.com/users/lpellegr/repos","events_url":"https://api.github.com/users/lpellegr/events{/privacy}","received_events_url":"https://api.github.com/users/lpellegr/received_events","type":"User","site_admin":false},"created_at":"2015-10-27T20:27:36Z","updated_at":"2015-10-27T20:27:36Z","body":"@fviale yes, tasks are forked. I agree that deploying _180_ PA nodes is really weird but I think this is another problem. As you said, we should investigate if the issue is really related to `getProgress` or not.\r\n\r\n@tobias I will try to answer all your questions below:\r\n\r\n> Why do you think is the thread limit with a sleep 25 not exhausted? \r\n\r\nWe have run a benchmark with the exact same configuration as described before and with `sleep 25` without getting any error.\r\n\r\n> Is it possible, that not all task were started before the first sleep 25 returned?\r\n\r\nNo, PA nodes usage has grown up to 100%.\r\n\r\n> How is it possible to crash or not crash with the same amount of threads and same amount of memory used?\r\n\r\nThat's what I tried to explain before. Tasks running on PA nodes are monitored by calling `getProgress` each _20_ seconds. Since this remote call is a ProActive method call annotated with _@ImmediateService_, a new thread is created at each invocation. Although these calls are supposed _short-lived_, they may take time to complete if the machine is overloaded. Thus, if a `getProgress` invocation takes more than _20_ seconds to complete, new calls will spawn new threads, which may explain the limit that is reached."}}