{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/462","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/462/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/462/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/462/events","html_url":"https://github.com/flux-framework/flux-core/issues/462","id":113878685,"number":462,"title":"Super-linear memory expansion launching jobs","user":{"login":"trws","id":660149,"avatar_url":"https://avatars.githubusercontent.com/u/660149?v=3","gravatar_id":"","url":"https://api.github.com/users/trws","html_url":"https://github.com/trws","followers_url":"https://api.github.com/users/trws/followers","following_url":"https://api.github.com/users/trws/following{/other_user}","gists_url":"https://api.github.com/users/trws/gists{/gist_id}","starred_url":"https://api.github.com/users/trws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trws/subscriptions","organizations_url":"https://api.github.com/users/trws/orgs","repos_url":"https://api.github.com/users/trws/repos","events_url":"https://api.github.com/users/trws/events{/privacy}","received_events_url":"https://api.github.com/users/trws/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-10-28T17:42:02Z","updated_at":"2015-10-29T23:18:02Z","closed_at":null,"body":"Running a test for scalability in terms of number of jobs over time, I ran into super-linear memory consumption on broker 0.  I do not have data for the other brokers, but the broker 0 memory consumption proceeds as below, expressed as a percentage of total node memory for a Hype node, which I think is 32GB.  This test was run on four nodes, one broker per node, launching 4-rank MPI_hello jobs in groups of 16.  It was run with this command, (note the lack of --pty, that causes other oddness, also the extra quotes are necessary, the broker eats a level of quotes for some reason, the parallel xargs is just there so this takes 10 minutes instead of an hour due to loading each one sequentially):\r\n\r\n```bash\r\n$ srun -N 4 -n 4 -ppdebug flux broker sh -c \"'seq 1 6000 | xargs -n 1 -P 16 flux wreckrun -n 4 ./mpi_hello'\"\r\n```\r\n\r\n| jobs complete | percent memory |\r\n|--------------------|------------------------|\r\n|1000| 5%|\r\n|2000| 20%|\r\n|3000| 42%|\r\n|4000| 75%|\r\n|4528| OOM|\r\n\r\nI'm working on getting a heap profile for this, but it's proving to be surprisingly difficult to do so in a multi-node configuration like this.  I'll report back if I succeed in getting more useful data."},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/152354332","html_url":"https://github.com/flux-framework/flux-core/issues/462#issuecomment-152354332","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/462","id":152354332,"user":{"login":"trws","id":660149,"avatar_url":"https://avatars.githubusercontent.com/u/660149?v=3","gravatar_id":"","url":"https://api.github.com/users/trws","html_url":"https://github.com/trws","followers_url":"https://api.github.com/users/trws/followers","following_url":"https://api.github.com/users/trws/following{/other_user}","gists_url":"https://api.github.com/users/trws/gists{/gist_id}","starred_url":"https://api.github.com/users/trws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trws/subscriptions","organizations_url":"https://api.github.com/users/trws/orgs","repos_url":"https://api.github.com/users/trws/repos","events_url":"https://api.github.com/users/trws/events{/privacy}","received_events_url":"https://api.github.com/users/trws/received_events","type":"User","site_admin":false},"created_at":"2015-10-29T23:18:02Z","updated_at":"2015-10-29T23:18:02Z","body":"Good idea @garlick, I'll add that command to my test runner and post what comes out."}}