{"action":"created","issue":{"url":"https://api.github.com/repos/commercialhaskell/stack/issues/644","labels_url":"https://api.github.com/repos/commercialhaskell/stack/issues/644/labels{/name}","comments_url":"https://api.github.com/repos/commercialhaskell/stack/issues/644/comments","events_url":"https://api.github.com/repos/commercialhaskell/stack/issues/644/events","html_url":"https://github.com/commercialhaskell/stack/issues/644","id":96276835,"number":644,"title":"Develop/Document multi-level parallelism policy","user":{"login":"rrnewton","id":274582,"avatar_url":"https://avatars.githubusercontent.com/u/274582?v=3","gravatar_id":"","url":"https://api.github.com/users/rrnewton","html_url":"https://github.com/rrnewton","followers_url":"https://api.github.com/users/rrnewton/followers","following_url":"https://api.github.com/users/rrnewton/following{/other_user}","gists_url":"https://api.github.com/users/rrnewton/gists{/gist_id}","starred_url":"https://api.github.com/users/rrnewton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rrnewton/subscriptions","organizations_url":"https://api.github.com/users/rrnewton/orgs","repos_url":"https://api.github.com/users/rrnewton/repos","events_url":"https://api.github.com/users/rrnewton/events{/privacy}","received_events_url":"https://api.github.com/users/rrnewton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/commercialhaskell/stack/milestones/1","html_url":"https://github.com/commercialhaskell/stack/milestones/Later%20improvements","labels_url":"https://api.github.com/repos/commercialhaskell/stack/milestones/1/labels","id":1111900,"number":1,"title":"Later improvements","description":"Arbitrary date chosen, just to get the milestones to order nicely","creator":{"login":"snoyberg","id":49415,"avatar_url":"https://avatars.githubusercontent.com/u/49415?v=3","gravatar_id":"","url":"https://api.github.com/users/snoyberg","html_url":"https://github.com/snoyberg","followers_url":"https://api.github.com/users/snoyberg/followers","following_url":"https://api.github.com/users/snoyberg/following{/other_user}","gists_url":"https://api.github.com/users/snoyberg/gists{/gist_id}","starred_url":"https://api.github.com/users/snoyberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snoyberg/subscriptions","organizations_url":"https://api.github.com/users/snoyberg/orgs","repos_url":"https://api.github.com/users/snoyberg/repos","events_url":"https://api.github.com/users/snoyberg/events{/privacy}","received_events_url":"https://api.github.com/users/snoyberg/received_events","type":"User","site_admin":false},"open_issues":24,"closed_issues":21,"state":"open","created_at":"2015-05-15T10:03:10Z","updated_at":"2015-07-22T13:15:44Z","due_on":"2016-01-31T08:00:00Z","closed_at":null},"comments":3,"created_at":"2015-07-21T09:59:47Z","updated_at":"2015-07-22T14:25:45Z","closed_at":null,"body":"I can see that `stack test` defaults to parallel builds.  But that refers to parallelism between different  test-suites, right?  In the case of both builds and tests (with test-framework or tasty) there's the issue of \"inner loop\" parallelism within each build or test target.  I suppose we can't parallelize the tests until we have some way for the build-tool to know it's a tasty/test-framework suite, not just `exitcode-stdio-1.0`, but that still leaves build-parallelism.  Does Stack currently pass `-j` to GHC?\r\n\r\nI haven't seen comprehensive benchmark numbers, but a small value like `-j2` or `-j4` for GHC offers some benefit.  The current implementation is not scalable however (for example, I haven't seen anything good come of `-j32`).\r\n\r\nNested parallelism of course raises the issue of `N * N` jobs being spawned for `N` cores.  As long as its quadratic oversubscription and not exponential, I think this is not that big a problem for CPU usage, but it very often can create problems with memory usage (or hitting ulimits / max # of processes).\r\n\r\nThere are several related cabal issues, but it's a little hard to tell the current status with them spanning a lot of time and various merged and unmerged pull requests:\r\n\r\n * https://github.com/haskell/cabal/issues/440 -- mentions how Gentoo uses an estimate of current load to gate creation of new jobs\r\n * https://github.com/haskell/cabal/issues/976\r\n * https://github.com/haskell/cabal/pull/1572 -- some talk of using OS semaphors to manage shared compute resources.  A draft of the [related code appears here](https://github.com/23Skidoo/cabal/blob/549c3797ac3ee7342ebf2c94c73c19c17b538e67/Cabal/Distribution/Compat/IPC.hs), in @23Skidoo's unmerged work.\r\n * https://github.com/haskell/cabal/issues/2623\r\n"},"comment":{"url":"https://api.github.com/repos/commercialhaskell/stack/issues/comments/123740003","html_url":"https://github.com/commercialhaskell/stack/issues/644#issuecomment-123740003","issue_url":"https://api.github.com/repos/commercialhaskell/stack/issues/644","id":123740003,"user":{"login":"rrnewton","id":274582,"avatar_url":"https://avatars.githubusercontent.com/u/274582?v=3","gravatar_id":"","url":"https://api.github.com/users/rrnewton","html_url":"https://github.com/rrnewton","followers_url":"https://api.github.com/users/rrnewton/followers","following_url":"https://api.github.com/users/rrnewton/following{/other_user}","gists_url":"https://api.github.com/users/rrnewton/gists{/gist_id}","starred_url":"https://api.github.com/users/rrnewton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rrnewton/subscriptions","organizations_url":"https://api.github.com/users/rrnewton/orgs","repos_url":"https://api.github.com/users/rrnewton/repos","events_url":"https://api.github.com/users/rrnewton/events{/privacy}","received_events_url":"https://api.github.com/users/rrnewton/received_events","type":"User","site_admin":false},"created_at":"2015-07-22T14:25:45Z","updated_at":"2015-07-22T14:25:45Z","body":"Given that neither stack nor cabal nor GHC has a notion of global resource management on the machine, I guess in the medium term what I'd like is enough knobs to experiment with this.\r\n\r\nFor example, it would be great to be able to \"turn it up\" to max parallelism -- parallel packages, parallel targets within a package, parallel modules within `ghc --make` plus parallel tests.  And then do some benchmarking to see what speedups and memory usage look like.\r\n\r\nWe can already vary things pretty easily by generating `.cabal`/`.stack` files that pass the right arguments through to GHC and to the test-suites.  I guess the critical thing on which to get help from `stack` itself is the third bullet @snoyberg mentioned -- multiple test-suites (plus profiling/non-profiling/documentation) in parallel within one package, which corresponds to haskell/cabal#2623.\r\n\r\nBy the way, as far as I know, no one's properly analyzed GHC builds from a parallel algorithms perspective.  I.e. we need to profile the dependence graph and do a work/span analysis to figure out what is limiting our scaling.  (We're working on a research project where we may be hacking on fine-grain parallelism inside certain GHC phases, but it only makes sense if there's a coherent picture for parallelism at the coarser grain too.)\r\n\r\nIn lieu of a GHC server mode (which has been discussed on various issues), we can't directly implement an inter-process work-stealing type policy that mimics the now-standard intra-process load balancing approach.  But we can do what that Gentoo builder seems to be doing and simply delay some tasks to manage resource use.  The more look-ahead or previous knowledge we have about the task DAG the smarter we can be in prioritizing the critical path.\r\n\r\n"}}