{"action":"created","issue":{"url":"https://api.github.com/repos/catkin/catkin_tools/issues/196","labels_url":"https://api.github.com/repos/catkin/catkin_tools/issues/196/labels{/name}","comments_url":"https://api.github.com/repos/catkin/catkin_tools/issues/196/comments","events_url":"https://api.github.com/repos/catkin/catkin_tools/issues/196/events","html_url":"https://github.com/catkin/catkin_tools/pull/196","id":68088092,"number":196,"title":"[major] executor/clean: Adding per-package cleaning, linked develspaces, and a new execution pipeline","user":{"login":"jbohren","id":1253450,"avatar_url":"https://avatars.githubusercontent.com/u/1253450?v=3","gravatar_id":"","url":"https://api.github.com/users/jbohren","html_url":"https://github.com/jbohren","followers_url":"https://api.github.com/users/jbohren/followers","following_url":"https://api.github.com/users/jbohren/following{/other_user}","gists_url":"https://api.github.com/users/jbohren/gists{/gist_id}","starred_url":"https://api.github.com/users/jbohren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbohren/subscriptions","organizations_url":"https://api.github.com/users/jbohren/orgs","repos_url":"https://api.github.com/users/jbohren/repos","events_url":"https://api.github.com/users/jbohren/events{/privacy}","received_events_url":"https://api.github.com/users/jbohren/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/catkin/catkin_tools/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/catkin/catkin_tools/milestones/3","html_url":"https://github.com/catkin/catkin_tools/milestones/0.4.0%20-%20Second%20Beta%20Announcement","labels_url":"https://api.github.com/repos/catkin/catkin_tools/milestones/3/labels","id":1161028,"number":3,"title":"0.4.0 - Second Beta Announcement","description":"This release will be included the second catkin_tools beta announcement.","creator":{"login":"jbohren","id":1253450,"avatar_url":"https://avatars.githubusercontent.com/u/1253450?v=3","gravatar_id":"","url":"https://api.github.com/users/jbohren","html_url":"https://github.com/jbohren","followers_url":"https://api.github.com/users/jbohren/followers","following_url":"https://api.github.com/users/jbohren/following{/other_user}","gists_url":"https://api.github.com/users/jbohren/gists{/gist_id}","starred_url":"https://api.github.com/users/jbohren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbohren/subscriptions","organizations_url":"https://api.github.com/users/jbohren/orgs","repos_url":"https://api.github.com/users/jbohren/repos","events_url":"https://api.github.com/users/jbohren/events{/privacy}","received_events_url":"https://api.github.com/users/jbohren/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":2,"state":"open","created_at":"2015-06-11T21:08:49Z","updated_at":"2015-06-16T20:55:01Z","due_on":"2015-07-15T04:00:00Z","closed_at":null},"comments":39,"created_at":"2015-04-13T12:43:28Z","updated_at":"2015-10-20T19:10:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/catkin/catkin_tools/pulls/196","html_url":"https://github.com/catkin/catkin_tools/pull/196","diff_url":"https://github.com/catkin/catkin_tools/pull/196.diff","patch_url":"https://github.com/catkin/catkin_tools/pull/196.patch"},"body":"## Overview\r\n\r\nThis PR involves a dramatic refactor of the core execution pipeline for building and cleaning packages in order to implement two high-priority features:\r\n\r\n* Per-package / partial-workspace cleaning (reversible build actions)\r\n  * Requires build \"linked\" develspaces, which are isolated per-package develspaces whose contents are symbolically-linked into a merged develspace\r\n  * Requires using the `catkin build` execution pipeline outside of the `build` verb\r\n* (#1) Automatic `stderr`-based warning and error detection\r\n  * Requires separating `stdout` from `stderr` during execution of each job\r\n\r\nAdditionally, addresses the following outstanding issues:\r\n\r\n* (#2) More general use of the GNU-Make compatible job server in the new execution pipeline\r\n* (#78) The linked develspace construction avoids potential race conditions in the generation of the .catkin file\r\n* (#28) (#45) (#65) The linked develspace provides the benefits of isolated devel spaces without the runtime complexity of generating (#28) and maintaining (#45) setup files, and partial cleaning (#65)\r\n* (#188) By default, the `-p` option is set to whatever the `-j` option is\r\n* (#59) The `catkin` tool should always exit cleanly when ctrl-c'd due to the new use of asyncio\r\n* (#87) Supports workspaces containing only vanilla `cmake` packages\r\n* (#63) Provides a brief build summary with important notices at the end of the build\r\n* (#66) The pending jobs are now calculated in a completely different way and are unlikely to deadlock \r\n* (#213) Running integration tests on Linux and OS X\r\n* (#208) Fixes jobserver support on Python3\r\n\r\nIt also provides the following:\r\n* More transparency into what the `catkin_tools` executor is doing via named job \"stages\"\r\n* Enables stages of jobs which are implemented in pure python and don't require subprocesses\r\n* Clearer designation of errors and warnings when they are printed to the console\r\n* In order to re-generate warnings a `--pre-clean` option has been added to the `build` verb to `make clean` in the build directory before running `make`\r\n* In order to build unbuilt packages, an `--unbuilt` option has been added to the `build` verb to build any packages in the workspace which have not been built\r\n* More introspection into the build status by displaying cmake package build percentages in status line\r\n\r\nIt also resulted in some unexpected benefits:\r\n* Build times can be sped up dramatically for no-op packages since the env.sh files aren't needed for merged or linked develspaces, for example:\r\n * A workspace with 188 pre-built packages completes in ~17s instead of ~55s (3.2x speedup)\r\n\r\nIt also revealed the following problems:\r\n* (#208) The jobserver doesn't appear to work with Python3.4\r\n\r\n## Per-Package Cleaning and the Linked Devel Space\r\n\r\nThis PR provides support for cleaning individual packages with the following syntax:\r\n\r\n```\r\ncatkin clean [-a | -b | -d | -i] [-f] [--orphans] [--deps] [--deinit] [[PKG] PKG ...]\r\n```\r\n\r\nWhere `-b` means buildspace, `-d` means develspace, `-i` means installspace, and `-a` means all three. Other examples include:\r\n\r\n### `catkin clean` Examples\r\n\r\n* `catkin clean` / `catkin clean -a`\r\n * Clean the entire build, devel, and install spaces (currently supported)\r\n* `catkin clean -af` / `catkin clean --all --force`\r\n * Clean the entire build, devel, and install spaces without asking for verififcation\r\n* `catkin clean -b`\r\n * Clean just the build space (currently supported)\r\n* `catkin clean pkg_a pkg_b`\r\n * Clean `pkg_a` and `pkg_b` from the build, devel, and install spaces\r\n* `catkin clean --deps pkg_a`\r\n * Clean `pkg_a` as well as any packages which depend on `pkg_a`.\r\n* `catkin clean --orphans`\r\n * Clean all packages which have been built but are no longer in the source space.\r\n* `catkin clean --deinit`\r\n * De-initialize workspace (delete `.catkin_tools`) directory\r\n\r\n### Approach\r\n\r\nDoing this right is hard, but it's an important feature. I think this sort of per-package `clean` functionality is one of the the final big features preventing `catkin_tools` from being complete. In complex workspaces with hundreds of packages, debugging can be a huge challenge, and this is only made more challenging by the inability to reliably remove all of the products of a specific packages from a workspace.\r\n\r\n#### CMake Packages\r\nVanilla CMake packages are handled trivially since their `install` target is used to write files to `devel` and `install` spaces. This target creates an `install_manifest.txt` which lists all the generated files, so to clean CMake packages, we can simply remove these files.\r\n\r\nThe normal location for `install_manifest.txt`, however, is in the package's build space. This file could get overwritten if the user builds both a devel and install variant, so we copy this file into the appropriate result space for safe keeping. Also, the `install_manifest.txt` file does not specify directories which are created by `cmake`, so the CMake clean job will remove any directories which are made empty through the removal of files listed in the `install_manifest.txt` file.\r\n\r\n#### Catkin Packages\r\nCatkin packages prove more challenging for a few reasons. For Catkin packages, there are two different steps which generate files in the `devel` space: the CMake configure step, and the Make build step. It's possible to clean the files generated in the build step with the standard `clean` target, but this still leaves numerous files generated by the configure step. Catkin packages also each generate some of the same files such as the setup shell scripts and each write to `.catkin` file which lists all the source packages in the workspace. This all makes for a big mess to clean up.\r\n\r\nThe approach used in this PR is to build each package into an isolated devel space, and then symbolically link the contents into the merged devel space. This prototype behavior is enabled by passing the `--link-devel` option to `catkin build`. Since `catkin_tools` does the linking, it can generate a `devel_manifest.txt` similar to CMake's `install_manifest.txt` which logs all of the files that a given package contributes to the devel space. \r\n\r\nIf a file that was listed in the previous link step is no longer found in the isolated devel space, it is removed from the merged devel space.\r\n\r\nThe above handles tracking the files generated by each package, but doesn't handle collisions very well. In order to handle collisions, the symlink generation step counts the number of times a collision occurs for each file. This information is stored in a file called `devel_collisions.txt`. \r\n\r\nIn summary:\r\n\r\n* The isolated devel spaces are: `DEVEL_PREFIX/.catkin_tools/PKG_NAME/linked_devel`\r\n* The devel manifests are stored in: `DEVEL_PREFIX/.catkin_tools/PKG_NAME/devel_manifest.txt`\r\n* The collisions file is stored in: `DEVEL_PREFIX/.catkin_tools/devel_collisions.txt`\r\n* All files from each isolated devel space are linked into: `DEVEL_PREFIX/`\r\n\r\nThe clean step for a given package reads its `devel_manifest.txt` and for each file that was generated it either removes the file or decrements the count in the `devel_collisions.txt` file.\r\n\r\n#### Setup File Generation\r\nOne challenge with this approach, as mentioned above, is that catkin packages normally generate several files in the root of the devel space used for \"sourcing\" that devel space. These include:\r\n\r\n* `.catkin`\r\n* `.rosinstall`\r\n* `_setup_util.py`\r\n* `env.sh`\r\n* `setup.sh`\r\n* `setup.bash`\r\n* `setup.zsh`\r\n\r\nWith the exclusion of `.catkin`, these files are all generated from [catkin_generate_environment.cmake](https://github.com/ros/catkin/blob/indigo-devel/cmake/catkin_generate_environment.cmake) during the CMake configure step of a catkin package, and several of them include information computed by existing Catkin CMake code. Since some of these reference the absolute path of the develspace, they can't simply be copied from the isolated devel spaces of the constituent packages.\r\n\r\nThe simplest solution to this problem is to have a \"ghost\" bootstrap package called `catkin_tools_bootstrap` which gets generated the first time a given resultspace is built. This guarantees that the setup files are generated correctly by Catkin, itself.\r\n\r\nA convenient side-effect of the `catkin_tools_bootstrinstallap` package is that it also enables `catkin build` to generate setup files for a develspace consisting only of vanilla CMake packages.\r\n\r\nFor the `.catkin` file, we make `catkin_tools` responsible for managing its contents. This is for two reasons: \r\n 1. There have been observed problems building Catkin projects in parallel and writing to this file\r\n 2. If we clean a package, we want to remove its source space from this file\r\n\r\nSince `catkin_tools` will be modifying this file in parallel from a single process, we can manage exclusive access to it and prevent collisions. Additionally, we can remove packages from it when they are removed from the workspace.\r\n\r\n#### Additional Notes\r\nThe current implementation works in Unix-based systems, but could be extended to Windows with [one of these methods](http://stackoverflow.com/questions/1447575/symlinks-on-windows).\r\n\r\n## Architectural Changes\r\n\r\n### Executor\r\n\r\nThis PR completely re-implements the execution pipeline using `osrf_pycommon` and the `trollius` library (a Python2 and Python3 compatible version of Python3.4's `asyncio`. This *yields* an execution pipeline which can be used by `catkin clean` (and other potential verbs) and enables us to extract and display `stdout` and `stderr` separately. As such, it solves #1.\r\n\r\n#### Execution Jobs and Stages\r\nThe new execution pipeline operates on a task model where a given \"task\" (build, clean, etc) can be decomposed into an acyclic \"job\" dependency graph, where each job is a sequence of \"stages\". Each job can be executed in parallel once its dependencies have been completed, and each stage within a job must be executed serially.\r\n\r\nA [`Job`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/execution/jobs.py#L20-L40) is a simple container with a few non-mutating utility functions. It is designated with a unique Job ID (`jid`) and has a list of the `jid`s of the jobs on which it depends. In `catkin_build`, these `jid`s are normally package names.\r\n\r\nA `Stage` is some atomic operation, which contains a non-unqie label (such as \"cmake\", \"make\", or \"install\"). There are currently two types of stages: [`CmdStage`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/execution/stages.py#L20-L56) and [`FunStage`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/execution/stages.py#L59-L78). The `CmdStage` describes a system command. This type is similar to the current stage implementation, except it also facilitates using asynchronous protocols for capturing `stdout` and `stderr`. The `FunStage`, on the other hand, describes a blocking Python function. This second stage type enables us to affect the filesystem in arbitrary ways without spawning subprocesses or relying on CMake's shell utilities.\r\n\r\nThe construction of `Job` and `Stage` instances should have no side-effects, and are designed to be lazily evaluated only once their dependencies have been completed. Once a collection of `Job` objects have been created, they are passed directly to the executor.\r\n\r\n#### Executing Jobs Asynchronously\r\n\r\nThe [executor](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/execution/executor.py#L108-L112) is an `asyncio` coroutine which executes each job asynchronously. Jobs follow a well-defined lifecycle. At any time, a given job is on one of the following lists:\r\n\r\n * *pending* - Jobs which are not ready to be executed (unfinished deps)\r\n * *queue* - Jobs which are ready to be executed\r\n * *active* - Jobs which are currently executing\r\n * *abandoned* - Jobs which were meant to be executed, but will not be attempted\r\n\r\nBy default, if one of the jobs fails, it will abandon all other active, queued, and pending jobs. This behavior can be controlled with the `continue_on_failure` and `continue_without_deps` options:\r\n\r\n * `continue_on_failure` - Don't abandon all jobs if one job fails.\r\n * `continue_without_deps` - Don't abandon the dependencies of failed jobs.\r\n\r\nJobs are activated subject to the availability of tokens from the job server. This happens whether or not the jobs erver is being used to manage `make` jobs since the job server treats the executor just like any other job client. Once a job is activated, each of its stages is executed in order, and if one stage fails, the job fails.\r\n\r\n#### Getting Feedback from Execution\r\nLike the current executor, the new execution pipeline uses a `Queue` to asynchronously communicate with the console thread. The executor writes [`ExecutionEvent`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/execution/events.py) objects into the event queue to be processed by any kind of output controller.\r\n\r\nThe default console controller has several options for output filtering and formatting:\r\n\r\n* `show_stage_events` False\r\n* `show_buffered_stdout` False\r\n* `show_buffered_stderr` True\r\n* `show_live_stdout` False\r\n* `show_live_stderr` False\r\n* `show_active_status` True\r\n* `show_full_summary` False\r\n* `active_status_rate` 20.0\r\n\r\n#### Creating `catkin build` and `catkin clean` Jobs\r\nJobs for building and cleaning jobs are created by factory functions defined for [catkin](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/jobs/catkin.py) and [cmake](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/jobs/cmake.py) packages. This makes the job objects task-agnostic, even if they require different types of information, as seen in the case of [`catkin build`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/verbs/catkin_build/build.py#L299-L318) and [`catkin clean`](https://github.com/jbohren-forks/catkin_tools/blob/clean-pkgs/catkin_tools/verbs/catkin_clean/clean.py#L148-L168).\r\n\r\n\r\n### Error Display\r\n\r\nThis PR dramatically simplifies the error output display in order to make errors clearer and less verbose. Since we have more controls to build single packages, it doesn't seem like it's necessary to have the \"command to reproduce\" and the numerous paths to different logfiles visible by default. On small screens, the actual error is usually even lost up above in the scrollback.\r\n\r\n#### Current\r\n![current_warnings](https://cloud.githubusercontent.com/assets/1253450/7790224/1746b164-024e-11e5-9a73-a8bef52df8eb.png)\r\n\r\n#### This PR\r\n![cb_new_warnings](https://cloud.githubusercontent.com/assets/1253450/7890457/3188e78a-0614-11e5-9666-2d5ede7de8fc.png)\r\n\r\n### Cleaning Individual Packages\r\n\r\nThis PR adds jobs for cleaning all files generated by individual catkin and CMake packages in the *build* and *devel* spaces. The \"clean\" jobs are different from the \"build\" jobs since they do not technically need to know the source path to packages. In fact, it's desirable to make them only depend on the information in the `devel_manifest.txt` files to be cleaned. This way we can support full orphan removal if a package's source is removed from the workspace.\r\n\r\n### TODO\r\n\r\n- [x] Properly filter output from \"python\" stages\r\n- [x] Move `devel_manifest.txt` files into the develspace so they can be cleaned without the build directories\r\n- [x] Move the `install_manifest.txt` files into the develspace so they can be cleaned without the build directories\r\n- [x] Move linked-devel isolated devel spaces out of package build directories\r\n- [x] Support complete orphan removal\r\n- [x] Refactor `catkin clean` pipeline to not require source packages to exist in the source space\r\n- [x] Make `--link-devel` the default behavior and add an option to build with an \"unlinked\" merged devel space like the current behavior \r\n- [x] Add load average and memory usage warnings to status bar\r\n- [x] Add file logging back in\r\n- [x] Add notifications back in\r\n- [x] Add `--unbuilt` option to `catkin build` to build any packages which have not yet been built\r\n- [x] Add notice to source the setup file if new packages are built or if packages are removed\r\n- [ ] Add documentation for new additions\r\n- [ ] Make \"isolated\" and \"merged\" devel space configs advanced (hidden) options\r\n- [x] Fix completion time display for packages which might not run continuously due to job occupancy  / scheduling (i.e. `cmake` stage runs and completes, then job is idle until the jobserver allows the `make` stage to complete)\r\n- [x] Fix queued package display to include \"idle\" jobs which are waiting for job tokens after being started (related to the above bullet)\r\n- [ ] Make `--unbuilt` cause any packages which depend on the unbuilt packages to be cleaned completely and rebuilt\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/11 Remove all unused imports\r\n- [ ] https://github.com/jbohren-forks/catkin_tools/issues/10 Cover with unit tests\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/8 Fix logging streaming behavior\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/7 Fix elapsed time display\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/6 Clarify load/mem high usage\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/5 Fix clean warning check\r\n- [x] https://github.com/jbohren-forks/catkin_tools/issues/4 Fix extra line breaks in captured input\r\n- [ ] https://github.com/jbohren-forks/catkin_tools/issues/14 Merge changes from upstream"},"comment":{"url":"https://api.github.com/repos/catkin/catkin_tools/issues/comments/149671973","html_url":"https://github.com/catkin/catkin_tools/pull/196#issuecomment-149671973","issue_url":"https://api.github.com/repos/catkin/catkin_tools/issues/196","id":149671973,"user":{"login":"wjwwood","id":100427,"avatar_url":"https://avatars.githubusercontent.com/u/100427?v=3","gravatar_id":"","url":"https://api.github.com/users/wjwwood","html_url":"https://github.com/wjwwood","followers_url":"https://api.github.com/users/wjwwood/followers","following_url":"https://api.github.com/users/wjwwood/following{/other_user}","gists_url":"https://api.github.com/users/wjwwood/gists{/gist_id}","starred_url":"https://api.github.com/users/wjwwood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wjwwood/subscriptions","organizations_url":"https://api.github.com/users/wjwwood/orgs","repos_url":"https://api.github.com/users/wjwwood/repos","events_url":"https://api.github.com/users/wjwwood/events{/privacy}","received_events_url":"https://api.github.com/users/wjwwood/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T19:10:45Z","updated_at":"2015-10-20T19:10:45Z","body":"Another note for myself: When the build fails it just reports that a certain number completes with warnings, but does not mention the failure (in the notification)."}}