{"action":"created","issue":{"url":"https://api.github.com/repos/machinekit/machinekit/issues/791","labels_url":"https://api.github.com/repos/machinekit/machinekit/issues/791/labels{/name}","comments_url":"https://api.github.com/repos/machinekit/machinekit/issues/791/comments","events_url":"https://api.github.com/repos/machinekit/machinekit/issues/791/events","html_url":"https://github.com/machinekit/machinekit/issues/791","id":117825061,"number":791,"title":"Build and CI infrastructure needs replacement","user":{"login":"zultron","id":1940736,"avatar_url":"https://avatars.githubusercontent.com/u/1940736?v=3","gravatar_id":"","url":"https://api.github.com/users/zultron","html_url":"https://github.com/zultron","followers_url":"https://api.github.com/users/zultron/followers","following_url":"https://api.github.com/users/zultron/following{/other_user}","gists_url":"https://api.github.com/users/zultron/gists{/gist_id}","starred_url":"https://api.github.com/users/zultron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zultron/subscriptions","organizations_url":"https://api.github.com/users/zultron/orgs","repos_url":"https://api.github.com/users/zultron/repos","events_url":"https://api.github.com/users/zultron/events{/privacy}","received_events_url":"https://api.github.com/users/zultron/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":153,"created_at":"2015-11-19T14:28:20Z","updated_at":"2015-12-05T09:16:17Z","closed_at":null,"body":"This issue is to track technical issues surrounding the replacement of\r\nthe Buildbot.\r\n\r\n---------\r\n\r\nAs we all know, the current Buildbot has had a hardware failure, and\r\nthe required time to fix it would be better spent propping up a\r\nbrand-new system that doesn't suffer from basic shortcomings of the\r\ncurrent system.  Above all others, the shortcomings that most need\r\nrectification are staggering complexity and fundamentally\r\nirreproducability, which, combined, create high-risk for the project,\r\nsince this critical piece of infrastructure cannot easily be replaced\r\nafter a catastrophe.\r\n\r\n# Requirements\r\n\r\nMichael and I have discussed this issue, and having eliminated the\r\nother options we're aware of (e.g. fixing the current 'bot and\r\nbuilding on OBS), we have a plan to build a new, simpler Buildbot,\r\nwith greatly simplified requirements, especially in the initial\r\nversion:\r\n\r\n- Capable only of building a single OS/architecture configuration\r\n  - e.g. Jessie/amd64; Wheezy/armv7\r\n- Capable only of testing a single OS/architecture/thread-flavor\r\n  configuration\r\n  - e.g. Jessie/i686/Xenomai; Wheezy/amd64/POSIX\r\n- Kernel threads, esp. RTAI, will be removed from configuration\r\n  options\r\n- Minimal requirements for compute environment:  builds and tests will\r\n  run in standardized container environment, dependent only on the\r\n  container technology and possibly CPU architecture (this requirement\r\n  may be relaxed through emulation)\r\n\r\nThese requirements are squarely designed to address the aforementioned\r\ngoals of simplicity and reproducibility.\r\n\r\n# Benefit:  Simplicity\r\n\r\nThe above requirements serve to strip away unnecessary complexity in\r\nthe system.\r\n\r\nScripts to set up and run containers and the Buildbot configuration\r\nmay be greatly simplified by only needing to build and test a single\r\nconfiguration, by removing build and run-time complexity introduced by\r\nkernel threads, and by minimizing moving parts and standardizing the\r\nenvironment through containerization.\r\n\r\n- Developers will face lower learning curves to contribute to scripts\r\n  and configuration\r\n- Operators will face lower learning curves to set up and maintain\r\n  a CI system instance\r\n- Minor system failures can be more easily diagnosed and fixed\r\n- A master and single slave running together in a single container\r\n  eliminates the infrastructure requirements of orchestrating several\r\n  hosts and services\r\n- Instances are decoupled, so when a single instance fails, other\r\n  instances will continue to run unaffected\r\n- Removing RTAI and kernel threads simplifies programming and\r\n  configuration\r\n  - RTAI imposes significant additional build-time requirements\r\n  - Loading kernel modules from within a container may be infeasible;\r\n    removing them enables the use of container technologies and the\r\n    benefits those convey\r\n- Removing RTAI and kernel threads simplifies regression testing\r\n  - Regression tests on kernel threads have often failed when userland\r\n    threads pass\r\n  - Regression tests have rarely, if ever, failed for one userland\r\n    thread flavor but passed for another\r\n  - So, if kernel threads can be eliminated, it may be sufficiently\r\n    safe to assume that results for other userland threads will follow\r\n    those of POSIX threads, and thus are unnecessary\r\n\r\n# Benefit:  Reproducibility\r\n\r\nThe above requirements increase reproducibility, enabling new systems\r\nto be quickly and easily installed from scratch on either commodity\r\nhardware or inexpensive cloud services.  Many benefits of\r\nreproducibility derive from those of simplicity, as described above:\r\n\r\n- Catastrophic system failures will interrupt service for only a\r\n  short while until a new system is brought up from a fresh OS install\r\n- Many CI system instances running in parallel eliminate the SPOF\r\n  (single point of failure) for the overall CI service\r\n- Cost is key to reproducibility:  with low-cost commodity hardware or\r\n  cloud services, more instances may be run, increasing quality of\r\n  results\r\n- Low learning curves, low cost and low effort result in low barriers\r\n  to entry, and thus reproducibility of contributors\r\n  - Conversely, if a system requires experts to manage, there will be\r\n    few people capable of managing the system, resulting in another\r\n    kind of SPOF\r\n\r\n# Technical overview\r\n\r\nFollowing is an overview of technologies selected for the\r\nimplementation and an overview of provisioning Buildbot in Docker, and\r\nBuildBot configuration for a Machinekit CI system.\r\n\r\n## Technologies used\r\n\r\nThe two major choices of significance are Buildbot for the CI system\r\nand Docker for containerization.  The following sections describe the\r\nreasoning behind the selection.\r\n\r\n### Buildbot\r\n\r\nThe CI system will be based on **Buildbot**, as the previous version was.\r\nThe main advantage of Buildbot is it is lightweight and thus suitable\r\nfor running on low-powered platforms, especially ARM; its\r\nconfiguration and its build result status pages are familiar to the\r\ncommunity; and its configuration is written in python, which is\r\nalso well-understood in the community.  While it is not a perfect\r\nsystem, it has served its CI functions well, and does not contribute\r\nto the fundamental problems being addressed in this issue.\r\n\r\n### Docker\r\n\r\nIn order to build and test Machinekit on targeted OSs and CPUs\r\n(esp. Debian Wheezy and Jessie, and amd64, i686 and armv7l) with\r\nmaximum independence from the host OS and CPU, the system will run in\r\n**Docker** containers.  Docker is a new technology, but has many\r\nadvantages.  Like most virtualization technologies, it provides a\r\nreproducible run-time environment where processes will always run\r\nidentically, irrespective of the host OS.  Best of all, Docker runs on\r\nnearly all current GNU/Linux distros as well as MacOS and Windows, and\r\nall major cloud services support Docker.\r\n\r\nBased on Linux cgroups, Docker is lightweight, and may be thought of as\r\n'chroot on steroids'.  Because it does not need to run a separate\r\nkernel from the host, it has lighter memory and CPU requirements.\r\nWithin Docker, Machinekit may also be able to use the host kernel's RT\r\ncapabilities and have the same latencies as running on bare metal.\r\n\r\nDocker also has a simple system for building container images.  A\r\n`Dockerfile` specifies a standard image to start from with a statement\r\nsuch as `FROM debian:jessie`.  Then this base image is built upon by\r\nfollowing statements, the majority being shell commands such as `RUN\r\npip install buildbot`.  The resulting image is similar to a chroot\r\nfilesystem where the Buildbot service will run.\r\n\r\nA Docker container is then run on the configured image, perhaps\r\nrunning the Buildbot services for production, or perhaps running an\r\ninteractive shell for development.  Docker provides command-line\r\nswitches to mount directories from the host filesystem in the\r\ncontainer, useful for data persistence across container runs and data\r\nexport, and to expose network ports, useful to expose services running\r\nin the container.\r\n\r\n## Buildbot and Docker image provisioning\r\n\r\nThe Buildbot will run inside a Docker container that is generated from\r\na set of scripts.  These scripts should be easily run from a short set\r\nof instructions in a `README.md` file.\r\n\r\nStarting from a newly installed OS, an operator will install Docker\r\nwith `sudo apt-get install docker.io` (on Debian Jessie, the\r\n`jessie-backports` repository [must be enabled][1]).\r\n\r\n[1]: https://docs.docker.com/v1.8/installation/debian/\r\n\r\nThe operator will then clone the GitHub repository (to be determined)\r\ncontaining the scripts.  The repository is expected to contain a\r\nsample YAML configuration file for setting the Docker base image\r\n(e.g. `debian:jessie`) and other simple parameters.  Then the operator\r\nwill run a script that renders the `Dockerfile` from a template and\r\nbuilds the container image.  The image will have Buildbot master and\r\nslave software and all Machinekit build and run-time dependencies\r\npre-installed.\r\n\r\nThe new Docker image will then be run with the git tree mounted at\r\n`/home/docker` and with the Buildbot web ports exposed.  For\r\nproduction service, a `supervisord` instance will be launched within\r\nthe container; this in turn will start the Buildbot master and slave.\r\n\r\nThe Buildbot daemons' working directories will be set to\r\nsubdirectories of the git tree.  Here, they will store their run-time\r\nartifacts such as Machinekit build tree, logs, configuration, etc.\r\nThe python language Buildbot master configuration will retrieve\r\nparameters from the same YAML file as the provisioning scripts.  It\r\nis undecided whether other Buildbot master and slave configuration,\r\nrespectively generated by `buildbot create-master` and `buildslave\r\ncreate-slave`, will be generated at first run or saved in the git\r\nrepository.\r\n\r\n## Buildbot functions\r\n\r\nThe basic Buildbot functionality follows the current Buildbot, and is\r\nvery well-understood.  It can be divided into two parts: RIP + package\r\nbuilds and regression tests when a new PR is submitted, and package\r\nbuilds and APT repository update when a PR is merged into master.\r\n\r\nThe Buildbot master configuration is contained in a python language\r\nscript called `master.cfg`.  A quick overview follows to establish\r\nterminology and identify the basic components of configuration.\r\n\r\nA complete Buildbot system consists of a Buildbot master daemon and,\r\nin this configuration, one Buildbot slave.  Major configuration is on\r\nthe master, and the slave only requires basic configuration, such as\r\nits own name and base directory, and the master's hostname, port and a\r\npassword for authentication.  After the master starts up, it goes\r\nthrough a workflow, described as follows, in terms of its major\r\nconfiguration items.  The `master.cfg` file sets up a dict,\r\n`BuildmasterConfig`, with keys for each major item.\r\n\r\nFirst, the master listens on port `9989` for its **slaves** (only one\r\nin this configuration) to connect.  The configuration `slaves` key\r\ncontains a list of slaves identified by name and password.  When the\r\nslave is started, it connects to the master over that port and\r\nauthenticates itself.\r\n\r\nThen, the master waits for a new commit.  The configuration\r\n`change_source` key has a list of **change sources** (again, just\r\none), with the Machinekit GitHub repo URL.  An external script running\r\nperiodically from cron, `github_buildbot.py`, polls the GitHub repo\r\nfor new commits; when one is found, it connects to the master through\r\nport `9989` and sends it **change set** data:  the changed branch and\r\nrevision, etc.\r\n\r\nThe master passes the incoming change set to the **schedulers**,\r\nconfigured in a list under the `schedulers` key.  The scheduler\r\ndeploys the build to a 'builder'.  A **builder**, configured in a list\r\nunder the `builders` key, matches a list of (one) build slaves to a\r\n'build factory' (see next section).\r\n\r\nThe **build factory** is a python object that represents a list of\r\nsteps to perform for a particular kind of build.  This configuration\r\nmay have two separate factories, a 'build and test' factory for new\r\nPRs, and a 'build and deploy packages' factory for merges to `master`.\r\nA basic Machinekit build and test factory would include the following\r\nsteps (separate steps to configure and build packages, omitted,\r\nsimilar to following example):\r\n\r\n- Pull the revision (specified in the change set) from the GitHub repo\r\n  (specified in the change source) and update the local working tree\r\n- Run autoconf, `./autogen.sh`\r\n- Configure the source, `./configure`\r\n- Compile the source, `make`\r\n- Run `make setuid`\r\n- Run tests, `runtests`\r\n\r\n\r\nA basic Machinekit package and deploy factory would include the\r\nfollowing:\r\n\r\n- Pull the revision (specified in the change set) from the GitHub repo\r\n  (specified in the change source) and update the local working tree\r\n- Configure the package, `debian/configure -prx`\r\n- Build the package, `dpkg-buildpackage [...]`\r\n- Update the APT repository, `reprepro [...]`\r\n\r\nEvery step has a name (`configure`, `build`) and a working directory,\r\nand some logic ('if this step fails, stop').  Each step has a python\r\nobject type that defines its behavior; e.g. `steps.Git` pulls the\r\nchange, and `steps.Compile` builds the source, scanning for warnings.\r\n\r\nFinally, especially of interest to Contributors and Maintainers, the\r\n`status` key contains a list of **status targets**; our configuration\r\nwill have one: the web interface that shows us pass/fail results on\r\nthe web display.  It needs configuration for the http port and list of\r\nuser credentials.\r\n\r\nThat's the quick overview of Buildbot master configuration.  There are\r\na few other trivial configuration keys I glossed over; these are\r\nwell-documented in the `master.cfg` file.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/machinekit/machinekit/issues/comments/162163522","html_url":"https://github.com/machinekit/machinekit/issues/791#issuecomment-162163522","issue_url":"https://api.github.com/repos/machinekit/machinekit/issues/791","id":162163522,"user":{"login":"kinsamanka","id":3701049,"avatar_url":"https://avatars.githubusercontent.com/u/3701049?v=3","gravatar_id":"","url":"https://api.github.com/users/kinsamanka","html_url":"https://github.com/kinsamanka","followers_url":"https://api.github.com/users/kinsamanka/followers","following_url":"https://api.github.com/users/kinsamanka/following{/other_user}","gists_url":"https://api.github.com/users/kinsamanka/gists{/gist_id}","starred_url":"https://api.github.com/users/kinsamanka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinsamanka/subscriptions","organizations_url":"https://api.github.com/users/kinsamanka/orgs","repos_url":"https://api.github.com/users/kinsamanka/repos","events_url":"https://api.github.com/users/kinsamanka/events{/privacy}","received_events_url":"https://api.github.com/users/kinsamanka/received_events","type":"User","site_admin":false},"created_at":"2015-12-05T09:16:17Z","updated_at":"2015-12-05T09:16:17Z","body":"@zultron, please send the PR.\r\n\r\nBut what happens if the commit to master fails?\r\n\r\nI'l add the sftp code later together with the docker container that will interact with it."}}