{"action":"created","issue":{"url":"https://api.github.com/repos/ev3dev/brickstrap/issues/17","labels_url":"https://api.github.com/repos/ev3dev/brickstrap/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/ev3dev/brickstrap/issues/17/comments","events_url":"https://api.github.com/repos/ev3dev/brickstrap/issues/17/events","html_url":"https://github.com/ev3dev/brickstrap/issues/17","id":123975134,"number":17,"title":"Supporting multiple board/build variants reusing the same configuration (board dir)","user":{"login":"cmacq2","id":13215950,"avatar_url":"https://avatars.githubusercontent.com/u/13215950?v=3","gravatar_id":"","url":"https://api.github.com/users/cmacq2","html_url":"https://github.com/cmacq2","followers_url":"https://api.github.com/users/cmacq2/followers","following_url":"https://api.github.com/users/cmacq2/following{/other_user}","gists_url":"https://api.github.com/users/cmacq2/gists{/gist_id}","starred_url":"https://api.github.com/users/cmacq2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmacq2/subscriptions","organizations_url":"https://api.github.com/users/cmacq2/orgs","repos_url":"https://api.github.com/users/cmacq2/repos","events_url":"https://api.github.com/users/cmacq2/events{/privacy}","received_events_url":"https://api.github.com/users/cmacq2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-12-27T12:17:17Z","updated_at":"2015-12-28T01:09:45Z","closed_at":null,"body":"Splitting of the conversation from: PR #16 \r\n\r\n> So, I like where you are going, but I think we can do better than the environment variables that you have proposed here. What if we made it so that a board configuration directory had more than one config file for different \"flavors\"? The config files opt in to the package groups and hooks that are needed for that flavor. I think we would need to add an additional set of hooks that are run before the configure-packages to account for small differences in the root files. With this, to make a one-off, now you only have to copy a config file instead of the whole board configuration directory. What do you think?\r\n\r\nHow does the opt-in mechanism work, exactly?\r\n\r\nI think the risk here is that it quickly becomes overly-complicated to use by trying too hard for a 'neat' suggestion. And hard to implement besides: selectively opting in requires somewhat interactive two-way communication between configuration & brickstrap itself (for which there's currently no support other than fiddling with internal brickstrap state which is a dangerous proposition and should be discouraged).\r\n\r\nFortunately I think we can do something conceptually simple that sufficiently flexible and doesn't require as much from brickstrap.sh itself. We do this using convention-over-configuration approach, but before I get to that, briefly an excursion:\r\n\r\nConceptually there are always 4 categories or types of configuration:\r\n 1. General, generic rootfs configuration (generic package selection, hooks, rootfs skeleton)\r\n 2. Arch specific rootfs configuration (like kernel version, bootloader).\r\n 3. Board specific configuration (like custom DTBs, firmware)\r\n 4. Variant builds. I.e. \"minimal\", to \"full fat\" builds of what is otherwise the same in layers 1-3. So essentially just extending/blacklisting packages.\r\n\r\nNow it's not important specifically where the boundaries of each category are, just that intuitively once you start sharing a board directory for different device & arch combinations it becomes apparent to the user what the most optimal partitioning of their configuration files would be. The user knows very well what is just a simple \"variant\" extension, what is board specific, what is arch specific, etc. to their build set up: precisely because they put it all together themselves!\r\n\r\nNote that we should not *require* the user to take full advantage or to obsess over getting their partitioning right: they can simply opt-in to the scheme and use it as and when convenient. \r\n\r\nSo instead of trying to mediate between configuration, we offer a declarative convention and rely on the user to set up their partitioning right: we introduce the concept of overriding paths, and assign prefixes to the 4 types (which can be stacked as follows, bottom-to-top of the stack):\r\n\r\n 1. The generic re-usable configuration goes in `$BOARDDIR` i.e. it lives in the board directory directly. This is to preserve backwards compatibility with existing setups as much as possible.\r\n 2. The arch specific bits go in `$BOARDDIR/arch/$arch/`, where `$arch` is the right arch. Essentially the arch is stacked on top of the generic config. Note that `$arch` may or may not correspond to a valid architecture for Debian archives, if it doesn't the user simply has to set the right `ARCH` variable in their `config` file (e.g. `$BOARDDIR/arch/something-weird-I-did/config`). Also note that an empty `$arch` is valid, basically in that case there is no arch specific layer to the user's configuration.\r\n 3. The board specific bits are stacked on top of this using `$BOARD_PREFIX/board/$board`. In this case `$BOARD_PREFIX` is either `$BOARDDIR/arch/$arch` or if `$arch` is empty then it is just `$BOARDDIR`. Again, `$board` is an arbitrary value like `$arch`, and again it may be empty (in which case there is no board specific layer).\r\n 4. Similarly the variant specific bits are stacked on top of that using `$VARIANT_PREFIX/variant/$variant` where `$VARIANT_PREFIX` takes the board also into account. Again `$variant` is arbitrary and may be empty.\r\n\r\nThat is: the top of the stack is the variant directory, and the bottom is the generic configuration directory. This convention permits the user to selectively override files simply by placing an alternative in something higher up the stack. This is generally a simple matter of copy-paste & then modify the contents of the overriding file.\r\n\r\nWhenever brickstrap.sh then selects a configuration file all it has to do is consider this directory convention scheme and honour overrides properly.\r\n\r\nThere are three partial exceptions to the general override mechanism:\r\n * The raw rootfs files (as used by `copy-root` stage) and \r\n * Package files. \r\n * Hooks.\r\n\r\nFor those three an additive scheme should be used: that is each layer of the stack adds to the next layer of the stack, but files with the same `basename` override others lower down the stack.\r\n\r\nThen if you also pull in PR #16 you can also override package selection in your `config` file by setting blacklists, permitting you to exclude specific packages on an arch, board and/or variant specific case-by-case basis.\r\n\r\nFinally, how does brickstrap.sh know what arch, board and variant to pick? Easy: we introduce additional command line switches to pass these in as parameters. (E.g. `-A` for architecture, `-V` for variant, and `-B` for board.)\r\n \r\n> Regarding non-32-bit-arm architectures, I think we just use the existing ARCH variable from the config files to pick the correct qem-user-static (or none at all). But you should open a separate issue for that.\r\n\r\nYes. I would go with the following refinement of this basic idea, though:\r\n\r\n 1. Permit the user to set the correct QEMU directly in their `config` file.\r\n 2. If the `config` file doesn't specify `ARCH` set `ARCH` to the value learned from `-A` if it is available. \r\n 3. `ARCH` is still empty, assume a native build.  Possibly recognise the special value `\"native\"` to assume a native build also. \r\n 4. If the user hasn't specified the right QEMU manually, use the value of `ARCH` to pick a suitable default.\r\n 5. If it's a native build set `ARCH` to the host architecture.\r\n\r\nThis way the user can also use QEMU which they built themselves, or if no official Debian is available for the arch yet. Useful when chasing QEMU bugs like missing syscalls, or if you are trying to port Debian to something new/obscure ..."},"comment":{"url":"https://api.github.com/repos/ev3dev/brickstrap/issues/comments/167452535","html_url":"https://github.com/ev3dev/brickstrap/issues/17#issuecomment-167452535","issue_url":"https://api.github.com/repos/ev3dev/brickstrap/issues/17","id":167452535,"user":{"login":"cmacq2","id":13215950,"avatar_url":"https://avatars.githubusercontent.com/u/13215950?v=3","gravatar_id":"","url":"https://api.github.com/users/cmacq2","html_url":"https://github.com/cmacq2","followers_url":"https://api.github.com/users/cmacq2/followers","following_url":"https://api.github.com/users/cmacq2/following{/other_user}","gists_url":"https://api.github.com/users/cmacq2/gists{/gist_id}","starred_url":"https://api.github.com/users/cmacq2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmacq2/subscriptions","organizations_url":"https://api.github.com/users/cmacq2/orgs","repos_url":"https://api.github.com/users/cmacq2/repos","events_url":"https://api.github.com/users/cmacq2/events{/privacy}","received_events_url":"https://api.github.com/users/cmacq2/received_events","type":"User","site_admin":false},"created_at":"2015-12-28T01:09:45Z","updated_at":"2015-12-28T01:09:45Z","body":"> If I'm understanding this right, I could, for example, use a variant to add extra packages, possibly from an arch dependent ppa. \r\n\r\nI envisaged `$variant` as a fully independent layer. So basically just pull in packages which you know work for (substantively) and just blacklist those that don't in your `config`. \r\n\r\nOr vice versa, populate `PACKAGES` with a whitelist in `config` if the balance of effort tips the other way.\r\n\r\nIt may be worthwile to introduce a `packages.blacklist` file for this purpose so that even blacklists can be inherited/overriden like any other file without touching `config` directly.\r\n\r\n--- \r\n\r\n> and another package that is arch dependent (at $distor/arch/$arch/variant/$variant).\r\nI noticed your other reply about the path lookup. Let's take a step back and rethink..."}}