Rwy'n cael ei wneud o'r ffordd, ac rydyn ni'n Llew Kyns ar y UK, rydyn ni'n gweithio'r axon, ac rydyn ni'n Llew Hatt CTO, rydyn ni'n gweithio'r enghreifft. Rydyn ni'n gweithio'r gweithio'r gweithio'r gweithio. Rydyn ni'n gweithio'r gweithio'r gweithio'r gweithio'r gweithio. 1970, cydwyddiadau y TPM iawn i ddosbyddiaeth gyda'r unigach wahanol, yn y sydd gweithlo'r sydd gweithio'r sydd gweithio yn ychwanegodd ymarfer. Ac ei wneud i gyda'n cael ei wneud ar gyfer y gweithio'r cydwyddiadau cydwyddiadau ymarfer. Mae'n gweithio'r gyda'r cydwyddiadau. Ychwanegau yma, mae eich cymdeithas eich cyffordiydd cyfnanaeth, mae eich cynhyrchu oedd yn cymryd yn y cyfnod oeddyn i ymweld. Fe yna'r cyd-m如果你g, mae eich cyfnod oedd yn cyfnod oedd. Mae eich cyfnod oedd yn cyfrifwyr cyfnod oedaf yn ceis. Mae oedd yn cyfnod oed o Gymrydolol o eich cyffordiydd, mae eich cyfnod oed yn gyfnod oedd yn cyfnod i fynd. I'm very much locked within that chip and it's created at manufacture time And we'll go into that quite soon. Its called an entity key And the other thing that TPMs can do alongside creating keys and so forth Is that they can do this signing of artifacts So essentially when I say story not signing a measuring of artifacts So this has ychydig o'r objeu, y ffirmweblob, y ffile, y bwtloader, y cernol, y cernol modul. Mae'r bwysig wedi'i gweithio gyda'r cyftograff ychydig. Felly, ychydig ychydig o'r 2, 5, 6 ychydig. Mae'r bwysig wedi'i gweithio'r TPM. Y TPM mae'r bwysig ychydig o'r bwysig ychydig o'r objeu. Mae'r bwysig ychydig o'r bwysig ychydig o'r bwysig. Mae'r bwysig ychydig o'r bwysig cyftograff ysgwrdd gyda'r ychydig o'r gwaith ar hyn ar eich byd. So, mae'n ei bwysig eithio er gynnig gwaith y hynny ychydig. Mae'n bwysig yn cyffredinol, oes arτι mae'n bwysig, â'n ddilyn a'r hollol, a'n gweithio'n gweithio i gael ladw. Ac mae'r bwysig ychydig o'u bwysig. Mae'r bwysig eich hwyl olef. Haiddydd pethol yn gorfod o'ch gweld yn ystod gweinwyr yma. Mae lŵw'r mhobl yn cael ei huniad dros hynny. Felly mae'r gwan ni'n gwneud ymddangos gwylo yma felly chylau yw'r gwaith yma. Yma sy'n gwylo ffwilio ymddangos chi ddyliad diwethaf yn gweld yn gweinwyr. Mae'r gwylo'n gwaith mae'r gwaith o'r gwaith. Mae'r rhan o'r rhan o'r Cyfrifoedd Cymru, oherwydd mae'n gynhyrchu'r cyfrifoedd. Mae'r cyfrifoedd cyfrifoedd cyfrifoedd y cwrwm ar y cyfrifoedd, mae'r cyfrifoedd cyfrifoedd ymlaen i ymlaen, a'n cael argymintech i'r cyfrifoedd, ond mae'r cyfrifoedd yn cael ei gydag. Mae'n cael ei zedlau, mae'n cael ei gondolaeth. Mae'n cael ei gondol ymlaen, mae'r cyfrifoedd ymlaen yn sicr i'r idea yw'r wrthyn. It's very easy to scale this. It's very performant. So it's asynchronous. It's non-blocking IO network. So the TPMs tend to be a little bit slow. So that way we don't get a bottleneck. So we can have a verifier scale to monitoring thousands and thousands of machines that have TPMs. We support the latest standard, TPM 2.0. We also support 1.2, but that's getting quite old now. So we'll be depreciating that. ac mae'n dweud o'r ffordd o'r ffordd o'r ffordd. A oedden nhw Keyline wedi bod yn ffawr o'r MIT, Lincum Lab Security Research Department. Mae'n rhaid i'r cyflawn gyda unrhyw, ac rydyn ni'n fawr o'r project oherwydd y cwyl ac rydyn ni'n rhaid i'r cyffredig o'r cyfrwyr o'r cyfrwyr o'r cyfrwyr o'r cyfrwyr, oherwydd o'r cyfrwyr o'r cyfrwyr o'r cyfrwyr. Rydyn ni'n rhaid i'r project oherwydd. Efallai rydyn ni felly rydyn ni'n rhaidio am hynny. So Keyline provide measured boot, ac rydyn ni'n rhaid i'n rhaid i'r ffordd o ysgolwyd. Dyna i'r measured boot arall, rydyn ni'n rhaid i'r ffordd o'r ffordd o'r cyfrwyr, mae'r shim a'r redhat shim project yn ysgrifenni i'r meddwl fonts jytaeth oherwydd yna'r cerdd ddwy, iChart fydda i'r cerdd o'r rhaid i. Llywodra, a'r cerdd o'r cerdd o'r pwy yn toggle on and off SE linux or audit or those various powerful flags that can be changed on the GRUB command line kernel options. So we can actually measure those as well to make sure nobody's tempted with those. The Unnet-Ramffs and the modules and then further into user land which we will go to next. Also Secure Boots. So there are various parts to secure DPS, Moc list, Vendor DPS, ffair y cerddyn nhw'n ei wneud o'r wathriadau'r ddweud o'r pwysig. Yn hyn ymgyrch yn ymlaen, oedd y cyfrifio mewn argytêw. Ymlaen o'r cyfrifio ar gyfer Linox ysgolain a'r cyfrifio'r cyfrifio'r cyfrifio ar gyfer Linox. Am Alfys, sy'n tu cyfnod celf yn ystafell 2.7, felly bydd yn gweithio'r ddechrau i'r gwahau gyda'r calws. Felly rydym yn credu yn ddim arno'r cyfnod siŵr i wneud â gweithio'r cymotig, mae'n iивno i'r gweithio'r sall ac yn archifolid i gael oherwydd hynny yn sicr y bwerdd hynny'n unrhyw o'r cyrgynod. Ar y dyma am'r CEO yma, mae'r cyfrifau a chyfrifau'r Caerdydd. yr Yma wedyn yn ei adwr i ddechrau'r cyfrifau, Mae'r clywedd yn cydwyr cyrcaf, sydd yn ôl i'r ffant, ac mae'n ddynnu'n gweithio chi'n gweithio'n ddynnu'n gweithio'r TPM, ac mae'n rhan o'u bod hynny yn gweithio'n gweithio'r têffadol. A byddai'n gweithio'r tîffadau. Felly, yn ymgyrch chi'n gweithio'r pwysig y bindur, We could have something like the IP binary and they swap it out with a Trojanised binary. It's obviously going to have a different cryptographic checksum. So we'll be able to remotely see within seconds of that executing that somebody has actually changed that binary and take actions from there. Also the same for kernel modules as they load and SE Linux labels and changes and so forth. Essentially we've got the initial boot, the system, secure boot and then run time. So we can measure all of these components remotely. The other thing we have as well is an encrypted payload and execution framework. So what we do is when a system proves its trust, so we know that nobody's tampered with that system, we can then release an encrypted payload on that machine. So that could be, for example, some private secrets that you have or certain files that have passwords or sensitive materials in. So the machine will prove its trust date, it's not been compromised and then we will then release an encrypted payload. If the machine fails to suggest that somebody has somehow tampered with that machine, they cannot access the payload. And finally we have something called a revocation framework. So this essentially happens when a machine fails. So somebody tampers with a machine, then we interface with a certificate authority, typically we use Cloudflare SSL so we can revoke a certificate and that certificate revocation in turn could invalidate IP sectonals or TLS connections and so forth. And then we also have these things called custom actions and these are where you delegate a list of scripts that are written in Python. And what happens is all of the nodes that are part of the keyline cluster, what they will do is they will execute these local actions when one node fails. And typically what you would do is you get those nodes to stoniff the node that's failed. So stoniff, if anybody's not familiar with that, I should know this, it's shoot the offending node in the head essentially, so ring fence it. So here it's very flexible, you can essentially anything that's programmatically possible on a Linux machine within Python, you can execute that locally. So it could be calling specific APIs, it could be shutting down network connections, removing hosts from Northrise, SSH key file, anything that you come up with really, you can set a script and then Keyline will execute that script once a node fails. So a bit of a high level overview of the different components. So first of all we have the agent, so what we've got here is to our left is consider that the sort of the remote data centre or perhaps an outside location where you have an IoT device or an edge device. This is essentially an area that's outside of your control, outside of your network. And this is where we run the Keyline agent. And the Keyline agent communicates with the TPM, so it requests these quotes, and quotes are essentially a request to get a measurement list of the current cryptographic state of the system, which is then sent to the verifier, which is now on premise. So this would be something that you have within your own trusted boundary. And the verifier accepts these lists, it does a check to make sure that it's a real TPM that sent that using the private key hierarchy that I spoke of earlier with the TPM, and it then performs a kind of a comparison between current state and what the expected state is. Now if that state changes then the verifier will then fail that node and then we'll have our revocations and so forth. But before we get on to that there's also the register. The register is relatively simple, it's a database, it's where we store the agent IDs, the unique IDs, their operational state, and we also keep the public keys of the TPM vendors. So normally they'll provide like an intermediate certificate which can then be used to vouch that a TPM is actually a real TPM and it's not a spoofed actor pretending to be a TPM. We then have our revocation service and the revocation service is like I described earlier, when a machine fails then we can have a series of scripts which kick off on all of the other machines to ring fence that particular machine. And likewise it will also connect to a certificate authority. And at the moment we support OpenSSL and Cloudflare SSL but we're going to move this to being a plug-in framework so we can easily integrate with other CAs. All they have to do is write their own driver. OK, so this is where it gets a bit hand-wavy, so do pull me up afterwards if something doesn't make sense. So initially we have to set up something called a hardware root of trust. So we need to establish is this actually a real TPM that I'm talking to? So what happens is the key lime agent, so remember this is the remote machine that we're monitoring. This will send an ID, there's no cryptographic properties to the ID, it's just a unique identifier. And it will send two public keys, an EK pub and an AK pub. Now an EK is the entity key, OK? And this entity key is burnt at manufacturer time into the TPM, OK? So nobody can get hold of that theoretically, not even the TPM manufacturer themselves. What happens is they, I don't know it too fluently, but they inject a random seed and then it sort of self-bootstraps its own cryptography and creates this entity key, which is locked within the chip. And then the entity key also creates an attestation key. So the attestation key is not fixed to the hardware. So if you was to reset a TPM, then the AK would be wiped and you would generate a new one. Whereas the entity key is a permanent fixture to the TPM. So we send the public counterparts in the ID to the register. And what the register does is using the public key, it encrypts the AK pub using a hash made with a key which we call the KE, which is essentially a challenge, OK? So we're challenging the machine to say, prove to us that you have the private counterpart of the entity key, OK? So what will happen is the agent will then make a HMAC of its ID and it will send that back to prove that it actually has the EK private key. And then, further to that, we'll also verify that the EK private key, sorry, public key is signed by a natural TPM manufacturer as well. And that allows us to then tie the attestation key to the entity key. So now what we can do is when we receive a cryptographic quote from a machine, we can be sure that it's an actual real TPM that signed that. So that is a bit hand-wavy, isn't it? So hopefully that went in. It took me a while to understand this stuff as well. So now we've got our hardware router trusted up. So we can now trust the hardware. So the second part is we do an initial attestation of the state of the machine, OK? So now you can see that we introduce, I don't know if there's a laser on this. Now you can see at the top there's a verifier. So the verifier is coming into the picture now. So what we do is, sorry, and there's a four factor as well, the key lime tenant, OK? Now the key lime tenant is essentially, this is you as the user, drive this, OK? And we provide a CLI, so you can type commands with arguments to kick off this process. But that CLI actually wraps around REST APIs. So if somebody had their own system, they could obviously develop their own system around our REST APIs to drive this. And what happens is the tenant creates a key, which we call the bootstrap key, OK? And this key is cryptographically split into two. So it's not a file that's split into two. It's actually done in a cryptographic manner, OK? And that makes the two counterparts, the U and the V. And we use this for unlocking the encrypted payload. And we'll see how we do that shortly. So first of all, we send the V part to the verifier. So this is the V part being half of the bootstrap key, OK? And we also send an agent ID, again, just a unique identifier, an IP address, and a white list, OK? And the white list is a list of hashes and a POSIX path to a file. So you essentially have a hash value of a file to the right, two columns. Pretty simple. And that's your golden state of how you expect the system to be, OK? The verifier then sends a nonce to the keyline agent, because we want to make sure it's fresh. We don't want somebody to try and send an old quote. So we send a nonce to make sure there's no replay attacks and so forth. And then the agent will communicate with a TPM and it will send back this TPM quote, OK? Which is using the nonce, it hashes the attestation key of a value called a PCR, which is a platform configuration register. And these are letter boxes where there's hash measurements that are stored in the TPM, OK? And this is sent back. And it also sends a public counterpart of a new key called the NK, OK? And the NK is something that we use to protect transferring secrets from the agent, sorry, from the verifier and the tenant to the agent itself. Now, when this comes back, we first of all check the validity of this attestation key, which goes back to the previous slide where we had the hardware root of trust. So again, we make sure that this is actually a real TPM that we've established a hardware root of trust with. If that trust pass, then the verifier will hand over the V counterpart and it will use the NK key that we spoke about, the NK public key to do that. So it's done in a secure manner. So this can all happen over HTTP, and it's absolutely fine. There's no problem with that. This is going to happen across completely untrusted networks. Now, the second part is we now need to get the U part of the key, second part of the key over to the agent so that it can unlock a payload. So now as the tenant, we request a quote, OK? And with this quote, we're not actually measuring the system. It's just a way of us showing intent to the machine to prove its identity. So we use a quote to do this. And it comes back with a quote that's signed by the attestation key so we can again check the hardware root of trust. And the agent also sends its NK pub, which we will use to safely transit the second part of the key. OK? Again, we'll check the trust, check the trust state, the hardware root of trust. And then if that is shown to be non-compromised, we will use in the NK pub, we will send the U counterpart, the second counterpart of the key, over to the remote host, along with a payload, which is just essentially a tarball, OK? And we're going to what a payload will typically contain shortly. So the agent then combines the keys again to one key. Now it has that full key. The secrets are unlocked. And then we have a deploy hook. So we have a script that will run, which could do anything. You could call an answerable playbook. You could have perhaps a bash script that deploys an application. It's essentially up to you then. So here's an example of a very simple example of an encrypted payload. So you can see we have our payload.tar. And in there we've got various secrets that we want to securely transmit to a machine. Once we know that nobody's tampered with that machine, we can actually trust the environment. So you can see there's various secrets and binaries. And then at the bottom you'll see these scripts that are prefixed with local actions. Now these are the scripts that are executed when a machine fails its state. So for example, these two scripts are making some calls with cube CTL around a cluster. And they're also making some changes to IP tables. So essentially what we're going to do is when a machine fails, a signed event will be sent out to all the machines, they'll run these scripts locally and that will ring-fence the compromised machine. And then you can see there's an auto-run.sh which we automatically execute again on. Once the trust is passed, the keys are combined, the payload is made available, we will execute the auto-run.sh script. And in here you can see a simple example that calls an answerable playbook. So the idea is you would deliver the secrets, answerable would run, deploy the application and your secrets are there. And you know that they've gone on to a machine that you can trust. Now once this happens, we move into the third phase which is continuous integration. So we've measured the boot, the secure boot components. We've cryptographically delivered the payload. Now we want to continuously monitor that machine for compromise. So here we use integrity measurement architecture. And what happens is we continuously poll for these TPM quotes. And this actually happens over a REST API and you don't even need to protect that. It doesn't even need to be on HTTPS, it can be on HTTP because these measurements as I say are cryptographically signed by a key which is locked within the TPM. So if anybody tried to tamper with those measurements, it's going to break the cryptography and it's going to fail. So the TPM quote comes back and we then move into continuous polling. So we do this about every two seconds. And what will happen is every time somebody makes a sys call or they make some sort of a change, IMA will write this to the security FS. And the security FS will then lodge the hash that is recorded into the TPM. And then we query the TPM for the measurement list. So within, as I say, we typically do polls of around two seconds, you could do more. So within a second or two of somebody running something on the system, if there's been a change to the object that's running, we will know about it and we can immediately then shoot that machine off the network. So remember earlier we sent our white list, our golden state of hashes of what we expect the file system to be. So IMA populates the security FS into the TPM. And then key limer tests the runtime trust state using IMA. And then somebody compromises the machine. There's an integrity failure. So we're now look at the next part which will be the revocation framework. So the actions that we'll take once the machine fails is trust state. So for example here you can see we've got the key line verifier, we've got our certificate authority and then we've got four nodes that are all being monitored. So these could be like an open shift cluster or any sort of type of machine essentially. One machine is compromised. So somebody runs a script as root that's not white listed or they somehow trojanize a file or they do something that tampers with the state of the system. Immediately the verify will send out a certificate revocation using a certificate revocation list to invalidate that certificate. And then if that certificate is used for the TLS connections of a machine obviously you're going to invalidate those connections or it could be used for IPsec or any sort of solution which uses a certificate authority. And then the other event that it does is it sends out these revocation events. And these are essentially, it's a list of metadata and you can place whatever metadata you like into this revocation list. Typically we have stuff like IP address or host name and parts that have failed and so forth but you can customise this list. And this is signed by the verifier so you know that it's an actual real verifier that you trust that is sending this out and it's not a hacker pretending to be a verifier and causing havoc by sending out these events. So yeah, so these events are received by the machines which then informs them to run the local actions and these are these Python scripts that we looked at earlier which can then programmatically do whatever you like. They can make shell executions or call APIs or take any sort of determined action that you'd like to take when a machine fails a state. So for some examples we could remove the failed node from SSH authorised keys. It's a very simple example or we could shut down a VPN tunnel or a Mensom IP table rules. Whatever you want to do, like I say this is all outlined in scripts. And then we can call the certificate authority to make a revocation. So there is actually a demo that I put up recently on the Red Hat community YouTube site and what we do is we monitor three SCD machines that are part of a cluster. We then compromise one of the machines in this instance SCD2 which causes the verifier to send out these revocation events and these revocation events tell the leader to remove SCD from the cluster and it does some other things as well like shreds and secrets and so forth that are associated with that particular node. So if anybody's interested you can watch that video. We also have some examples as well using Libra, Swan and Raccoon. But like I say this is Python. So anything that you can do with in Python you can do on your machine. See it's really is up to you. You can, the world's your oyster essentially. Anything that you want to achieve and automate programmatically you can do so and Keyline will start that for you. So we try to keep ourselves being agnostic when it comes to use cases and have the user really use their imagination around what they would like to do when a machine fails. So a little bit about the project and the community and where we are at present. So as I say we're relatively young but we're continuously growing. Organically contributors are coming along and finding the project which is really nice. And you know the volume of commits is increasing and all the metrics say that it's a nice growing open source project. We meet weekly okay and all of our meeting notes are github issues. We've got a meetings repository and this works really well because we can link to pull requests and comments and then github will automatically reference that we've pointed to them and it really helps tie everything together. And we also have a Gitter. Gitter's like a kind of a I guess like a Slack sort of type message in platform where we meet and anybody that's trying to use Keyline and wants to get it to do something or doesn't understand how something works or wants a bit of support can come along, jump on and we're a pretty friendly bunch and we'll help you achieve or discuss a particular use case or something that you'd like to achieve. So currently what we're working on is our agent so the agent being the component that runs on the remote machine that's currently in Python and we're porting that to Rust okay and the reasons for this are Rust is arguably a bit more performant because there's no garbage collection and so forth and it's also a very good security so the compiler's pretty strict around ownership and so forth so you get good memory safety and thread safety. We're working on VTPM support so at the moment we work with a hardware TPM. There are virtual TPMs but the problem with a virtual TPM is you're back to that issue of your secrets being on disk because the virtual TPM will store, it's emulated and it's running within the virtual machine. So you don't get that true hardware route of trust with a VTPM. So the Geiser MIT and some interns from Boston University, they came up with a proposal that allows us to extend the trust from the hardware TPM to many virtual TPMs but we don't absolutely bomb them the hardware TPM and cause a bottleneck so what we're doing is we're aggregating all of the quotes into a Merkle tree and then we can have a single operation against the hardware TPM which will then allow us to attest many, many, many VTPMs and there not be any sort of throughput issues. So this will allow us mass scale and it extends that hardware router trust. So this is something that's been worked on at the moment. VTPM support is available in QEMU, it's pretty well established and there's a patch into the container runtime as well for VTPM support and that's looking close to merging soon, it's in pretty good state. So we hope in within, perhaps as short as a few months this might be there to try out and use. We've got a prototype that's working that people can look at. Lots of other stuff as well so we're revamping the UI, we're working on a token system to build the initial stages of multi-tenancy so at the moment for authentication we're relying on mutual and server TLS but we're going to use Java web tokens so that we can scope tokens and have some form of multi-tenancy and access control. For the revocation events we're going to set up different levels so at the moment what happens is a node will fail whereas we want to have like a kind of a less serious action level of notify so that means not necessarily blow the machine off the network but perhaps monitor it more closely. We're working on, well we haven't started yet but we're going to hopefully have some sort of repository kind of like a galaxy where people can share revocation scripts that they've developed. These are the scripts that for example shut down network connections or terminate VPN tunnels or update IP table rules or call a part of Kubernetes or a part of OpenStack and so forth. We're packaged into RawHide so Fedora32 you'll be able to install KeyLine. We're starting to look at how we can integrate with Fedora Core OS and IoT so KeyLine I should mention here is it's a very good fit for sort of your cloud computing type scenario of a cloud consumer and a cloud provider but it's also very good for IoT as well so anything that's remotely outside of your trust boundaries that's where KeyLine comes in. OpenStack and OpenShift integration we're starting to look at that and evangelism which is what I'm currently doing at the moment which is getting out and talking about KeyLine to people. So how are we doing for time? Okay we're good, we're good. So this is kind of like a kind of like come and join us type pitch. So at the moment we're looking for anybody to help out.  Anybody that can contribute is welcome and as I said in the previous meeting we try to be really welcome in, inclusive, we support new people. We know what it's like to approach a new project and try to get a pull request merge. So I mean I will personally go out my way to help anybody that's going to contribute to the project. You don't need to be a security guru in case you don't need to understand crypto to a really deep level. We've got lots of stuff around standard login frameworks, little bugs that need fixing, typos, spelling, all sorts of stuff. So if you even know basic level of Python then you can come in and get your feet wet and help contribute as well. And we have other stuff such as Ansible roles that are available and so there's lots of stuff to work on. Our documentation needs a lot of love. So you know there's lots to do and so you don't expect to be some crypto math genius or anything like that. It's really a, that's only a small partner system and that's pretty much complete now. It's just about making things more resilient and robust and mature. So we have a website, keylime.dev. If you go on to there, then you'll find out where our GitHub is, where we meet, where our white papers and so forth, that point you towards everything. And as I said earlier, we've got a community chat channel. If you've seen anything which you think is perhaps a little bit concerning, maybe don't share it in a public area. We've got a security disclosure list that you can report security vulnerability to. We're a young project. We're trying to do things the right way, but as I say, if you've spotted something and it's a little bit questionable, you can always pull me afterwards or we have our responsible disclosure system. Last of all, you can test drive keylime in a VM. So we've got an answerable role. It comes with a vagrant file. Most of you are familiar with vagrant. This will allow you to vagrant up and it will call an answerable hook which will install keylime. It will install a software, TPM emulator. So you'll be able to log in and run keylime and then perform all of the sort of use cases that we have. So we've made it really simple for people to get involved. And we also use this as a development tool where we mount a local code repository into the virtual machine and run tests and so forth. We have got keylime running in a container as well. So you can run it in a container. The only caveat with these two is that we don't have that VTPM hardware trust extended. But if you just want to play with things, use it as a sandbox, develop some stuff, test some stuff, it's perfectly adequate for that. So I think at that juncture, which is quite good because my voice is going, I don't know if you can hear that, my voice is starting to give up, I can take some questions if anybody has any. Hey, you sure. I just have a question for you. Is that going to be a test that the community is welcoming? I just want some proof of that. Yeah, yeah, sure. Good to meet you. Yeah. And so indeed, it's welcoming in here and for each other to contribute as well. So a few questions. I've got some notes. You mentioned we could have a white list of fires we want to control. And I was thinking maybe that would be a little too sweet tax. They provide the kind of safety checks on the fires one. So who? Sweet tax. Oh, I mean the question as well to the microphone. Yeah, so the question was around the white list and the hashes that we have. And there's a system that provides these. You mentioned. Oh, I see you're like an RPM header. Yeah, very good point. So that stuff that we're actually exploring is there are some patches around taking the hash from an RPM header and then recording that integrity measurement architecture which then extends into the TPM. So that's something that we're following because that will make white list management a lot more easy. And the other one as well is with the mutable operating systems. Obviously they can change with RPM OS tree, but they're pretty static to a degree. So with the structure of you have a commit history and there's a checksum for a particular commit release of that operating system, we're going to start to look into ways that we can perhaps generate white lists from those as well to make it a lot more easy to measure. So the sort of vision is that if somebody, I might be reasoning the wrong words here, and there's somebody from CoreOS that can pull me up, but if somebody changes operating system version, they load a different commit hash, that OS tree, then they would have a white list immediately available that they can then attest that that has happened correctly. So that way they can check the security integrity, but also the operational integrity that they've actually got that right particular image point that they wish to have deployed. Cool. Sure. There's the same blogging for RPMB tool to generate IMA signatures for the operating packages as well. So whether lack will be deeper and then people will use it or not. Yeah, I'd love to explore this stuff. So we should connect and have a look at that. Definitely, yeah, yeah. Because this is one of the things with white lists. There's always been the problem of you. You can have like a kind of golden set of hashes of your machine state. And what we typically do is we take init ramfs and then we iterate for all the files using a SHA256 sum. So we build a hash list. But then what happens if somebody runs DNF update and all of those hashes changes? So this makes a lot of overhead for an admin. So this is where we want to find these better ways of sourcing white lists. And it seems like there's a good few things coming along that's going to really make that a lot easier. Cool. OK, this chap here. Yeah. What about the question? Yeah, so the question was around what about verification of the LUX header. OK, I'm not too much of an expert on LUX, but we do measure secure boot. So I believe that has a level of interaction with LUX. Well, it's normal. Oh, I see. If I have an folder and you change your username, sorry, user passphrase, then I can overwrite with the old one. Understand. At that point, I can use the old passphrase to unlock the hardness. I see, yeah. Something like that, we could look at extending into the TPM. So, you know, if we can hash an artifact, then we can cryptographically extend it into the TPM. So, yeah, I have to chat with you about that. Yeah, definitely, yeah. Like I say, anything like this, it would be great to talk about it. And we can prototype some stuff and see what we can do. So this will come when I add to that. Stick around if you've never got this information going on, because the LUX header could be actually encrypted into your... Oh, really? Because I've been shipping since RAIL 7.6. Yeah, because I was the... ...worked on the Red Hat disk encryption, so I've been seeing emails from... We've been able to tie TPM at LUX together since 7.6. Oh, there you go. Yes, we'll stick around. Is this gentleman at the front? Yeah, we've been so much about containers and the integrity in terms of vulnerabilities that can be introduced by third parties with RAIL, like a real image in a container that is not the real one. And now all the vulnerability tools are just looking at the RPMs or package manager versions to double check if there is any kind of vulnerability right now. Do you think that kind of technology could be used for getting some hash and check every single file in a container just to double check that apart from having real, not vulnerable RPMs also, every file in the container can be... ...can be secure by this happen? Yeah, we could do that using IMA. So we're waiting for IMA to be namespaced. That's something that's happening. So we're relying on a few things landing, a few patches. But yeah, you'll be able to build a golden state of hashes of how you expect that container image to be. And then that will be monitored.  And then using the local actions revocation system that we have, you can then, you can sort of make calls into Kubernetes or a container runtime to sort of fail a cluster or a pod or just a single container or. Yeah, so that will be possible but we just need a few patches to kind of land. But it's looking promising. They're making good progress. Yeah. Okay, five minutes left. Any other questions? Cool. Okay. So I can say I'll be here for the rest of the day. So do come and grab me. You know, it's some interesting stuff to talk about. And thanks for your time. And I think I've been straying outside the camera a lot. So I tend to do that. I tend to wander around a lot. So apologies to anybody watching. And thank you.


He says, allow me to share a personal experience, if you will. One afternoon in the chow hall, the cafeteria, a brother approached me and asked me to do a haram that I had made toba for. So obviously, he has something that he was known for doing haram. He asked him to do that. He said, I made toba for that. And he said, I made toba for it. And I declined and explained why. Fortunately, the brother understood. Later on that evening, I was sitting in the cell, in his prison cell, decompressing from the stress of the day. I had grown exceptionally weary of the unit that I had been on. So because in the prison, sometimes, you know, there's certain parts of the prison that are much more difficult than other parts. And you can ask to be transferred to another place. Some prisons are more difficult than other ones. They'll ask to be transferred. He said, it was really causing a strain on my mental well-being. I briefly recalled the hadith of the companions of the cave. I spoke to Allah in my mind, remembering the would-be transaction, a thought occurred. If you know I did that for your sake, meaning what just happened in the cafeteria. He said, if you know I did that for your sake, put me on the bus out of here. He said, not a moment later, I shook my head while thinking, don't mind me.  He's like saying, oh, Allah, I'm sorry. I learned this. Don't mind me.  During rack time, the guard came around with his count. And he stopped at my cell and informed me that I would be going on a bus. I was going there. I bear witness there is no God but Allah. I thought things like this only happened to people in hadith. But it happened to me.


Do you want us to provide a stock video footage library for your freedom family for free? Like we provide a lot of music and graphics? We would love to know if you are using sites like this behind me which is videohive.net. If so, send us links down there to the videos you are creating using the stock footage in these libraries. Don't be shy, tell us exactly your wish list for stock video. Is it explosions? Is it bottom thirds? Is it different kinds of animations like a cute cat sitting on your couch? We would love to know. For example, this shiny new set behind you is one of the 3D sets that we are building for your freedom family. And if I zoom in and start playing, you can imagine what it would look like if you were behind this desk talking about your videos or live streams.  We would like to know what you want included in your sets when we have this ready for release. Tell us in the comments below. Here is another 3D set that we will have ready for you freedom family. This is the ESPN set designed just like the official ESPN set you see on television. We tried to imitate a lot of what you would expect to see on the TV show, but you can put it into your videos if you're doing a gaming review show or something that's appropriate for this kind of set. And we do listen to everything you tell us. One thing you asked for was more promotion. So we created this Google Form, and here are the responses of all the videos you sent us requesting promotion. Today I'm going to feature this video, but before I do, click that i if you want to get your video submitted into this list for a chance to get promoted on the George show. Like I'm about to right now, this video by iGodsoFluigi is a review of Back to the Future for the Nintendo Entertainment System. Let's listen to the video. Roll it! Back to the Future is a 1989 video game released by LGN for the Nintendo Entertainment System. The game was meant to be based on the film, but it became a to see the video. Click the link down there in the description below and give it a like if you enjoy it. I personally enjoyed it, and I found his conclusion to be refreshing. It certainly was a critical review, but you can tell it's an honest review. There was a concerned comment from Fine Gaming Connoisseur and others who said hope everyone at freedom is safe after the gunman attack in Manila. We are safe. This was an isolated incident. No one we know of was involved. Thank you for your concern, and I'm sad that these things happen, but I'm not sure what we can do to help solve these kinds of problems, except to help make the world a better place by doing the best we can to improve ourselves and to teach others to be good people. Do you have any other suggestions on what we can do? Tell us down there in the comments below. Who am I? I am Papa George. George Sunpai and many other names. You, the beautiful freedom family, has given me. I am George, CEO and founder of Freedom, little freedom today, for freedom and the freedom family. And you'll notice I'm using the 2D set again because some of you said you really like this set. So I'll use a mix of sometimes a 3D set, sometimes a 2D set, so we can get a balance for those of you who prefer one versus the other. I am George and you've been watching. Click that I to partner with Freedom and join the freedom family so we can all grow together. You get many perks like position music. You also get epidemic sound, a lot of other access to royalty-free videos, sponsorships and many things to help you grow. Just click the links down there in the description below to get involved in our community, our forums, our Discord chat servers, meet our graphics team, meet our community team, all of that on Discord and the forums. What are you waiting for? Get started. And we will grow together as a family because this is the freedom family. You are part of it. We are all part of it. And we're all growing together. To get more George, click that big F. That will subscribe you to Freedom Central, home of the George show. And PewDiePie gave one of you, Freedom Family, a big shout out. Click that video to see the shout out and to see our new 3D sets for you. And click that video to see what YouTube recommends you watch next.


We are glad to know that you're still there, still the run-up and we did promise you that we're going to bring you Mr. Achika Chude who will be telling us about a stick holder summit that is being or has been held in Abuja It ended yesterday and we want to have some takeouts from that summit Mr. Chude welcome to the run-up this Today Okay, just tell us about let's begin with that tell us about what the the stick holder summit was about So that we get a feel of it and then know how to proceed from there Well, I think essentially it's about continuous engagement with the polity it's about you know civil society's contribution to the electoral process in this country and to the proposal for the proposal of a good governance and Of course, you know that civil society has been heavily involved in the democratic process from 1999 to date And they have invested so much time energy effort In trying to ensure that we put in place processes that will make elections more or less seamless and the credible free and fair and accountable and so the Stakeholder has met it put together by the Nigeria civil society a situation of course with some a major contribution by development partners Was just that to ensure that we you know Push, you know the frontiers and I will continue, you know to work I've been able to ensure, you know better pros, you know electoral process and if you look at the main people who participated in terms of the invited guests Apart from members of civil society showroom that came from all over the country, you know And the friends of station different civil society organizations The man in the eye of the storm That is the man with the most contentious Almost difficult job today in Nigeria Beyond the job of the president and governors. That is the next chairman himself was there Then we had a representative of the inspector general police and then the Representative of intrapati advisory council ipak. So you have political parties Who are primary normally who are stakeholders in the electoral process? You have security Of course, you know that the Nigerian secure police is the lead security agency in elections in the country Others are just there to give a helping hand to support them And then of course you have the inec Itself that is a responsible for conducting the election. So the It was a good representation and then the essence was to look at what has happened You know what is happening and what is likely going to happen regards to the election in 2023 And then look at the possible gray areas If any and to give advice to the inec chairman And so that some of these things that are changeable can be changed And some of these I mean the proposals can be reflected In the way you know election is being conducted. I think I think as always acknowledge the role of civil society In helping them to navigate the murky waters of Nigerian politics, especially during the electoral period You know, so I think the inec chairman was grateful For the opportunity to come before the people and to also Give I am adherence an idea of what I think and what is doing and what it is going to do In the weeks and the months to come So to ensure a proper in the electoral process Okay, they you just mentioned what was what is and what is likely to come And the capacity of the cso's to advise inec and all that so what did you take from what was And what have you taken from what is in order for you to advise inec About what is likely to come. What are the strongest points that you have taken from the panel president? What was in this instance is the fact is the feeling that You know that progress has been made in our electoral process from 1990 date I mean, especially we regards to the rules and activities of inec And so I think there was that general acceptance that we have made some progress Of course, there are so many, you know, there are some areas that still need to be fined tuned But there is a general feeling that Instead of a feeling of despair and despondency There's a feeling of hope You know that we have not done too badly that is in terms of the electoral empire And the response of the Nigerian citizens and civil society organizations But I'm not exactly sure that that's feeling of optimism You know You know that we can also attribute that feeling of optimism Also to the behavior and performances of the political class I still maintain that the weakest link in Nigeria's, you know, political evolution is the political class They have not shown any serious sense of responsibility They have they have their own patriotic and they are not nationalistic in by any stretch of the imagination And we look for any possibility You know any loophole that they can find to undermine the electoral process for their own gain You know, so it's a very selfish You know and the corrupt political class and that is why the country is where it is So we have inec on one hand that is trying to do well You know, we have the civil society organizations that are also trying to push inec To the part of a credible elections Then we have the politicians that are trying to undermine inec And the efforts of civil society organizations over the years, you know, so that is I think where we are now then but obviously we The other things we are looking at, you know, I don't know whether to preempt you but obviously in terms of where we are The inec chairman also had some worries about the one of the two major worries that he talked about One he said the issue of security He said that security still remains to constitutes a problem a worrying, you know, development for Inec Our regards to elections in the country and just I felt that it is due to the responsibility of everybody You know the security forces civil society organizations the media and Nigerians in general You know to ensure that we can have A a a more conducive environment for the conduct of elections in the country That is his first worry. I think the second one also had to do with His worry over the the issue of campaign financing in clear violation of You know, electoral law Of obviously there are caps that have been, you know, that have become part of the law in terms of capping the You know monies that go into campaigns and other political activities And obviously unfortunately that has been observed in the bridge Because you now see stupendous obscene amounts of money being spent by candidates across both political parties And that is something that is worrisome to inec and then the inec chairman You know says that part of the solution would be to synergize more with civil society organizations And there are other groups that are very good at tracking campaign financing And so that, you know, I work and monitor The extent of compliance or no compliance with the law on campaign financing Then he also talked about The pvc, no, no, of course the pvc collection. You have to know the fact that they were delays about that He was giving assurances that very soon that the inec will reach out to Nigerians To give them details and information about the processes for the collection of Cards, you know, you know by voters Then then the last one, I think if I remember correctly, he was also talking about the means of identification for party agents And there was no happy about a situation where party agents are not compliant with the directive by inec We regards to, you know, identification.  So he's hoping for the cooperation I found out called on the ipak representative You know to Look in that direction and the man did say Did say that That they have had issues with them or the political parties that many of them have not Been in compliance, but that they will keep on doing their best to ensure that the right thing is don't wear regards to identification You know at the, you know pulling units, though He also said that is the ipak man that the two major political parties the pdp and and apc Uh, uh, you know a corporate when it comes to issues of attending meetings of a, you know, by the inter, you know party interparty You know Organization that is by the parties belonging to the ipak that they have shown some level of arrogance and disdain You know for other political parties because perhaps they feel that That they are Far bigger do not need any kind of advice from the smaller political parties Let me let me come in here. You've mentioned how You know how you've mentioned a lot of things about the collection of the PVC's and all because you can feel the anxiety in the air with nigerians Can you hear me now? Can you hear me mr. Trude? Okay Can you hear us now? I don't know why he can't hear Can you hear us now? Okay, please speak up. Let's see. All right. Can you hear me? Can you hear me now? Okay, I think uh, there's a bit of a complication really connection Well, you can hear frankly at least right Okay, mr. Trude is missing something You know when he mentioned stuff about collection of the PVC's and all you know because you can feel the anxiety that people are Uh at that point where they don't want to miss the opportunity Especially people who are really interested, you know in participating fully in the f**k coming elections But then I wanted to ask him the INEC has been very bold, you know When they say things about the measures they've put out about buying, you know, how to curb it I wanted to know, you know during the summit if this was mentioned And if there was anything different that was said about it And you know what the measures would be because If we're going to say the truth, you know as well the engines that we are you will see that you would know that vote buying comes In different ways and measures. Yeah, even the way that you least Expect it. I mean these people are very smart about it I would really, you know have wanted to hear if you ask me I think even vote buying has begun, but in a subtle way like you said Ways that people will not even suspect this vote buying has so it would have really been lovely if he had Lent his voice there telling us if INEC chairman meet pronouncements How they are going to deal with these people and the rest of the thing or prevent it at least before it gets to that point Well prevention would be difficult I think when he was mentioning it he mentioned it particularly that on the day of election So maybe before the election nothing much would be done. I think I think I think today's is back. Can you hear us now? Okay, so before we had that interruption I was going to ask you about You know how that the INEC has you know promised to roll out measures to tackle vote buying at the polling units on election day in 2023 and you know can INEC track the spending of politicians during the campaign period possibility of INEC, you know to do something the issue of vote buying. I think if it was also one of the things that INEC chairman talked about when he was talking about campaign financing I forgot to also mention You know that aspect he was exceedingly very worried about that but don't also forget that you know INEC is and that's why the INEC chairman also talked about the special Offensives commission that will be set up to try Electoral related offices and he was really looking forward to that because he talked about the Boding that INEC has you know has to be a Assuming the responsibilities and there was was happy that it was on And there was hoping there to be passed soon by the national assembly And so that a different body will be solely responsible for you know actions that are during the electoral process that are compatible with electoral law and and so So obviously that's why the police were there. That's why they were also invited for the event yesterday INEC is limited in its Uh, you whatever I You know you can do in terms of stopping a vote vote buy Yes, I mean INEC is not an arresting agency. It is not the prosecuting agency That is the responsibility of the security forces. You have the DSS. You have the police and other security agencies So when they see a contravention of the electoral act, especially we regards to put Put buying in this instance And then they have a duty to respond and to arrest people that are engaging in them I mean in that act and I think it was put straight to the INEC By by by the convener of a Psycho-scientific situation which was put straight that question Observation was put straight to the police Where she also indicated that that sometimes when this put buying is goes on openly You know before I mean in the presence of the police officers or security forces and they do nothing about it And then the issue of those that have been arrested in past elections, you know And she asked what has happened to those people. So It is something obviously INEC has indicated that they want to try to resolve But it is not the resolution of the issue of vote buying goes far deeper And that are most involved in the segments of the Nigerian society including the security, the people themselves, the political parties All of them must come together to say look, this is a very shameful Uh, you know situation and that we need to put a stop to it Uh, I don't know this did a question Ever arise in the in the course of this I don't know because you're talking security now I don't know by way of definition Maybe I would have asked whether vote buying is a security issue that only the police has to do but What INEC is advocating for a different body to be in charge of this thing Is it not just going to be a kind of duplication of These things like some people till now in Nigeria cannot understand why there will be EFCC and ICPC at the same time What they do differently that made them two bodies Why not just expand INEC and give it more Departments that can take care of various needs of the commission instead of Getting an entirely different body Well, well, well the issue of EFCC and ICPC is a different discussion of that Potentially the issue of the EFCC or commission is not an advocacy of INEC It has never been If you remember if you go back to the past under the PDP Dispensation When later Umar Musayara came to power as a president of Nigeria He had knowledge that the election that brought him to power was not exactly stellar You know and that he left much to be desired and he was going to do something about electoral reform He set up the wise commission And that came up and recommended the breakup of INEC some of the you know to have other You know activities that have been performed by INEC to have them be performed by other Other you know organizations Obviously he was taking a cue from what is happening in other parts of the world, especially Africa, South Africa Uh, you know where the electoral commission is not as potent as that of INEC and it has not just been It was not and and and that recommendation by the wise panel was as a result of advocacy by the Nigerian people civil society organizations, you know by the media uh, you know to To relieve INEC Of any kind of burden that INEC has had to bear. So I do not think it's a duplication In any sort I think that that is a right thing to do I let INEC face The responsibility of conducting elections so that where INEC defaults would be ready to call them out and to take to To hold them accountable For failures of the electoral process All right, just very recently there's been a court order that came up, you know asking INEC to resume continuous voter registration Uh, until 90 days to the election. How do you react to this? You know court order and what do you have to see about it? Yeah, well, I don't know where the court order came from but obviously there are two contending orders One is reflects what you just said now that asked INEC to continue with the review of a voter's You know register while the other one is the exact opposite that the court declined Are to to ask uh, you know for INEC Are to continue with the voters registration from that this this For the statement of the court, he says that look it's even You know too late because INEC doesn't even have the time anymore to do that. So let's just say that it's a contentious Court ruling and nobody knows exactly what the ruling of the court is since both of the rulings that The media has carried or you know Is that the metrically opposed to one another? But obviously even if it's a genuine ruling by the court It is something that will be applicable not now, but perhaps in future elections So I make we have to take note But first of all, you need to identify You know, you need to confirm the genuine the genuineness of the ruling and then know which of these two You know, it's actually the ruling if it is one that favors And the continuation of a registration exercise to 19 days before the You know a conduct of election Then obviously it's not something that will happen now because we are already there You know, we're already there. So there is nothing to be gained by By that a court ruling by INEC trying to do anything in that direction Well, um, there could have been so many to Questions to ask you but just like in 30 seconds if you may because the time is off From the discussions you had in Abuja from everything INEC said and the civil societies also said and suggestions you brought What is your projection for 2023? How confident are you of the process leading up to 2023 and the outcome thereafter? Well, well the the part of the election in 2023 is true with some difficulty so obviously and the difficulties stemming from the actions or inactions of The political elites that I have already described as irresponsible and uncommitted So they are the ones that try to undermine the process of political gain And so our duty our responsibility not just civil society. You people you guys they have a responsibility to And that is to also monitor the situation and to call out Iran politicians, you know, and their misbehavior And we need to be watchful We need to be very very watchful because 2023 is a very pivotal year And we just also need to get it right and hope that the right choice is made by the Nigerian people So that we can begin to move away from the country where all virtually all the citizens start running their way to one That prosperous and some of us will be proud of Okay, Mr. Shude, it's been an honor having you on the show today. I would like to thank you so much for joining us Thank you, you're welcome Okay Every time we talk to people it comes back to the press as well Something must most like Nigerians will say most gommas Don't gong the press But I think yeah While they are right that we have a humongous duty to the general public Everybody should also know that Not everybody listens to radio or watches television and all that so whatever corner you find yourself advocacy should be carried out And so so that we don't tread blames The media has to do this political parties have to do this iron egg has to do this and all that We have to know that It's a Nigerian project and the Nigerian project is our project If you want it, even if you run away from this Country you will still have relations that once in a while you will talk to and you want them to be all right And you call home I wouldn't want to see it as throwing blames because when you look at it if really truly in your words It is an Nigerian project then it means everybody has a role to play and I feel like if everybody plays their role Then things would you know literally Work out properly and then if you want to say that not everybody listens to the radio or watches tv There is somebody whose responsibility is to give orientation To you know, uh, make sure that that the right amount of publicity that is required Is given and it is that person's responsibility to bring these informations to radio to tv and take it to And every other places that it is required. I'm not calling anybody's name, but you know yourselves and I feel like Um, there hasn't been enough orientation going on, you know for you know in the build up to the 2023 general elections So I am hoping that one day we'll get to have a conversation with the noa guys And they probably tell us what they're doing towards the 2023 because to me They seem very quiet and you see that is also even a problem For them to have to tell us before we know Because they should have been doing things that before now You don't need to ask Maybe you should be asking what what can we also do to help you in what you're doing? Or what else do you intend to do that? We've not seen actually This is a period where there supposed to be very very vocal and very loud screaming at the top of your lungs There was a time I was asking just for fun I asked a few a few people if they know who is heading the national orientation agency right now And they didn't know do people know that there is So I don't even know that that kind of an agency exists and I'm sure they get budgetary allocation each year So what happens to like you said responsibilities that are given to you?  Nobody takes responsibility So if we become the rats Then the things that need to be done would not be done because we're waiting for the the next person to do it So in your small corner, you can contribute something take responsibility Well, let's take a break and return when we return we'll be talking the the bill the Education bank bill that has just been passed by the national assembly. It has not been accepted to but Something has been done regarding that. What do you feel about it? Theophilus akatuba will be joining us shortly. Stay with us


So if I was told my story in 60 seconds I would tell you I have been raised to be persevering and strong. I'm very lucky with the background that I have. I feel like that all shapes me to be the person I am. I was in a car accident in 2010 which left me in the chair paralyzed from the waist down. I've loved horses my entire life so getting back on a horse was not an option. That's what I was going to do. So now I am the only paralyzed professional racer in the entire United States. I feel like that everyone is going to have some sort of a challenge you know and sometimes they can look like mine you might see them on the outside and sometimes they may not. The one thing I do believe that is across the board an option for every single one of us is to turn that that challenge into something that can inspire and motivate those around you. So I am glad that I've been able to open up a platform and open up an avenue for people to watch me not just I don't know overcome life's challenges but even try to succeed right and and be anything and the best that I could be at anything I wanted to pick. There is no point when you wake up in the morning after something and say today I'm going to become an inspiration that doesn't happen. So it's really just how can I survive today and sometimes it's even how can I survive the next minute. So that's a that's a continuing thing. There are some days that I feel like I do really well at this and there are other days still 12 years down the road that this is not what I planned and I call those days the days my lap is round because I feel like it doesn't matter what I said on my legs it's going to fall off and it's just enough to just irritate you through the entire day. So I feel like we all have those moments and I still have them. So it's a continuing battle. I feel like if people ask me how I became a speaker I can tell you it's because of my involvement in the FFA. That was where I really like honed in my skills as a speaker and then of course that was the year that I got in the accident when I was serving as state FFA president. So I gave a retiring address on overcoming obstacles which I had written prior to my accident and then wrecked a month later and then was literally living that speech to get on convention to give that and so I feel like there's a lot that's been intertwined with the FFA for me and I'm very thankful to have an organization that has allowed me to become who I am. The one thing that I hope people leave the room with is the ability to recognize they have the strength to overcome anything that's happening in their life.


Okay, let's see, who am I? I've just been around a bit, I've been in quite a few reasonably decent companies. I've contributed to quite a few open source projects, probably most well known for XFC. I currently work at DirectEye and I'm going to shamelessly plug the place. This is my nice logo because it's got a nice logo if you're coming to work for us. Well, no, other than the logo, it is a nice place to work. You have interesting people like me and probably people more interesting than me. So shameless plug aside, let's get to the meat of this thing. And when I was coming in for this talk, I caught up with a bunch of people and they all said, you know, I can't find any information about the stuff that your dog came by. And you know, it's a good question, because it's paperware, I haven't released it yet. So Varenade is basically, it's an allergic system that I'm writing. I faced certain, how can I describe it? When I was using some previous other tools, I'll address that. There are certain things I didn't like about it, so I started writing this. I was hoping actually to get to a point one release before this conference. This was supposed to be a launch talk. Unfortunately, real life kind of got in the way. So I don't actually have code to release right now. I don't have anything to actually demo. So I hope you forgive me for that, because I think the ideas in it are worth discussing. And I think there's a lot of potential there, naturally, because I'm writing it. But I mean, what I'm looking for here is maybe a little validation for my ideas, trying to get some feedback and see whether there is any real interest. Obviously, there's interest from other people that give me a little more motivation beyond just scratching my own itch, my own problems. So that's why, you know, like for those of you who don't get the Duke Nukem reference, Duke Nukem is the most famous bit of paperware ever. They were talking about it for what, a good 10 years, I guess, before it came out. I hope to be together a little before 10 years. Maybe a lot less than 10 years. Okay. So as the slide, you know, the title side says, you know, Validate Distributed Resilience Scalable Alerting. I tried to fit in more buzzwords, but the slide was only that big. So yeah, to get to the point, there are already tools in this bit. I mean, the tool that kind of owns the space is nagging us. So what's the problem with nagging us? Why not just use that? And, you know, the fact of the matter is that very serious organizations are using nagging us. Yahoo. You know, I know for a fact, I don't know who else, but you know, I think Amazon uses it. I was just talking to the guy from Flipkart, he uses it. I think pretty much anybody who's got an alerting problem uses nagging us. So what's the problem with nagging us?   Now nagging us is, I would be the first to say that nagging us is actually a pretty unique, it's functional, it does the job. People are using it for very complex things, but it is not the nicest tool to work with. To start with, it's, you know, configuration languages, it's verbose. It's very painful. It's not very, it's not very oriented towards a lot of things. It's not very oriented towards, say, generating it. So, I mean, even a, you know, a normal size deployment, it's very easy to go to hundreds of thousands of lines of code. Fine, I thought I was getting some censorship from, you know, God doesn't like what you're talking about. Yeah, so what was I saying? Yes, so the, the syntax is general. Quite often now, the syntax of these things is actually generated by a configuration management system, because people want their single source of truth to be a configuration management system, some kind of post database. Or a puppet or a chef or something like that. And honestly, a lot of the structure of how the configuration works is kind of painful to do in, in, because of its syntax and the fact that it's kind of centralized. Some time back a problem that we were facing in, in direct time, was that the amount of time it was taking to generate the, the, Nyagya's config out of puppet was taking like half an hour, 45 minutes just to compile all the information that we had. I mean, we were able to fix that by going to a better back end. But the fact is that the sheer amount of stuff that needs to be generated was one of the causes there. Right? So what are the other problems other than this configuration syntax? I mean, said mail, for example, has had a horrible syntax for the last 30 years and people are still using it. It's possible to work around that, but that's not the only problem. Redundancy in Nyagya's is pretty hacky. If you want to have redundant Nyagya's, the official mechanism of doing that is to set up two identical instances, make sure the config goes up to both of them, and to use a failover using something, you know, some kind of IP failover, some kind of third-party failover tool to get to that. And that's kind of hacky because there's a certain amount of, you know, state that is in memory for Nyagya's. So when you do a failover like that, that turns together. There are ways to get around that. People are doing, I've got mechanisms for this, but it's ugly, it's hacky, it's more complex than it needs to be. Similarly, load balancing. Load balancing Nyagya's. Now, we're not going into the details of how to load balance Nyagya's. What I can say about the way load balancing Nyagya's, you can take that as akin to how you would do sharding in a database.  So that's not really very flexible because, especially in a dynamic environment, cloud-based environment, you may not be able to do that. You may go in one side or the other. So your load balancing is very static. And, well, yes, you can work around that, but again, pain. So this is a theme that you're going to see again and again. It's possible to do a bunch of stuff with Nyagya's. It is not always easy to do that. Finally, Nyagya's itself, I mean, from putting on my, taking off my sister admin hat, putting on my developer hat, Nyagya's, the architecture of Nyagya's is what our place is, all I can say. I mean, it was designed in the 90s for the problems of the 90s. And everything since then has been layered on top of it and batched in. And a typical Nyagya's setup now, we have stuff that's written in C and C++, you have Perl, Shell, God knows what else mixed into it, right? Every new, a new architectural fix that needed to come in has been layered in different ways. So, you know, you have things, it is, it is complicated. And the various elements of the solution have got different sense of dependencies, right? It's been, you know, so you've got different things like these, NCSA, NFBE are different ways of getting checks executed. Kierman is a way of distributing, adding all that stuff is, it's all add-ons on top of that. Similarly, people have done rewrites for different reasons, for UI to add usability, trying to be backwards compatible. And quite often you have setups where you use a combination of those just to get a reasonable configuration set. And because of that, it's reached a situation where you have Linux distributions, the open monitoring distribution, an entire Linux distribution there, just to help you get Nyagya's setup. And God say, this is something that's just getting, checking alerts and getting the results back. Yeah, it's not a trivial problem, but it should not be so complicated. That's the problem that I have with Nyagya's. And this, actually, this motivation for this, it came from a very specific event, a specific project. A few months ago, I was working with some members of the audience on moving an application from a data center onto AWS, on some web services. For various reasons, we needed the dynamic nature of Amazon. And so this is a thing. So now one of the things that we learned as we were trying to study Amazon and what kind of, the kinds of architectural decisions that you need to take is one is that you have to plan for failure. Amazon is not necessarily a highly reliable, it's not a highly reliable environment. So you have to architecture your solution to just deal with failure as it comes, because you are going to be seeing a lot of failure. However, doing that with Nyagya's is not easy, as I said. I mean, redundancy is difficult, load balancing is difficult. So, I mean, for example, how do you take, how do you hand the back, if the monitoring mode goes down? We have to go through the redundancy stuff that I was seeing, I was talking, well, we could do that. In fact, we're probably planning to do that anyway. But it's not good, it doesn't feel you, it doesn't leave you feeling very comfortable. Similarly, the thing is, as you add nodes, on a cloud infrastructure, you typically be adding new application nodes, database nodes dynamically, adding them, reducing them. Every time you want to do that, I have to send a sig up to Nyagya's. It's a reload, come on. That is like so 90s, man. I really don't want to do that. Ideally, this part of the entire problem of deploying to the cloud, you want everything to be node-driven. The node comes up, it pulls, it gets its function, provisions itself, sets up its own, starts pushing out metrics, starts pushing out alerts as needed. It shouldn't be driven from a central point. And this is, again, the centralized nature of Nyagya's in itself, I have issues with that. So, yeah. So these are the reasons why I thought of writing. So, when I started thinking about this problem, it's only been a few months. I started writing code only fairly recently. Now, one thing, obviously, simplicity. I'm not such a smart guy, so if I'm simpler, I keep it the easier it is to manage it. And I think that's true of most of us. We've got a whole bunch of things that we need to deal with. Let's try to keep all our tools as simple as possible. From a development perspective, both perspectives. Be extensible. Just look at the history of various tools in Nyagya's and how it's been hacky to add on to it. It's probably a good idea that from day one, you should be ready to add new functionality. Don't hard limit the functionality early on. Be configuration management. Everybody's using configuration management of some kind, Chef, puppet, CF engine, whatever, what have you. So be friendly to that. Try to push a conflict to that.  No web-based configuration of alerting, no ways. That's not very configuration. That's not scalable at all. Provide mechanism, not policy. What is that? So this is something like, you know, it is don't take too many decisions about how the system should work because every deployment is different. People running that deployment will have better ideas of what they need than me, upon my algorithm, trying to fix and find out a solution that will work for everybody. So try to make as few decisions as possible. Try to make things as flexible as possible without being too configured. Peer to peer. Peer to peer is not... What does that mean? I struggled with that particular line. I think it's probably something related to simplicity itself in that don't have too many moving parts. Try to reduce them as much as possible. Don't try to have too many code bases out there. So ideally, if each... If that one code base that I have is flexible enough to do different things, that'd be a great thing to do. If any of you have used...  This interacting of being flexible and being simple. A lot of it comes from collective, if any of you used collective. I've also been inspired by things like Senso and Monkly. They're probably not very famous in the... But they've definitely got a lot of buzz, especially Senso. I mean, why I just didn't go with Senso is probably out of the scope of this conversation. We'll probably take it up later for those of you who are familiar with Senso. So... I wanted to draw diagrams, I'm not very good at drawing diagrams, so I'll go with text. Okay, what's the architecture of... Of Baronet? Essentially, on each node, we run two main processes. One is a scheduler, the other is a pipeline. The scheduler, fairly straightforward. It, from the configuration, figures out what checks need to be executed and runs that on the schedule, gets the results back, then just sends the result back to the pipeline. Now, the pipeline receives these check results and processes that will arbitrarily see this, or what are called policy models. Now, policy modules are things that it sees a check result and takes a decision and does something with that. Okay? And you can... Now, policy modules, what are the kinds of things that a policy module can do? It could potentially change a check result. So, for example, say, I get a critical error from one node of a 500 node cluster. Okay? Well, in the bigger scheme of things, that's actually not very critical. So maybe you could have a policy module that downgrades the criticality of... So it could... That's one use case for modifying. There could be other reasons why you might want to... Maybe you'd want to increase the criticality of something. Maybe you'd like to put in more information into it. Not sure. We'll figure that out at one time. That's what I'm saying. I don't want to take too many decisions up front. It could delete something. So, supposing... I don't know, maybe you... Say, supposing you've acknowledged an error. It's known that, okay, I've seen this problem. I don't want to get any more alerts for this. I've acknowledged it. So then I can, for as long as the period of that acknowledgement is valid, just delete the same kind of error as it comes. It could check... It could create further results. Now, perhaps... Let's go back to the cluster situation. Instead of modifying an existing result, you might want to push that result untouched, depending on this thing. And just figure out another error, okay, or another kind of notification. So you could say, you get in the data from multiple places, and based on several different alerts to develop a heuristic to generate another kind of alert. So I've got a database that is giving some problems there. I'm seeing also some disk errors, something, something. I'm going to make a mistake and come to the conclusion that, okay, maybe we have an incipient disk failure coming on. If you can work out a heuristic, shove it in, so it can create a new alert result from that, right? And the final thing is, I was saying something, I'm sure it was useful. Yes, we had last talked about creating check results. And the last thing is just run arbitrary code with that data. For example, send the check result to another machine, to a pipeline or another box, for example. So now my expectation now is that typically on a node that is running some application, I mean that is being monitored, a monitored node, you would probably not have a lot of policy there. You're likely to have only just a retransmit policy node there that just collects and then sends it to some central processing node, right? So that's the general, now the pipeline is interesting enough, can actually receive checks from, can be configured. It can be configured to get results, all being on check results can be configured to be completely distributed. But so like for example, to take an example of something like a Git repository, right? There is no, the Git does not impose any architecture on you. But effectively for any normal development network, there would tend to be a central node anyway. So, yeah, I mean it obviously doesn't make sense to have, collect alerts unless you're pushing them to some central location where you can see them. Well, maybe, maybe not. I mean if you don't want a UI you could just be sending out notifications directly. What are you thinking of them to have the same So, yeah I'll get to that. I have some difficulty. Yes, okay. Okay, so as I said, so policy modules can run arbitrary code. So let's go on with that. So anything, so this is the core of the system. Anything that is not concerned with executing checks or routing those results somewhere is either a plugin or an external application, right? The plugins which I'm expecting which I haven't gotten yet. Obviously, as I said, transmit which is going to send results elsewhere which I assume is going to be the most commonly used one. Maybe one for doing something like flag detection is just an alert that's coming up and down. Maybe a logger for logging stuff. Things like filtering for acknowledgements or down times. Maybe an email notification. Rather than email notification, I probably just do a major duty integration or something like that. Job execution? Okay, so these are job execution Well, yes, so that's something I actually neglected to put in the slides altogether. When the initial release what I was planning to do with job execution is just execute Nagios plugins. So Nagios has got a kind of standard for that. It's just an external executable which returns data in a specific format. And with my initial releases I just thought, so just use that so that we can just use that entire initial library. Maybe I might move away from that later on. It has its own icky bits but we can go with that. But that's not necessarily a plugin. I mean that probably would eventually become a plugin architecture. It is not a plugin in a sense because these are all external executables. So initially Nagios compatible in the first release. And then finally pushing that to some kind of permanent storage. Maybe to a database or maybe a writing to Ajax. So both have got their pros and cons. I'm still thinking having quite the set. They are exactly how In fact, we might have both options for different use cases. Then probably obviously some kind of UI application so that you can see the results and do things like making acknowledges and setting things down and so on. So these are the big elements of the architecture. So few bits of technology bits. Configuration purely in JSON. Use JSON. JSON is a format that is pretty easy to generate in scripts. So generally it's also fairly available by human beings. So if you actually have to go and edit it, you can. My expectation of course is that a lot of the configuration is going to be auto-generated out of some configuration system like Popette or something like that. Maybe with some kind of database system at the back. But this is a nice common format which can be used. Communication between this transmission internally between processes as well as between nodes using 0MQ. 0MQ it's small, it's light, it's fast. It is it deals at a slightly higher level of abstraction so that I don't have to deal with low level socket nonsense but it's still reasonably lightweight. I'm thinking now currently with the earlier reasons I'm just going to be sending a payload of pure JSON. I'm still trying to figure out whether that's a problem or not. My thinking is at this point that with 0MQ and JSON going over it, that's sufficiently cross-platform that the various bits and pieces of the system could be written in other, if other people want to contribute and they want to use they have a different preference with choice of tools or maybe the problem that they want to address is best addressed with a different technology base they can. I am currently writing the code with a booby. I'm kind of comfortable with it. It's a nice language but as I said I'm trying to keep things sufficiently isolated with that maybe later contributors can write stuff in other languages. I don't want to be tied to one individual. Okay, so how's that? I mentioned some goals earlier. As I said, right now we're talking about implementing a design. It's not fully implemented yet but how well does this design approach the goals that I had set out earlier. Simplicity, well since the actual underlying core is not taking a lot of decisions, the core of the system is going to be very, very small. Right now my best estimate is that it's going to come in 100-1000 lines of code at most. It would probably be a lot less than that. The design and we're talking about on the vast majority of nodes we're going to have just a single agent running over there probably as simple as a gem install or perhaps a single IBM install or something. And it looks to be fairly simple to manage with some kind of configuration management. Extensibility as I said. Now, trying to have defined interfaces with the communication so in theory anybody who starts listening on the same, anybody who can understand the same protocol of 0Q plus JSON. JSON itself is a very extensive program. I'm not imposing any expectation on what the contents of that JSON data structure be. Various, the internal processing is going to be this has been written in such a way that it is, it does not make too many assumptions about the contents of that JSON data structure that's moving up and down except for some obviously the certain things that cannot be left out but anything, for example any policy model needs to add additional stuff can do that. I'm hoping that way it's going to be fairly extensible and it can grow as needs change. And I said that internally also most all serious logic is going to be a plugin. So if after some period of time there's a complete new kind of architecture that we need to live with I'm hoping we should be able to adapt to that and grow along with it. Configuration management friendly as I touched. It's all JSON text, it's very easy to manage. Organization, say for example things like how does each user know what checks are relevant to it or how does a notification plugin know which check results to send where. The organization is tag based so essentially tag based can be easy it's possibly I think a good balance between flexibility and excessive complexity. So load balancing so now let's just discussion about the problem load balancing in this architecture a lot of the every logic is actually spread out if you look at an Agios instance a lot of the time it's just deciding which checks to execute and the scheduling logic is all being pushed out to the nodes. So in each node the amount of actual overhead is going to be very less because there's not going to be that many checks on each node. So that's spreading out the nodes I assume expecting now that the central nodes should not should not get heavily loaded I'm now then there comes the bits about how do you analyze the results store the results now if you're pushing into a database with a web based frontend that's a solved problem everywhere we know how to load balance it's not difficult and I am assuming that this is going to be very trivial to do it how do you do redundancy I figured I need to have at least a couple of drawings otherwise it's going to be a complete hand way beat up so yeah so each transmission the very nature of how 0x works makes it very easy to just transmit to multiple nodes when you're transmitting data so just push data through multiple collection nodes each check result will have a to identify it so even if there is a duplication going through the system it will get eliminated when it finally gets persistent so redundancy shouldn't be too complicated to do once once the agent itself understands the concept of multiple head nodes but in this case it's completely peer to peer then shouldn't be too much of a problem to do redundancy the other thing is distributed alerting now doing multi data center monitoring with nagios again there are existing solutions with that it is not very nice in this case since every node assumes that it's going to be able to send to another node suddenly this becomes fairly trivial the other thing is that since we are talking about a simple JSON data structure it's easy to start batch updating within the same protocol so there is nothing to be sending batch updates across it if for example that's the best way to handle say a link links between multiple data centers so here since it's by the very nature of it it's hand-off data across so multi DC should not be a problem we'll have to see how it goes in the real world of course so now that we've seen what bright ideas I have when is this crap going to be available for you guys to use so the core should be done I was hoping to be done before this but a conservative estimate about a month or so then I have to get started on plugins that since I haven't figured out I haven't looked at that problem in depth I'm not willing to put a timeline on that but it's probably a couple of months in the real world the web view I should be fairly trivial to do I'll use something there are lots of frameworks and that's hoping somewhere later on but things that I was trying to do here is sort of try to validate my ideas try to get some feedback find out exactly what are the big holes in my grand thinking and that's what I'm trying to demo here and try to get some feedback so I do give stuff to you guys to play with but I think that will take a little long I'll probably tweet about that as soon as I've got a release and I'm hoping you guys will try that out so criticisms, feedback insults not an insults I hope yeah, go ahead so can we change can we allow this to go in can we change or allow this to go in do these here how do you do it open question okay it's quite, it would have to be done at each node and I'm not going to need to figure out exactly but to ensure that we have something that's actually going to be unique I think they are existing solutions for that if not we'll have to work that out so so yeah so I have to do the harmony there it's also something that I think I'm not entirely thinking about it I'm not sure that it's going to be a problem until we start doing multiple routing and multiple routing that is essentially going through multiple collection nodes and trying to do fully regardment system which is probably something that I'm not going to be trying out very early on so yeah it is something that needs to be fixed but it's probably not my number one project at this point of time is there something that I mean are these so called policy modules things that they question to do from somewhere or are they just there so policy modules if you look at the actual implementation it's nothing more than class that implements a particular mix a kind of particular interface and the way the loading of that works is drop a .rb file where with the definition of this into a particular directory and it will automatically load this up so I'm expecting that what people are going to do is basically push out policy modules using something like puppet and drop them into this and probably a configuration to do is that they can modify the check results or add check results sequence in which policies are executed is significant so probably a configuration on exactly how the policy modules are going to be stamped on the pipeline or sequenced on the pipeline so that will be probably pushed out in percent probably won't be sent yeah I have it is interesting I thought that basically slightly overkill for what I was trying to do if at a later point of time it does look if it looks like that one I'm trying to keep it as the score essentially just going to be calling a callback kind of thing call a thing get a result back out of it if that is since I'm not really looking at things like like forking I'm not trying to do that that would be kind of implicit by doing things like maybe having a transmit module with some decision but at this point I'm not entirely sure whether that's needed so which is why I'm trying to let's not add a lot more features than I really need so yeah I looked at it I thought maybe overkill for now might be worth looking at I got to check I got to check if one is going to be okay yeah so the see there could be multiple ways of doing this but yeah you probably need to do something of that kind it might be that as I said I need to figure out exactly what kind of space I would need for that GUID so what kind of routing would it be okay so the routing so that's why I said I'm keeping the policy kind of relatively free at this point okay okay so in the way that I'm thinking of it right now I'm not so the routing is more useful for things like building redundancy and so on I'm not expecting a situation where certain nodes will have certain functions and other nodes will have other functions so for example it's not that node A is going to be a flap detector node B is a right okay okay right yes yes so okay so I'm so one of the things that the API for this thing the policy module will have is access to some whatever persistence system that you're going to have so it is necessary obviously that your persistence system is going to be accessible anywhere where a policy module needs to access it so those are going to be some certainly it's not going to be a completely free form peer-to-peer kind of thing but I think that's part of the API that the policy module should have so I don't want to decide too many things and advance them more I decide to advance the less flexible policy model but what I would like is so give us much raw information to the policy so I can figure out what it wants and again give me as well historical information access to it so these are things that probably many different policy modules like history is going to be something that so for example a problem when you have checks being fully passive checks that have been pushed out of the node if the node suddenly dies how do you know because you're not going to get any negative results out of that so there's got to be some other way of checking that so a couple of ways of doing that is probably having say another node that's monitoring this node which kind of gets back to the central problem again or you're going to have some kind of check on your head node which would be checking your persistence to figure out when was the last time I got a result from this node so things like this trying to figure out the history is going to be important and history is important for other analytical reasons I mean that's one of the things that I have with Nagios as well is that it doesn't persist by default you have to do things to get it to persist it is so there is a github repository it's a little out of sync right now but by that time after that I want to go into a fully transparent node actually there's no real reason for it not to be transparent but in fact I'm lazy as that well I just have a lot of real life interrupting me but yes I want to do this in the open as I said earlier pretty much I've done a fair amount of stuff open source development in the past I used to describe myself as an open source and evangelist open is better I think going in open will end with result in a better solution at the end of the day people than me will hopefully will not be able to do that I think we are out of time I wanted to put my email address here I don't know if I did but it just says when you guys promise family anyway but yeah viju.ch at directi.com send me mail there viju.chaco at gmail send me mail there viju.chaco on Twitter if you have failed catch me oh yeah I had that question a little earlier so there is a class of lizards called monitor lizards and their genus is marinas so an instance of monitoring is that any one kind of monitor is that in Latin could be called a marinata so this is a kind of monitoring lizard it's a monitor of some kind Latin pun that's what it means so some species can actually come to six that's what I thought actually I didn't know that I was just putting on the monitor part so maybe we can get them to do some monitoring how many of us