Hello everyone. Welcome to our talk. It's great to be here in Amsterdam today and today we'll be talking about tales from an EBPF program's murder mystery.  We'll figure it out soon. So this is basically a story about how probably one of the most important EBPF programs in our Kubernetes clusters randomly vanished and this is a story really about how we investigated the issue and how we got to the bottom of it. So quick introduction about ourselves. I'm Hemant. I work on the computer data point team at Datadog and I'm joined by Will who works as a security engineer on the cloud workload security team and if you haven't heard of Datadog we are a cloud monitoring and observability company and here are a few quick facts about us but the most important one here really is we run hundreds of Kubernetes clusters with thousands of nodes in them and all of them run Selium and we run on all major cloud providers. So Datadog does a lot of things but the most important ones for this talk are a couple of things. So Datadog has a component called Datadog agent which runs as a daemon set in Kubernetes clusters and Datadog agent allows for collecting things like logs, metrics, traces and some other telemetry to power many other products and within the Datadog agent we also have a product called cloud workload security that allows us to detect threats in runtime using eBPF and here's a quick outline of what we are going to talk about today. So we're first going to talk about how our users notice this issue to start with and we'll cover some quick background on Linux traffic control and how Selium and Datadog agent uses traffic control and then we'll get into the core of the investigation and some of the lessons we learned during the process. So it all started with an investigation with an incident of course and we manage a compute platform internally for Datadog on top of which all the Datadog applications run and one of our internal users started reporting connectivity issues to us and our crime scene looked a little more complicated than this because the issues they were reporting to us were super short-lived and by the time we could get there and collect some evidence the issue would totally vanish away. So our internal compute platform users were reporting connectivity issues for some pods and the internet containers seem to be constantly crashing and the readiness probes were failing and there are also some unexpected network policy denials and the whole issue was really hard to reproduce because it was constantly being resolved automatically. So the first thing we do with Selium generally whenever there is a connectivity issue is we do Selium monitor logs and Selium monitor logs is like looking at a real-time CCTV camera footage into your network, into your Kubernetes clusters and you could also use Hubble for a similar use case. So when we started taking a closer look at the Selium monitor drop logs we noticed that we were seeing packet drops for traffic that was going from an identity called 93739 to identity number one and identity one is actually a reserved host identity and 93739 is just one of our endpoints and these packets look like their SYN Act packets so they're actually response packets to some other request. So because it's the destination is reserved host this is actually a response packet to health checks from kubelet. So this is the reason why our health checks were failing and our pods were constantly crashing. So a quick background on how this particular flow works so in order to allow traffic from kubelet to endpoints on the same host, every endpoint has a default policy in place which allows packets to go from kubelet to the endpoint itself so that pods can respond for health checks and whenever Selium sees this ingress connection Selium also updates connection tracking entry so that when the response packet is ready to be sent from the endpoint to the kubelet the response would automatically be allowed based on the connection tracking and it looks like for this case we were not seeing an entries being updated in connection tracking and a quick background of how Selium in our cluster is set up so every pod that is being created on a node gets its own virtual ethernet pair and every pod gets its own network namespace so this pods network namespace is connected to the host network namespace using this vEath pair and by the nature of a vEath pair every packet that is sent on one end of a vEath pair gets transmitted immediately to the other end and Selium also installs a few route table entries and IP rules to make sure that the packets can go in and out of the host or out of the pod and Selium also implements a lot of features using eBPF and one of the most important BPF programs is a program called BPF LXC and it has two sections one is for ingress and one is for egress and Selium uses traffic control to invoke these BPF programs on those interfaces and I'll hand it over to Will to talk more about traffic control Thanks, Hemans. Yeah, so we've been saying a lot today that Selium uses eBPF to monitor traffic technically speaking there are dozens of ways to actually use eBPF to monitor traffic and and potentially modify and mangle with network packets and traffic control is one of them and the reason why we need to deep dive into it is because you need to understand basic concepts about traffic control in order to understand the murder mystery See traffic control in general is a pretty complex subsystem of the newly external that is used usually to shape the network traffic on ingress and egress in other words this means that given a specific interface you can actually monitor all each and every single packets that coming in out of this specific interface it usually works with something we call a queuing discipline so there are a lot of different you know queuing disciplines but the one we care about here is the cdsact one so this one is specifically what is was specifically introduced for eBPF use cases and has two main hook points ingress and egress hook points this means that on these hook points you can attach eBPF programs that we call direct action TC filters they're called direct actions because whatever the output of this program is is going to decide the fate of each and every single packet so you have three different output out of all those but the main ones are those three ones tcact okay which means that the packet is allowed and go through tcact shot which means that the packet is dropped no matter where it was going and tcact on spec which means that the filter doesn't really want to make any decision yet and allows the next filter to make the decision for you filters are identified by handles so these are either hard coded or you know numbers decided by the kernels and allocated at runtime and you can also specify priority levels for your filters so the priority is actually used to define the order of execution of those different filters the lowest priority is actually going to be executed first and at a given priority level the program that was loaded last is the first one to be triggered so with this background let's see how selium uses and configures its own TC programs there are three main things that you need to know about how selium uses TC the first one is that the TC filters always answer either tcact okay or tcact shot in other words this is one way to drop packets and this is another also a way to make sure that a packet is allowed to go through to either a workload or to go out the point here is because of this and also because selium hardcodes the handle and the priority to one also side note this is expected selium you know owns the network data path so it was expected for them to use you know a priority level and the handle that is high enough so that they know that they are going to be to be the first one to be called while a packet comes to an interface but because of this and because we wanted to use tc as well to implement our own network policy use case we had to work around these parameters and figure out a way to still introduce TC filters while making sure they would be triggered and this is why we used so first of all we decided to hard code again the priority one as well because we wanted to use you know the execution ordering rule that says that the latest inserted program is going to be the first one triggered and again this is you know the the rational behind this was that selium would set up the parts set up the networking and then we would load our own instrumentation of the different interfaces we care about so we didn't have hard code any handles because we didn't want to mental or to choose one specific number because you know selium is only one cni odds there out there and other people could be using all their handles as well and yeah we made sure to always insert tc act on spec to be sure that selium would make the ultimate decision when it comes to network packets and that would be you know pretty much it in theory we also added a periodical check to make sure that our filters were still loaded again we knew that other people might be using tc so we had to make sure that once we say that we instrument and drug interface we are still here after a certain period of time and yeah again that's pretty much it on paper everything should work fine except that it wasn't but yeah so at this point we've been to a lot of crime scenes and we started to see some patterns over time and what we realized is that the murderer whoever that was was only interested in new pods they were not interested in pods that were already running at all so all the reports that we were getting from our users were only talking about new pods and as I mentioned before these pods were completely vanishing away by the time we could get to them so we really needed a reproducer so at this point because we understand that this is only happening for new pods we created a test workload and removed the readiness checks on them and just ran it in a loop until we could hit the reproducer and soon enough we had one so and once we had that pod and the host ready to investigate we wanted to answer the question is celium actually setting up the pod network correctly right so we could do this in few ways but the celium version that we were using at this point in time was actually using the tc binary that's part of the celium agent image itself to install bpf programs through the tc subsystem and we could use the same tc command to actually inspect of the interface of the pods interface has the necessary bpf programs or not so tc filter sure dev and the pod network interface would actually tell you whether the bpf programs are installed or not and here we could see that the egress programs were completely missing for this for this specific pod and the ingress programs were totally fine right so we were totally confused at this point and we wanted to understand why so bpf has a rich ecosystem of tools available at disposal so we decided to use bpf trace to try and understand exactly what is happening and bpf trace comes out of the box with a lot of tools that you can use immediately right so we decided to use exec snoop and exit snoop as i said before celium was using the tc binary to install the bpf programs so exec snoop will basically allow us to trace all the programs that are being uh started up on the host so we ran exec snoop with the filter of with a grip of tc filter and we saw that tc filter replace commands were actually being executed for both from container and to container this corresponds to egress and ingress sections and we also ran exit snoop which basically tells us the exit code of these commands and we saw that they were all completing successfully and no issues at all so this totally confused us again so we reached out to the community and created an issue upstream because we were thinking that maybe this the tc binary image that was baked into the celium agent and whatever was running on the host was different and maybe it's some weird kernel thing so we reached out to the community and asked for help and within a day we got some response from paul and daniel and they seem to think that the presence of another ebpf program called classifier ingress security on the host was what was correlated to this uh connectivity issues in the past and they asked us to look into it but we still needed proof to understand why this is happening and why exactly this is happening so the answer to that is writing a little bit more bpf through bpf trace again but this time we had a custom bpf trace program and we had we were monitoring we we had some hook points k probes and trace points on both tc replace add destroy and also on q dis destroy and create and we made sure to log the pid and the probe name and also the user stack at the point where the where we hit this trace and once we ran this we started to get some really really interesting answers and at the beginning here you can see that the from container and to container are being successfully installed by a pid ending with 395 395 and 398 and there were classifier ingress security bpf programs that were being installed by a pid called 2659 and the same pid was actually destroying bpf lx e2 container and even sometimes from container right so what is 2659 it's data dog agent so was it really us all along deleting this and here when we looked at the stack trace it actually told us that we were calling something called flush inactive probes so for some reason data dog agent was thinking this was inactive and at the same time we also realized that data dogs on cws network policy coverage was also missing and something's broken there too so if only celium agents programs are being deleted why was data dog agent also being impacted so to learn more about that I'll let we'll talk about it so at this point we also wanted to understand what other netlink messages were being sent because the tc binary does not use the netlink library directly but the cloud workload securities data dog agent was using netlink binary to interact with the kernel so we wanted to capture all the netlink messages that were being sent to the kernel and we realized that there was a kernel module called nlmon that allows us to monitor all the netlink messages so we built the kernel module loaded it and you could actually run tcp dump on the virtual interface and you could get all the netlink messages but what we realized is that the protocol that was parsing this netlink messages did not have all the information that we wanted so we wrote a custom one that captures all these messages and we'll we'll talk about that yeah exactly so we were still not really confident about what was going on because it was clear that data dog was deleting the program but with all the gods you know the god whales that we put in place and you know the perfect plan that we had in mind um you know everything should be working fine so what's going on why did we end up deleting cillian's filter um so we built another tool to get a bit more context about what exactly was going on because you know the the the scripts that him was talking about were great to identify you know the full-tip program but didn't really provide the context of the different events that eventually led to the deletion so we wrote tcprobe it's an open source project you can have it on github it will you know output more context about the different tc operations that happen on a host so on the right you're going to have the output of the program on the left you have some kind of a graphical representation of what's going on so the race condition happens on starter when a new pod starts and when a new interface is set up by both cillian and data dog so let's see events by event what's happening and how things unfold to uh you know a murder basically so first a new interface comes online and then you know cillian creates a new cls act to disk to instrument this specific interface so far so good this is expected then cillian moves on to installing its own tc programs so this one specifically is on ingress and as expected it has prior one handle one so far so good as well and then cillian picks up on the new interface so we have ebpf programs to register i mean to detect when new vet pair interfaces are registered so we actually get you know notifications from the kernel when a new interface is ready to be instrumented and this is how we detect the new interface and decide to add our own tc programs as well so here you can see that we have our own classifier ingress security program and it's using handle two again we didn't hard code any handles so this handle is actually provided by the kernel and the race actually occurs when we end up being faster than cillian to instrument the interface that cillian created and the bad news is because we did not hand hard code any handles we end up getting handle one because this is you know the first filter on this variety level and the kernel just grants us handle one the issue here is that cillian didn't prepare for this because they did use handle one as well but hard could and because one of the use cases is to atomically swap the filters they use by default you know the flag to swap the filter at handle one which means that they atomically swapped our filter and well to to their point we didn't prepare for this as well because we have a theoretical check to make sure that our programs are still loaded and by default if they're not then we clean up everything that a kernel gave us in this specific case we still believe that handle one is ours and we clean up behind ourselves which means that we end up deleting cillian's filter so that's pretty much you know the the gist of the race and the race condition is really that in some cases we ended up being faster than cillian to instrument interfaces and this is what explains you know the fact that it was not always happening and was pretty hard to do back all right him and take it away thank you so how do we fix this issue today so cillian as of 1 12 started supporting custom priorities for tc filters and cws also rolled out an update that bumped the default priority from 1 to 10 so that people using datadog agent and cillian things wouldn't break for them and if today if you want to use both of them together you'd have to redeploy cillian with a higher priority with a priority that's greater than 10 so what did we learn from this incident right so we learned that tc filter ownership is very racy by design and a growing number of products are leveraging ebpf these days so over time this issue is only going to get worse without explicit coordination so every player that's using tc ebpf needs to follow the same kind of rules so folk should always use tc act unspec or tc act short but never tc act okay because if you use tc act okay the packet would be scheduled immediately to the network card and especially monitoring products should not do this and do not ever hardcore the handle of one because you would not know what other products are using and for power users make sure the priority and handle can be configurable so that then they can mix and match different components and deleting cls act q-disk is also very racy so you cannot do it and you cannot delete it in a safe way so that's for right now but how do we fix it the right way so the right way to fix it is with bpf links and the kernel had support for bpf links for a long time but the traffic control subsystem does not have support for it so there's some work happening upstream to add the support for the traffic control subsystem and i think it'll be merged sometime around may 2023 and there's also a great talk by daniel bachman talking that gets into a lot more details about bpf links and traffic control bpf links and how we could use them and do check that out so in conclusion this was actually not a murder this was just an accident because there were no clear guidelines established on how different people should use them together and this was a complex incident that actually took us several weeks to get to the bottom of and we were really thankful for the community for all the help and i would also like to thank some of my team members jared eric lorant and maxim for all the help and if you're interested in working on weird and fun networking issues like this we're always hiring you can reach out to either of us on email or twitter thank you


I'll trade this to our chart here with me. I'd like to draw your attention to sterling dollar on a four-hour time frame.  Looking for one, two, three, four, five, six, seven, and nine-swing impulse cycle to complete ideally into this one-forty-seventy area. Weekly range resistance one-forty-eighty and those prior support zone which gave way. For prices to fade as we initially test that zone for a three-way corrective move to test support back down to the one-thirty-eight area, from there watch for bullish reversal patterns to re-engage on the long side, ultimately looking for a read test of the prior highs and monthly range resistance up to one-forty-three. As always, trade us, plan the trade. Trade the plan, most importantly, manage your risk. Until next time, thanks very much.


All right, I guess it's time to get started. So hi, everyone. It's good to see you. My name is Greg. I've been working on the SEP distributed software system since 2009. It's been a long time. Last time I gave this talk, it took about 37 minutes, and I tried to cut it down. But we'll see how it goes, which means that if you have a question that comes up that you want to ask, maybe raise your hand to yell out while I'm talking so that in case we run out of time, you get your questions answered. Who here came because they think they want to learn something about SEP? Good. OK, who here mostly does testing during their day job? Who mostly writes software during their day job? All right, good mix. Excellent. OK, so we got three slides that I'm going to do as fast as I can about SEP just so we know what problem we're dealing with. So SEP is a distributed storage system and provides optic block and file interfaces to people who want hundreds of terabytes or petabytes of data. A SEP cluster is built on top of the reliable autonomic distributed object store, which is responsible for actually storing data and serving IO. And that rados cluster is composed of mostly optic storage demons, which actually are involved in serving every IO and doing data replication and recovering from failures. And then a very small number of these monitors, which we really don't need to worry about right now, that just say, OK, here's Susan in the cluster, but aren't involved in data movement. So if you write an application that runs against the SEP rados cluster, then you've got a big cluster. Even a small one is probably several dozen demons running all together. And then the application just talks to it via a library binding, a set of library bindings. So if you want to test the SEP cluster, it's a little complicated.  Or maybe you have a big giant data center switch that you can turn off, but then you don't have a test system running either. So we needed some way to do testing of SEP. And so we came up with Tuthology. Tuthology is the academic study of cephalopods. So it's just a pun on the name. And it started about seven years ago. So in 2011, I had been working as a very wet behind the ears software developer for a couple of years. And it was mostly me and a couple other people sitting in a room together. And I was the new guy. Someone else had started the project several years earlier. But we really needed to formalize step testing. At least once, we had a situation where we ran a test, like FIO or something. And we said, hmm, there was a bug. And we tracked it down. And we said, oh, this if condition is backwards. So we swapped it. And then a few weeks later, one of the other three people in the room ran a test and tracked down and found a bug. And they tracked it down. And they said, huh, this if condition is backwards. And they swapped it. And we eventually realized that they were the same if condition. And it didn't take account of enough state. But because we were just sort of running test ad hoc out of shell scripts on our laptops when we felt like it, then we didn't have a good way of identifying that problem. And so we really needed a solution. Luckily, we hired a guy who was much smarter about testing and administration than I was named TV. And he set out to try and build us a test system. The first one involved using auto tests, but it just didn't work great. And we couldn't find anything that was accessible to us that would work for distributed systems because they were all mostly set up around single nodes. And we really, really needed to be able to say, hey, that machine needs to turn off. Or we need to kill this process running on that machine over there. So TV sat down. And he wrote some Python code. And he called it an orchestra module. And the purpose of orchestra was to let us SSH into remote machines and do things to them. And it sort of presented these machines as Python objects. One of them was a cluster object. And you could run commands on every node in the cluster or a filtered set of nodes in the cluster or get a single node out of the cluster and save that and run a bunch of commands on that one node. And that worked. It let us control clusters. And so then he created the tuthology test running system. Tuthology was built up around orchestra. And its job was to take a list of tasks that you gave it, like set up a stuff cluster and then mount a kernel client and then run some tests on it. And run those tasks on target nodes in ways that are defined by roles, which I will just illustrate here real quickly. So a set of targets is just a list of users and machines that you can log into with SSH without passwords so that the system can automatically log in. Roles are a mapping from kinds of demons into nodes in the cluster. Now, in our examples, these are gonna be SEF based because we're testing SEF, but the roles that are available are not defined by the tuthology framework. They're interpreted by the tasks. And so in this role, we have a monitor on each of three nodes and a couple of OSDs and this is a metadata server and then a client role. And then the tasks are lists of other kinds of Python code. And we got a question. This sounds a lot like your invented Ansible, no one knows this. It's still up. Okay, so I actually know very little about Ansible as a front end technology, but yes. Ansible was not really available at the time. Or maybe it was available, but no one had heard of it. Yeah. So, and then, but like this list of tasks, each of these tasks is actually a separate Python code file. And so the SEF one says, you know, install and set up SEF. This is a one for mounting the kernel client. And then this is a work unit one, which is a generic one that lets us invoke shell code tests. And it's going to invoke it on all of the clients in the system. And this particular one is the dbench.shell script. The tasks are kind of interesting. They can be something called context managers, which basically means that they don't need to execute on one go. The SEF task actually sets up a cluster and then it runs the yield command. And then another test can run after it while the SEF cluster is running. And then when they return execution flow, these back to the SEF task, then it does the tear down of the cluster in the cleanup. And then tuthology automatically can combine individual YAML files into a single test. So you can like have a file that's your targets to run on because these are my machines. And then you just like keep on changing the, the particular test you want to run by changing the kclient dbench.yaml file or something else. And it automatically combines those YAML fragments into a single test job, which I think is familiar in many test systems now, but I don't know, it was the first time I saw it. And when it's done with a job, the tuthology logs all that output into an archive directory you specify. So you have a set of, this is probably pretty small, but you have a remote folder that contains all the logs from all the nodes that were in the system. You have a tuthology log file that's the actual test running output. You have a summary.yaml file that specifies whether the test failed or succeeded. If it failed, it has a reason. Whatever, whatever failure condition it triggered it, the original config that was used, the version of stuff it was run on, stuff like that. Now that was seven years ago and TV was smart, but it was still only, you know, a couple weeks or maybe a couple months from time for him. So it's a little bigger and different today. Tuthology today is mostly, but not exclusively used inside of the upstream CPO lab. This is about 250 machines that are available to the CEP community. I think they're physically hosted by the Red Hat open source and standards group, but it's just, it's not controlled by anyone. It's anyone except for the CEP community itself. And it's devoted to running CEP tests full time and we grant SSH access to any engaged developer. So it's a big system and it takes a little more than just sitting there at your desktop and being like, oh, like run this test now. So first of all, we have a locking server so that we can say, hey, I need three nodes, give me three nodes that aren't gonna get trampled on by someone else. And you can use this to grab a target's file in addition to actually locking them. Second of all, maybe all the servers in the system are busy right now so you can't lock any, so you can schedule a job to run later at some later date instead of running it immediately from your desktop.  Oh, there is. Okay, this job takes three nodes. Let me lock three nodes. Hey, I locked three nodes. Let me execute this job. Hey, I'm done with the job. Let me store the log files. And instead of, yeah. So we have the lock server and we have schedule and instead of actually scheduling individual jobs, we instead tend to specify schedule suites. So suites, obviously, well, suites do test a specific category of the step interface. We have suites for the file system. We have suites for the underlying rate of the object store. We have suites for our S3 compliant gateway. We got suites for a lot of different things. And in particular, instead of being a full collection, a full listing of all the tests we wanna run, suites are actually directories of the ML fragments that we can combine. And this is really useful because it means that if we wanna add a new test for a new API we built up, then instead of having to do all of the different combinations of things we can test against that API, we just add a new API, a new test this API YAML fragment and the system picks it up against all the other stuff. So in this example, this is a trimmed down version of our rados verify suite. And we've got a couple of different YAML configs that are gonna run on everything and we've got a description of the cluster and we can choose whether we want to do thrashing against the cluster or not, which we'll talk about later. We have a couple of different local object stores that run on a single hard drive and so we can configure different ones of those and then we have different tasks. We've got one for testing whether the monitor is recovered for whether the API is actually work and for something that we call classes. And so when we run a run tuthology suite, then it'll first say, okay, well, we run, we include both the YAML files in the top level directory and then I go into the clusters directory and I'm gonna pick out some files and oh, this directory is tagged so that we actually use both files in this directory for all the tests but in the thrash directory, we pick one YAML file so I'm gonna pick the no, we're not gonna thrash YAML file. In the object store directory, we're gonna run on XFS with our file store backend and I'm gonna run the class tests and then the next test is all the same things except for running the API tests and the next test is all the same things except for running the monitor tests and then the next test, we're gonna switch to BlueStore and iterate through all the tests again. So it's a combinatorial explosion but it's very useful because when I add a new thing, it's like, hey, I need to verify that my new, I don't know, that my new thing works, that I add a new task that says, hey, like, test that I successfully mount and unmount correctly if I found a bug that does it 30 times, where 30 times in a row it doesn't work and then it still tests against all the backends and things. We have a whole bunch of suites, so coverage is pretty good and I mentioned the thrash directory so this is a very important part of testing Ceph. Ceph's purpose in life is to deal with failures so we actually need to test it against failures. You may have heard of the Netflix Chaos Monkey, that's a very popular concept today. It probably, we probably read a news article before you wrote this but it's just sort of the thing you have to do so it's one of those tasks that runs in the background and it runs around and does things to the cluster. We have thrashers that you can configure to turn OSDs on and off randomly but only so many at a time so the cluster should still work or maybe enough that it doesn't still work, you can configure it. We have thrashers that let you change the way data is sharded across the cluster or force a reshuffle of the data so it has to move while IO is still happening, things of that nature and these are very important to testing the recovery state space of stuff. So that's sort of an overview of the tuthology user interface but it's also important to talk about how we use it. So first of all, we test the development branch. I can write some code up and say, all right, I think this might be ready or I think this one thing works and I want to make sure that this one thing works but I'm not ready for a pull request yet so I can push that code to our cefci.git repository and we have a system that automatically builds packages out of anything that gets pushed there and then I can schedule a job in the CPLab to test that code and I can get it to email me or I or anyone else can just go look at our Polpito website that reports all the jobs, passes and failures and let you look at logs. Second of all, after branches are tested by the individual developer, we test every pull request that comes into the system. We use GitHub, we get a lot of pull requests from people that we don't know and a lot of pull requests from people we do know but don't have quite as good testing as we do so we have to test all that before we merge into the system. Tech leads and people who do a lot of reviews have some tools to help them but basically we go through, we make sure that a branch is in good enough state that we think it's worth actually testing on. We merge it into an integration branch with three or four or five other pull requests and we run them through whatever suite is appropriate or whatever sets of suites are appropriate and we look for problems. And again, those results are publicly available if there are no problems then hooray, we can merge that code assuming everyone agrees. If there are problems, we figure out why we go back and report to that pull request and hopefully we get an update that goes through it again. But it's very important, nothing merges into the CEP project without passing the appropriate set of tests. So our master branch, things get in sometimes but it's pretty stable. And finally, we actually do nightly testing. So CEP's a big complicated distributed system. Sometimes there are races that will only turn up one in 10 runs or one in 100 or even one in several thousand. And so we run nightly tests all the time against the in development code. We run nightly tests against the stable branches that we are supporting upstream. It's long-term stable, things of that nature. Depending on the tests, they'll mostly run every, they run between every other week and every night and those results are also publicly available. So that's all the ways that tuthology is great. There are some problems with it, some gaps that I'm gonna tell you about. But I do wanna, before I tell you the awful things about my project, I wanna say, okay, I think we're pretty good at testing. We do a lot of functional coverage. I've sort of run through just the framework about that we use for it but we have a lot of specific stuff inside of that framework to make it work well. We have this CEP test rados thing that can issue arbitrary numbers, like we run it for hours at a time of rados operations, of reads and writes and more complicated things against the cluster and verify that the results are exactly what we expect to get back. And we run that against all kinds of thrashers that do all kinds of terrible things to the system. We deliberately inject failures with like two demons. We inject failures within demons. We have code in the CEP base that we can set to just trigger an assert at important places and make sure that the system recovers correctly. We have code that can inject delays and delivering messages or in granting locks to try and expose those races. We go out and as it says, we fiddle with the raw objects or with the raw disk state and we make sure that the cluster detects that and recovers it in the ways we, and then recovers from it or at least responds to it in the ways that we expect. And doosology is not the only way we do testing. We have a make check build target in our repository that runs, we have, it's not as good as we'd like it to be but we do have unit tests that test actual like code modules built around the G test framework all the way up to user interface tests that turn on a very small local stuff demon cluster and issue all the commands that you can issue to the system and make sure that that reacts correctly when you issue the command or when you issue the command incorrectly. Some of the other components in the system such as our stuff ansible which we use for deployment in most cases or the stuff volume thing that we use for provisioning they all have their own test frameworks that they get used for. But with that said, there are some pretty important gaps that we've discovered over the years. First of all, doosology handles demons by SSH into the test node and directly invoking them. That gives us a lot of cool features very cheaply like the ability to issue standard in and receive standard out and signal stuff without going through hoops but it means that we don't test well now it's system D or in the past upstart or says five-minute scripts. And sometimes that can be a problem like when you expect your script to only allow three restarts in 10 minutes but actually it's just restarting infinitely and so when there's a failure that it causes an assert that you expect to kill the demon it just never goes away. Second of all, doosology does its own package and solves and cluster configuration. Now when doosology was created in 2011 we really expected that most of our users were going to write their own puppeter chef scripts and so this wasn't a big deal. But as we moved forward we had had the step deploy thing we now use a lot of stuff Ansible we may in the future start using a lot of Rook which is the Kubernetes operator for storage from the CNCF and other partners like Suza uses a thing called Deepsea and none of these package installation and cluster configuration systems get tested in our nightlies which means that they might be doing something wrong and we don't ever notice it. First of all finally, well not finally performance testing is very important we're a storage system like we're a distributed storage system so no one expects us to get 100% of every IOP on visible to every client but they do get angry if we degrade performance by 15% from release to release and we really don't have any way to do that in toothology right now. And finally scale testing is not really feasible it's like toothology's integration testing system we have that expectation that we'll finish within hours or else the scheduling doesn't really work right and we want them to run as little space as possible because we always want more space. There are some proposed solutions and some things that we've actually done so we aren't doing it yet but we are talking about this week or yes this next week we're gonna talk about starting to do things like we wanna expand the toothology framework API so we can restart and signal using the init system instead of issuing commands like hey kill all dash nine on this particular note over there or hey run seft OSD with the sets of arguments on that note over there and make sure that it doesn't die. We are talking about on Wednesday morning although we haven't announced it yet we're talking about writing a new API within the toothology framework so that we can request installs and then they can be satisfied by a thing that installs using stuff ansible or a thing that installs using Grook and Kubernetes or a thing that installs using deepsea and that the rest of the tasks don't care about that and just get given an install and we can switch to new ones as they become important and we actually test that these behave correctly. We don't have a great solution to performance testing right now. And so in the short term we actually have a new performance suite which is running the performance test that we do have from another solution but it runs on random nodes in the system so we're gathering data right now and seeing, hey, can we have this performance test fail if it's 10% slower than we expect or is that gonna turn up every third run? Things of that nature. In the long term, we really need to do some kind of analysis solution based off of the performance numbers and the nodes that they run on and the variations that we've seen and do in alerting but that'll be some kind of machine learning thing which we just don't have the expertise or knowledge to do right now. Finally, for scale testing we made a decision that actually an integration test framework is not a good place for long running scale tests because it's an integration system. So both in the community and within companies that build products around it we've got groups that are designing tests to run longer term in their own lab allocations. We might slice out part of the CP lab for some of these from time to time. They might, we might get it spaced from heart from partners or from downstreams but that's the immediate solution and we'd also like to in the medium term figure out how to mock up tests that simulate longer term systems. For instance, in CEPH usually the amount of data in the system is not where our problems come from. Problems tend to come from things like our S3 compatible object solution having too much metadata for the system to handle correctly but the actual data objects are fine. So we can start injecting very large metadata indices and then making sure that those behave correctly without having the data to back them up but none of that exists yet. Finally, there are some weaknesses about Tuthology itself in order to the framework itself not just how we use it. Right now, Tuthology has become strongly tied to CEPH. Orchestra should be easy to use elsewhere it's just a control system and when TV built it he really did design it to support any other multi node system but nobody else uses it so it's sort of gotten stuck a little more together. I say should have what I mean is to avoid this problem may or may not have been worth the expense but it really would have been good to find at least one other community to use Tuthology with us when we built this. If you have this problem today you should find something that exists instead of rolling your own. At this point we're not really worrying about it too much like there's stable code bases that fit our needs switching to something would be monstrously expensive and get us very little but we do occasionally keep our eye out and say hey, is there some other distributed storage system that could use a test framework? Maybe you'd like to work with us although we haven't gotten any takers yet. Another problem with Tuthology is that it's strongly tied to the actual CPO lab. It's supposed to deploy anywhere and there are other people running it but sometimes hard coded values like an expectation that you can pull packages from a CEPH.com domain sneak in and if you have your own development system because you have an old fork or something then you can't pull it from CEPH.com you need to pull from your own local service. The documentation is limited like we have a group of people that maintain and run the service and every several years go and go have a frenzy of updating for some portion of the documentation but there's a lot to work through if you actually want to set up a system. So, right, okay. So, but it's more than just other groups. If you need to write a test and debug it then you have to actually write the Python or local machine, push it to a repository that the Tuthology framework can pull from. You need to have machines locked. You need to invoke that and you need to wait through the machines to get imaged and for the test to install and to run before you can get any feedback. You need local, or you need built set packages now when TVAC initially wrote this we actually just all standardized on a particular version of a distribution and so we could build CEPH on our machine with just make and then it would SCP the files over to the remote nodes and invoke them directly but that became infeasible as we got more people and as we needed to test more distributions and more combinations of things and so we need packages now but that means you need to wait for CEPH packages to appear and all this stuff makes it hard for third parties to contribute and for new CEPH developers to do anything with their patches because they write a patch and they're like, I don't know what I can do. I ran and make check but I don't think I tested it at all. There are some solutions. If we had to do this again and this was important to us we should have made regular teardown and set up part of what we do. We have a lot of infrastructure and configuration as code but we don't invoke most of it very often. We have in the past found groups that came to work with us and they had trouble with setup and install and we sort of did a one time walkthrough with them to help them get going and we didn't take a lot of that back or and make a changes to the system to prevent it being a problem in the future. We made some changes like the packages deliberately but maybe we should have considered how people who didn't have lab access would be able to respond to that and done some kind of workaround or bifurcation or whatever. And in the past we actually had someone write something called tuthology open stack and that was pretty cool. It was a simple script you could invoke from wherever and if you had access to an open stack cluster like including a public one it would set up a tuthology instance and run a suite and then shut down and archive the logs to some storage system. So you could just do one shot tests but it had a few problems. The person who was most interested and it moved on to other things the open stack APIs at the time were ludicrously unstable and unworkable so it used the open stack CLI commands directly but those have also changed since then so it no longer really functions very well. There is at least one group still using it but they forked their using well they did fork tuthology to use it and it's diverged pretty wildly at this point. We have some better news there's a thing called vstart runner. So in the Ceph repository there's a script called vstart.shell and it turns on a local Ceph cluster just on your laptop or your development box or whatever. And the vstart runner provides you a restricted API a subset of the whole tuthology framework API and if you stick to that API then it can run it against those local demons. There's a lot of tests in the file system tree or in the Ceph of S file system suite that use this there are some tests in the Rados Gateway ofm S3 and Swift compatible object store suite that use this and it means that as we hopefully transition to more of the tests being built against this API then external first time contributors can start running tests with it on the thing that they most care about. And we are investing now a lot more work to foster a community. Well I just started a Ceph test weekly meeting a couple of months ago to discuss things. If you're interested it's Wednesday mornings my time in the Pacific coast and they said that URL and I'm gonna start sending out reminder emails. We had a Cepholicon conference several months ago in Beijing that sort of sparked this. We met a bunch of groups that we had never heard of before that were using Ceph and several of them were like oh yeah we have a development team and we were on tuthology and it was kind of a pain to get going but now we've got it hard-coded to our needs and there were like four of them and they had all done their own localization port and so we're trying to work with some of those groups to get those patches back. I mean that maybe not directly because they took their shortcuts too but figure out what those patches were and what we need to change in the upstream system to make it work better. And eventually we haven't started yet but there are now stable OpenStack APIs. I think LibCloud is the one that's come up and that actually works now and we'd like to rebuild tuthology OpenStack so that it keeps working and it's easier for people to deploy systems and that'll have other benefits for us too like the ability to test the tuthology framework itself in an OpenStack cloud instead of rolling it out to our infrastructure and hoping that nothing breaks. This is sort of the performance thing that I talked about before. We don't have good trend analysis tools in tuthology. We can robustly fail individual tests by looking for core dumps, by seeing if there are errors or warnings in the logs, by having a task run a test condition and failing on whatever arbitrary thing you cared about in Python. It's easy to look at the individual runs whether they passed or failed. Sweets show up as green or yellow or red and then individual jobs do the same but there's not a lot of granularity there to say oh a particular job has failed the last 10 times you've run it or hey this branch is broken or whatever. In the future, hopefully we get to fixing this but so far it's just not something that's been this hard enough but a lot of people would find this to be a problem. The sweets explode in size. Like I said, it's combinatorial so as you keep on adding yaml fragments it gets really, really large. The main rados suite is up over 124,000 jobs now. And there's no way to prioritize those tests within the foray work. There's no way of saying oh this is a very important failure to analyze new users don't really or resource constrained users don't know what they should run and you just sort of have to watch it and know whether a test matters if you're trying to get faster feedback. We do have some solutions. Well it's a couple years old now but there's a subset functionality that you can run so that instead of using all the combinatorial tests it makes sure that every fragment gets used but maybe not all of the fragments. And so you can specify hey I wanna run subset one of 200 and as long as you then go on to run subset two of 200 and three of 200 and four of 200 all the way up then you will eventually get the entire combinatorial tests and all of the individual pieces get run in every suite. But it lets you run them down much smaller so you can get a reasonable test system out of 397 jobs instead of a couple hundred thousand when we use this. All right yeah so that's that. We also have other things so you can filter that so when you build a test then it gets a name or a description that's just the names of all the YAML files so you can filter against, you can filter and only take the tests that match a particular YAML fragment or you can exclude the tests that match a particular YAML fragment.  And in the future we may do more stuff but again it's not become a critical issue yet. What is a critical issue for us is that scheduling is very primitive. Like I said it's a beanstalk queue and jobs just sort of get picked off and on the main like test node system or the test running system we've got like I don't know the exact number but it's something in the neighborhood of 50 tuthology worker demons that just sit there and go hey do you have a job for me? Oh hey you've got a job for me. Let me lock two nodes or lock five nodes or lock 10 nodes and if you lock five or 10 nodes then you've got 49 other people competing with you and they only want to lock two nodes. They always win. And sometimes because it's just a queue we'll have nightlies get scheduled but you know I get impatient and I jump the nightly queue with my test run and then a week later the scheduled nightly job is actually starting to get picked off the queue but at that point you know it's the stuff shot one on master branch from a week ago and we're running it through now but oh by the way we just like 20 minutes ago scheduled a new run on this week's code and that's just not very useful because eventually the queue gets backed up enough that we start killing nightlies but we've tested our old code and not our new code. There are and this is sort of an illustrative of an issue so we've gotten several hacks and we talk every several months about how this is a problem but then if you ask someone what problems they have with deutology they forget about this one because our minds have sort of been molded to it so this is something you want to be aware of when you're building test systems. Right now all of our jobs run on two nodes because you know we can mostly squeeze our jobs down into two nodes and then they all and then they actually get executed. One of our queuee people looks at the Beanstalk queue and says oh it's like 7,000 jobs long now and we have three rados nightly runs in there so I'll just kill the two oldest nightly runs. In the, soon it's not done yet but a really easy fix is to do something a little more intelligent with the locking where the workers can order up and queue for locking and say hey I need five nodes and then we wait for five nodes and then those get locked before you try and execute any other tests. In the future we're not quite sure how it's gonna happen yet. We've talked about actually just switching to Kubernetes and it can solve some of these problems but we really need to write a more robust scheduler or install a more robust scheduler so that we can do more intelligent things. Oh yeah, I've only got like two minutes left. Good thing I'm almost done. And then we wanna be more intelligent about the way our nightly runs work. So are we gonna stick with it? Well yes, we're keeping our tests. It's proven effective over seven years. We're incredibly stable for the things we're doing. We've got it running right now. It doesn't take a lot of maintenance to keep working the way it is working. But we are actively exploring ways to make it better for new users and developers. Should you do automated testing?  There are a lot more of them available today than there were seven years ago and a lot of projects have over the last decade written custom frameworks. So if you wanna build your own framework, you should think real hard about it. Are you really sure something doesn't work for you? If you're really sure then like, isn't there something that does mostly what you do? What do they do? Why is what they do insufficient? And what makes your needs different from other people's? You need to answer all these questions before you build your own testing framework or you're doing the wrong thing. If you answer those questions and you're really sure you need to, do the simplest thing you can. Small frameworks can always be combined into larger ones and try to keep your components discreet so they can be replaced later. That is the end of my talk. For more information, a lot of URLs. And we have like 30 seconds for more questions. Do we have a microphone? Hi, so I mainly work on Fedora QA. So I'm generally interested in integration testing. So something I notice often comes up with systems like this. I'm guessing the system is mainly built to test a potentially broken Ceph in a known good environment. Like you have known good Fedora environments, known good Ubuntu environments and then you run an unknown Ceph code based on those. Is it possible to sort of invert the flow so you run a known good Ceph in a possibly broken environment? And if not, have you thought about that? We haven't thought about it. At this point, we freshly image all our machines for every test because doing that, when you're trying to test your system, it's really obnoxious. But you could build broken images and run the test against that and see what happens. Right. So you can't at least sort of draw the images. I mean, it's not something we've ever discussed or actually tried, but yes, we could drop in a broken image and see what happened to it. Yes. How difficult is it to deploy tuthology on how to test the Ceph cluster? Question is how difficult is it to deploy tuthology to test the Ceph cluster? Yeah, so it's pretty difficult. I haven't ever done it personally. If you go way back in time using the tuthology OpenStack 1 made it pretty easy. And I think maybe if you go grab SUSE's fork, it still is. But I haven't actually done that myself. And there's a whole lot of services running here that we just sort of tangentially pushed on like the service that builds Ceph packages, the service that hosts and serves them up, the POPEDO service that stores job results. It's not as easy as we'd like it to be. But if you're interested, we would love the input on what makes it hard. Do you get 100% test passes on commits or do you aim for something lower? Do we get 100% test passes on commits? Not quite. We just run enough jobs that honestly, we notice whenever a package repository goes away. And so it's not unusual to have like one or two jobs out of a couple hundred fail on that just because of some network blip or connectivity issue. When there are failures, we audit them and we don't merge anything with new failures. But sometimes we do have known failures in a system that are pending on some other PR and we merge in that state. That has punished us in the past sometimes pretty badly. So like there's a reason people say you should be 100%. We're a lot closer to it than most systems like us that I know of, but we don't have a 100% programmatic requirement. It's all human driven. All right, well, I'll be around wearing a Ceph shirt for the rest of the day and also tomorrow. So come grab me if you want to have any questions and thanks for your time.


say hello to Joel and Juliano on the talk on BMC firmware management. Hey so thanks for joining us today. This is a bit outside the Linux user space. The basic talk is about how we actually manage a lot of hardware that enters the data center and lifecycle and to do that we built this toolbox and that's what I'm Joel Rubello. I work at booking.com with this guy. Hi Juliano Marchines. Yeah so what we're trying to cover here is there's there's a bunch of challenges when dealing with bare metal at scale and this this introduces like various problems that you try to solve with different kinds of software and the problems that you see with different vendors and so what we're talking about here is the baseboard management controller the toolbox that we built that tries to work with these controllers and just some projects that we ended up building on top of the toolbox. Yeah so some of the challenges is that so now this booking is in a bit of a weird position so we're not like a hard-paced scale like Google and Facebook but we are also not as small companies so we have a huge number of servers but we cannot just start using open compute and we cannot stick to only one vendor so we got in a weird position where we had like four different vendors about 50,000 and growing count of bare metal servers. We would like to treat servers as light bulbs so in the past it wasn't like that so people would treat hardware as pets so for them it would be pretty tricky to move between servers and so on but we were able to move past this this point. We would like also to simplify the vendor validation and adoption right so as I mentioned so we are growing sometimes we need to validate a new vendor and then to adopt so how to define standards to know if this vendor will be good enough to work with us. How to inventorize this data right I will mention a bit later but it gets it gets interesting that sometimes because of network connectivity and so on you might lose hardware inside of your data center. Reliably interact with the hardware that's also interesting because when you have all of these vendors and a huge amount of hardware and you try to do IPMI sometimes it gets funny right so rebooting a machine sounds really simple but can get really complex and also reduce the mono-intervention so our team to manage these bare metal servers has five people so if we just to access a BMC right so because it needs to load the interface and if it's a chassis interface it's a it's a hardware that was developed like 11 years ago so it won't be fast so it can take like 11 15 minutes maybe just to open and access it takes a lot of time. Just to expand on like treat servers as light bulbs the idea is that if something fails we don't want to spend more than even 10 minutes looking at it because this is a lot of hardware. The failure rate of hardware like in the day center in the first year itself is 5% so with that many servers you have a lot of hardware failing so if it fails put it in the broken bucket take care of it later. A bit to show how it works right so you receive a hardware you need to inventorize it first then you keep while people are not using the hardware you keep it always up to date to make sure that when it goes to production it has the latest firmwares and everything else. You install this machine people use it if they don't use this hardware anymore or if it's spent more than x days running production it should be sent back to you to be repurposed and after three to five years we would like to retire this hardware so this is how it works the life cycle of the hardware for us. Yeah so that kind of covers the challenges I hope that gives you like an idea of the environment and the kind of problems we see when dealing with this kind of hardware so the next up we're going to talk about the baseboard management controller so the idea with the baseboard management controller is it like sits alongside your server right or your jboard or jbo for this data center hardware provides out-of-band access and you've probably heard of this a lot lately in the news the chipset there is what like a lot of vendors use like super micro and quanta it's an AST all of them like use software development gets brought by ASpeed which is then modified to display like their own logo and but essentially this is like a software on a chip that has on a single die the flash and VGA video graphics controller and then it has access to all of these PCI devices over the south bridge and it can also interface with the the nick itself and send packets by itself and because it has like this access to all of these components you when machine itself is not responding or maybe the data network you through the out-of-band connection you can actually do a whole bunch of stuff so and since it has access to all of this hardware you can also invent rise the hardware the whatever is on on the server itself and you can trigger various hardware actions like in here as a last resort for things yeah and and how to access this thing right so we have IPMI that's it's kind of common across all the vendors and IPMI so it has evolved over time but not much so if you go and you check for instance the IPMI to common the source code it's pretty interesting and for a lot of things it needs to do magic right because although is a standard it doesn't mean that vendors will follow it properly right and a lot of things that you need to get data via PMI there will be raw commands right so and then it gets a bit weird that you need to know all the raw commands to get information for specific things for every single vendor these are the other issues with IPMI and I hope that in the future we don't need to use anymore that's the hope with Redfish I'll talk about it later there is SSH which is better for our case our cases but there are some actions that for instance if you are using UFI with a specific vendor and you trigger PXC and reboot through SMASH CLP that SSH it won't work so you need to use IPMI there is the web interface as I mentioned so there are different versions and vendors and generations right so now it's getting better and better but if you think that the hardware life cycle is like from three to five years we always need to manage things that are not as awesome and deal with buggy and slow hardware from the past those Redfish that's the future but not just yet so it's much better than when we started BMC Lib BMC Lib actually started using Redfish and then we decided not to use it for now because it wasn't really adopted by the vendors and there is the undocumented APIs that we use it a lot so for instance the same APIs that are called by the web interface we ended up using these APIs retrieve data right yep and about the BMC to boss right the library and tools so the library so what what we decided to do is that we started with Redfish and it was nice and so on and one vendor had support for almost everything but then we couldn't get the license of the BMC of this vendor and then we saw that on the Redfish is back there wasn't definition for licensing okay cool so then we can actually fall back this part and then we check another vendor okay so and then this vendor doesn't expose the hostname but okay cool so then so the hostname I will fall back to the to the old APIs cool so now I'm retrieving the network interfaces oh this vendor doesn't support the network interfaces yet and okay cool so then Redfish it will be a thing but at least one year and a half ago it wasn't there so then with what we decided so let's let's make a library that we'll use the best available connection method to do what you need and we will ensure that regardless of the vendor that you call the behavior will be the same because for instance some vendors you need to reboot and PXC and others you need to PXC and reboot but then we ensure that when you ask something like that we do in the proper way so so that it works so this is an interface right so we have different providers for each interface so when we want to integrate a vendor this will be this will be a common interface that we will use so for instance for the BMC interface we have configure we have BMC collection and here we have PXC ones right so how how does the providers look like so these are the ones that we have in our master branch so we have like super micro X we have a dummy provider for tests we have ILO we have iDRAC9 iDRAC8 and for CMC that the chassis management controller we have for M1000E and C7000 this is also interesting because sometimes if you are using a chassis it's much more interesting to trigger actions through the chassis than the MC itself because they become more reliable so things if you need to if you are using pizza boxes the way for you to remove machines from from the power is that you need to go there and pull the machine right if you have a chassis you can actually do a virtual power receipt so actually remove the power and put it back on and this is actually get pretty interesting for us so here's an example of PXC once with iDRAC9 so with iDRAC we saw that we can actually use the SSH and it works all the time so what we do is that we issue an SSH login we connect to the iDRAC we should we issue the command to to set the PXC and then we put it on and if everything works with PowerCycle and then this machine will reboot properly right and then with HP so with HP we saw that if you are using UFI with Gen9 servers and actually you cannot you cannot use the SSH and there is a bug also that what happens is that you cannot just PXC and PowerCycle the machine because it will fail so you need to power on the power on action will fail but if you don't do it it won't work right so then so actually so this this type of things what we did is that we encapsulated that so that you don't need to deal with that so we used to have a code base in as part of our CMDB that would need to know all of these things right and it doesn't make sense to have it as part of the CMDB yeah no so the other bunch of functions that is available in the BMC interface is the configure actions the configure actions that are actually used to manage configuration on the BMCs so why would you really want to do this right well mainly passwords so we had machines we had machines that were running for like a long time with ten-year-old passwords and also like accessing the BMC interface itself is it doesn't have a certificate and so it's it's not trivial to have all of this done in a standardized way across multiple vendors so okay so let's have this into interface that you declare a bunch of functions that should be implemented per vendor so for example like this one is the ILO the HP hardware essentially you can generate a CSR on the on the device that is the BMC and you sign that certificate and then you upload it back onto the BMC so in HP it's it's very simple you just fetch the CSR and you post the resulting sign certificate but with super micro it's different because super micro actually does not support generating a CSR on the device you have to generate the CSR yourself sign it and then upload so the whole idea with this these examples are that basically that we encapsulate all of these different differences into these functions and yeah you don't have to worry about it yep and what we support currently right so BMC Libs support data collection for Dell HP and super micro ILO 3 actually I would like to remove it because it's too old and we don't have this hardware anymore but it's it's there and for configuration support we also we we do it across all of them yeah and with some of the redfish support this will extend to most of BMC's but yes at some point the tools so after we built the BMC library the idea came of building tools with these things right so the first tool actually so the the tool that inspired BMC library it's Dora at the time it was called Thermalnator so it was a tool to collect thermal and power data from from the data center to understand how much heat we were generating across machines and then from that we saw okay so this is actually interesting right so if we can collect that what else can we know right and if you look at the payload right so there are some some interesting stuff there we have like some faulty slots so it means that one Dora connects to a chassis and tries to read the information if we have blades inside of the chassis that are failing we will have an array with the number of the slot so then for our data center operation team they don't need to keep searching for things inside of the data center they have a list with everything that should be fixed so these also collect the information about the blades fans nicks PSUs and storage place that we have there right so then it actually it summarized issues things that we know as well right so for instance it has a status so you shouldn't try to provision a machine if you know that this machine is broken right because it means that it would generate toil for us then we can use the the status information from Dora to know if this hardware is usable or not another thing that it helped us so imagine that depending on how your a cabling is done at the data center for us is that we have two interfaces right so one for data and another one for our off-band to make sure that these things are isolated and we have a chassis so the chassis receives one cable from Dora of band and it spreads on 16 connections for every single BMC that we have so the BMC is always reliable so we know that if the BMC connection is there we'll be able to collect information about this machine and after we deployed Dora we found out like old hardware that has been bought in like quite few years back and there was like wrongly cabling on data and this hardware was was missing in the data center so this also helped to generate like a dynamic inventory of what you bought and what you should have received so now we use also that information to know that the deliveries that we get from vendors are actually the things that we are buying. Also that Dora is able to scan the whole data center in like a few seconds. So the other thing is that we used to have a similar process but this process would take a week to scan the data center after we moved to Dora it started taking six minutes so it's much faster and then because some vendors are slow and so on nowadays we can do in 24 minutes but it's like 24 minutes we scan about 50,000 assets. Yeah the other tool that is part of the BMC toolbox is BMC bottler. The idea was that this will be like a tool focused on just configuration for BMCs like I mentioned earlier but in this case it essentially just retrieves data from the inventory that is Dora and starts to attempt to configure those BMCs. So in this way we are able to like try to maintain a consistent state across all basewood management controllers and it can do a whole bunch of configuration like you saw in the interface itself. Usually it's not that you cannot configure a BMC but if you think about the installation life cycle what you would do is that every time that these machine probes are installing the first time you set out the configs right and then imagine that you need to update a C name for your C log server or even you would like to change the password. To move it backwards it becomes tricky with every single vendor and now with bottler it becomes also much simpler. Another tool that we built on top of it it's actor. So the idea of actor is to be able to trigger actions across these BMCs and CMCs. So if you want to PXC a machine so instead of using IPMI or calling BMC directly actor is both an API pretty simple that you send a payload telling what you want to do and for instance if you would like to do a sleep right so you want to power cycle sleep for 30 minutes and power cycle again you can actually send this as a job and it will take care of it. Another useful thing that it can do for you is to take screenshots right so if you want to take a screenshot of the BMC to understand what the state of that machine it can do that for you. Yep so we covered the tools and the control itself and this is just something that we want to share that we did in the hackathon it was an interesting idea so you guys might find it. So you saw this picture earlier but basically it's machine life cycle right you have a machine transitioning between these states and you want to make sure that this transition is actually successful or you want to know that if the transition was not successful which state machine is in and we're talking about 50,000 machines here like and we reboot or install a hundred or two hundred other. So our flow to install machines is that now we are in a process of making sure that machines can only live three weeks so sometimes we have like from 200 to 1000 and to 1500 machines being reinstalled and reboot actions it's a lot because yeah so hardware is not reliable so one thing that you do a lot is reboot machines. So the idea was that could we grab the screenshots and try to classify what state the machine is in because to like a sysadmin I can look at that screen and I know that is the the bioscreen and when we update firmware on these boxes they would a lot of them would get stuck like that that Dell screen there is it being stuck on that so it takes a lot of time to like look at each box and why so can we automate this right so if you go back so there's like different stages there's the OS install like you could probably identify that for the so send us OS install there's also like the there's a pixie boot screen there and there's a grub screen so these are all different stages the boxes would fail in so we we grabbed all of these screenshots and tagged them and basically we trained this model completely like you know I don't know what I'm doing but here let's try and oddly enough so we got actor to like grab the screenshot run it through this train model and then give us a probability of what where this machine is stuck in or what is it stuck in and it was quite successful like there was a 97% success rate across like close to 200 different checks that we did so that was quite high and it was just fun to build this yeah it's interesting because at the beginning we didn't believe that it will work that well but it predicted a lot right so it actually so if you think that 20 minutes so if you if you can so if like 12 or 15 of these and get stuck during the day so if you cut 97% oh it's a lot of time that we can actually do something interesting yeah yeah that's cool and what are the next steps for for BMC to box so one thing that we would like to do so now we have a lot of API service and services right but these work for us but if people want to use that it's not crumbles on but it's a bit difficult right because you have all these these different components that you will need to understand every single one of them to start using so what we want to do is act to create a CLI that allows you to interact with all of the services then it becomes easier to use also is the same thing for the front end right so one thing that's useful is that if you want to generate a report of the number of machines that you have with faulty PS use or the number of machines that you see that have odd number of memory right so this is a easy one to to see that you have a fater right so if you have odd odd not number of memory it's very likely that there is one socket or one die that's failing right and also integrate Redfish for generic vendor support using GoFish so we started working on GoFish so this is actually a pretty nice Go library actually it was the best one and the first for Go that can parse decently all data format and we implemented a lot of the features that are missing so what we will do is that we will have a generic driver for for BMC Lib but it doesn't mean that it's gonna work across all of the vendors so for instance to give an idea right so when you go to to the iDirect 8 the Redfish version that's there is view 102 so with this version you cannot collect disks inside of a machine for instance okay so then for this type of hardware we cannot we can use Redfish right for a Direct 9 it's a bit better and it can actually tell you if your link of the data network is on or off so it's more useful but it also has some missing bits same thing with ILO, Supermico, Quanta and so on one thing that happens is that I think as soon as vendors evolve and starting use more Redfish it will become like a default standard but it's I think that we need to keep the drivers for all the other vendors forever. The cycle at which hardware vendors release hardware versus DMTF releasing Redfish specs is different like you get specs more often and hardware vendors take a long time to implement that spec so they are basically behind all the time and they are also making mistakes in the implementation or implementing it incorrectly so hopefully we get to a stage where has constant some constant in all of them and then it's usable. Yeah but what we will do is that every time that we integrate someone so if something doesn't work with the generic Redfish we will have a way to to retrieve or set this information as required so this is a good example with the certificate management and not every single vendor supports that but we can always do it through the through the APIs that they call on the web interface. Yeah and how can you help right so it's open source please use review submit bug reports if you have vendors or hardware that you use and you'd like to contribute it's actually it's not difficult if you follow the interface there are tests to help you to understand and we are there to help as well and getting touch if you are if you work with ironic mass or just a bunch of bare metal servers. Yeah so we've looked at some of the implementations that ironic and mass and these guys use to interface with the BMC's and they even foreman and so they have a Redfish.py and then it's just calling an IPMN command or it's doing a Redfish call but essentially there is a lot of context that is involved when you're interacting with BMC's if you want your action to be successful and that's why I think that it makes sense to have a separate library for this. Yeah in a summary right so why we started with the BMC to box right and put the message that way we want to pass so the idea is to simplify management and adoption of multivander hardware ensure that you have the same behavior and state to provided interfaces reduce the bare metal troubleshooting time more reliable bare metal interaction and single view of hardware health across your fleet. Yeah there's one tool which we didn't mention here is we have something known as BMC FW update we've not completely tested it in all hardware we do use it internally but because we need more feedback and the only way you can actually test firmware updates is by having access to the hardware so if you if there's people interested then have a look at the toolbox there's a link and maybe give us your input on that. So yeah do you have any questions? Thank you guys. So I developed some stuff in iron regarding this I spent like two years doing this and I was like a disaster after this so I had one very specific use case which was driving me crazy so I had 30,000 super micro boxes and it wasn't x10 and what was happening in my case throwing random numbers around up from 10 to 20% of the BMC controllers were just going down randomly when issuing a lot of commands into them because all the stuff I was doing to go through this workflow it was using purely IPMI tool so no SSH no anything more fancy and then you know you have a workflow which issues 10 IPMI 2 commands after the second one the BMC just dies in the box and this is something so fundamental that once it dies you cannot revive it without going there physically sending someone physically and you know if you have 10 boxes it's pretty easy but if 10% out of your 30 or 50k fleet starts behaving like that what do you do what what do you do in your scenario in your setup because I never found a solution for this and for me I still keep this vendor on the list you know kind of if I can decide not to buy them I will not. So when the hardware yes there is like bugs all over place we found some you send packets to HP ILOs and they would just drop off the network completely so this it's kind of tricky to say that which vendor you should pick because essentially all of them have bugs and this tooling tries to abstract away those issues but with Supermicro I think with most vendors you need to like have this constant relationship with them where you give them feedback and try to get that stuff fixed and you so some vendors have been trying to use our toolbox to validate the information they provide through the BMC so instead of them giving us the tools maybe we build the tools and we tell them to use this because it's more standard but I don't think I answer your question of like which one you should pick and that's but one so the library in most of the code I guess just a set of random rules how to behave you're having a specific vendor yeah do you think we as community of people doing exactly this stuff from that side so not from the vendor side have some means to impose more standardization of vendors because they do not care at all that APMI should be standardized they all do whatever they want and they they are in this position up whatever happens anyway you have to buy us so can we do something so they start behaving in a more mature way this this is what we are we are trying to do with the relationship with them it's always feedback about the issues and so on but if as you said right so if you check the code from IPMI to you see that it's tricky because there are some things that it's fixed in the code because it's easier so this is how we do as well so we keep reporting bugs and a lot of them get fixed but for instance they release cycles like three six months right so you're not gonna wait like six months for the bug to get fixed right so you need to keep installing these machines so what we do is that we create the rock around and make it work as soon as we get the the fix we change the behavior but I think that keep feeding back this information it's essential right because if they don't know they won't fix it yeah so if you buy a lot of hardware you have influence on who you're dealing with and how you what you're negotiating for and this is what we see when we have meetings with our vendors we bring all of this up all of the bugs that we faced and then that kind of gets them to like get rolling and fix all of these things and then yeah yeah but it's tricky I am Ed from Packhead we have a lot of bare metal hardware so this is near and dear to my heart we've been working one level down in replacing the BMC software with tools like open BMC have you met with that community to figure out how your tool set can validate their work and vice versa and if not let's talk about it yes so we actually so we were we were doing a proof of concept with with OCP and one of the things that we would like very much it's actually to to be able to contribute to open BMC because we wouldn't need to wait for people to fix the bugs right we couldn't fix we could fix them ourselves but because of the number so we are now we are not as big to start buying these things right so we are not building data centers so we have we rent space but it's not designed for this type of hardware so there are constraints right for instance so during our tests we we tried to to put a rack and it was too heavy it wouldn't pass through the door so with the hardware that we can actually use open BMC and we would be more than happy to use it because we can manage that and in our domain but for some of the hardware we are not able to do that just as a follow-up the other piece of things we're working on is a project called open 19 which takes a lot of the ideas from OCP but puts them in a standard 19 a track format which by design fits in the same sort of rented data center space that we have access to that you have access to the world of standard data center shaped stuff not purpose built data centers so we'll talk afterwards also to answer one of the first question was that so are we integrating with open BMC is what I saw is open BMC is implementing redfish as a service so depending on which spec they implement we will have support for them yeah yeah I just want to iterate I think you spent so much time working around the box of the vendor firmware one should really push to get open BMC and if you buy a lot of service and you have so much power to that the vendors already talk to you that they fix firmware box I think you everybody who buys service here should in the request for offers should put in there that they want open BMC or you BMC because in the end the as you said that the BMC is a fully fledged operating system and actually it's better to standardize it also and I know Facebook with their open compute stuff they I think they actually run CentOS on the BMC and then they can use all their tools and uses quite normally so yeah we should all put pressure on the vendors there's no reason to have different BMC firmware set all okay hey so just last week I was looking at another library for managing redfish it's called go redfish from Nordics which is some kind of open source foundation so take a look at that it looks cleaner than the go fish okay and I have a question about the screenshots so how do you take a screenshot out of band I know you can take a serial over land but not the graphics or so the vendors expose an API which they use to show the screen preview on the BMC's interface so it's a vendor specific yes yes it's all vendor specific and because we have BMC we can abstract so this is so for instance right so if you look at the code of BMC Lib because we couldn't get these things throughout fish right so then we reverse engineer right so so when you open this page so how does this get this screenshot where it comes from oh this is in point okay cool so then and then you see that oh actually to retrieve this type of hardware I need to set these X headers okay cool so we do that so with newer generations and talking with the vendors things get better but everything that sold they won't fix right so for instance so there is there's another thing that we had to implement because we would we would like to use LDAP but if you just check LDAP configuration across all these vendors is a nightmare so it's actually it's simpler to have a proxy right so you you make the BMC call your LDAP proxy and you do what should be done properly because your life's gonna be easier and this is what we did some there some of them don't even implement LDAP tree right so it's a software that was written like 12 years ago so how they they say so this thing it's going end-of-life next year so we can we can make it better in the next one but this one we cannot fix and maybe you should request the redfish had to have the option of requesting a snap screenshot from the VMC yeah so I think I've seen it in the OEM section for HPs I don't know if all of the other vendors should be standard yeah the same thing with the license yeah yeah cool thanks for joining us thank you