fferent type of different type of calling. However, we can However, we try to replace it with source span function in yeager and we Modify the functionality of yeager api to start the span instead of using yeager Using blocking the spans and then we have the event and key valve function in blocking the event function is used for as for annotating The span with the timing stamp and the key valve function is used to annotate the span with With an integer value or with a string value However, on the other hand open tracing api support block kv Which which is used to annotate everything that we want. We can we we don't we are not limited to very specific type of Type of annotation we can have any annotation. We want using Lock kv and then open tracing have any tracer Which which which is used to attach the trade the application The application component to the tracing engine which is presented in the system However, because blocking is not is not using an always on method and we have to turn it on and turn it off We don't have such a functionality in blocking. The other thing is Is that for propagating the information between two different components? We have the inject function in open tracing api and the corresponding one was encode trace in In blocking api. However, it was not again a one-to-one mapping and we need to modify some And we need to have some modifications to to be able to add To add the inject function the other one is the code trace which is used by the receiver of a request and extract the metadata from the other node extract the context data from the other node and then Build a new trace based on that and it is again. It was not a one-to-one corresponding between extract function and the and and the Decode decode trace function and we again have some modification to make those possible Take a look at our implementation What we have done was that we first want to have the least amount of modification that could be that could be added to the system so if you look at the figure one by one on the left, I Present the blocking architecture and on the right. It is the Jaeger architecture We have safe components and we didn't modify any safe components in in our first round of implementation And we have safe components the same then we have blocking api and we use blocking api both for Jaeger and for For our Jaeger functionality and for blocking functionality and we replace the blocking functionality underneath to to instead of inserting to instead of inserting Information into trace points into lttng It sends them to Jaeger agent and then we we enable Jaeger agent on each node and we connect the Jaeger functionality to to our Jaeger agent And Jaeger agent again sends data to Jaeger collector actually after the blocking api We Removed everything and we replace them with the Jaeger functionality Our modification has 200 lines almost 300 lines of code However for adding new trace point because we want to be compatible with open tracing api and we want to compatible with open tracing with open source community So for adding trace point we adding new trace points We are using open tracing api and we are not using blocking api for new trace points that we are added to the system And what was so hard in our modification? There was no contact consistency on context propagation and we couldn't have exactly one-to-one mapping between the Between the components also safe but to be able to to be able to provide Context propagation safe modifies the api of safe functions But it was not what is desired and we implemented to have a more advanced feature And here is an example of traces as you can see in this figure each Each line shows different component and each column shows different component each line shows a single Span and the lengths of each line shows the latency of this of data span as you can see It's it's a little bit small but In here you can see that we can also we can also able to annotate the latency of each Span within the within the trace the other thing that you can show is that the hierarchy the hierarchy that is visible through the span is the is the causality relationship and happens before relationship between different Terraspond and spans in the system Take a look at our evaluation For evaluating our work. We evaluate disk overhead We evaluate memory overhead and we also evaluate the cpu overhead We want to evaluate because all tracing infrastructure comes with an overhead And first of all we want to see what was the overhead that we impose to the system So our experimental setup for our experimental setup. We had two type of nodes one of them was the note that we set up the It was a physical note that we set up self cluster on that and it was it has 64 gigabytes ram. It has to turn you to cpu it has 10 gigabit per second nick and it has two hdd drives and then for the collector note that we want to collect every Traces from the system together. We use a virtual note that has 32 gigabytes ram and 12 virtual cores and a virtual disk We run our experiment for 15 for 15 minutes and the reason was that we during that 15 minutes We issue several read and write requests However, there are some background activity that happens in the step and for those activity We also generate some traces. So we want to be able to capture those races Then we define a time frame of 15 minutes and we capture And we get our statistics during those 15 minutes Is the disk overhead for writing and reading data and also for the For the background Overhead as you can see the x-axis shows the time duration of 15 minutes and the y-axis shows that trace That the amount of data that was generated due to tracing And we run our experiments for different sampling rates for 20 person sampling rates for 50 person sampling rates and 400 person sampling rate and as you can see it was not a linear relationship between different sampling between different sampling rates Because it's a probability based model and then the other thing that is visible here is that this This model of sending data that we send nothing and then send data and then send nothing and then send data is come is coming from the from the fact that we have the We have caching on both agent side and on the collector side We also look at the memory overhead that was imposed by our Infrastructure as you can see in the figure the x-axis again shows the time of 15 minutes within the system And then the y-axis shows the memory overhead of the memory overhead in megabyte in megabyte and again We run our mechanism for our tests our evaluation for 20 percent for 50 percent and for 100 percent of For sampling rate and as you can see the amount of the amount of memory overhead Was slightly different between different types of the between different types of sampling and it was not a huge overhead And it was less than one person one one tenths percent of memory overhead overall system Then we look at also cpu overhead and for looking at cpu overhead again We the x-axis shows the time which was 15 minutes and then the y-axis shows the cpu usage in percentage And as you can see we run it for different sampling ring 20 percent 50 percent and 100 percent and our sum our maximum Maximum cpu overhead was almost eight percent for the for the collector nodes For the future work. We are seeing to have several future works here. First of all, we want to use the out message to We want to use the out log message because Because cpu code has a rich body of log messages and they also they They almost have log messages all over the code on all components We want to use those log messages to annotate our traces because we believe that those information are really valuable and Reproducing them needs a huge effort. So we we prefer to use those log messages This is one of the first effort that we want to have as our future work The other thing is that we are we are trying to add more trace points on different components. For example, currently Steph has several components, but not all of them has trace points We we try to add more trace points to client side to be able to capture more sophisticated and more complicated traces The other thing that we are interested to capture is synchronization point because in order to be able to present a correct view of the system We need to know when our concurrency is finished When our concurrent threads are merged together and provide a single A single and reach to a single point and we need to capture the synchronization point for that And also we want to have we want to be able to find frequent patterns within those Traces within those traces to be able to have a better debugging To provide better debugging opportunity for the users because for example if we can be Retrived if we can be able to retrieve frequent patterns We can detect we can detect what was happening wrong in other patterns But we don't need to look the holes the whole traces to be able to show that we can We can look we can look at specific parts of the trace to be able to Detect the anomaly in several traces And as a summary we enable advanced end-to-end tracing in seph using yeager We replace blocking with yeager and we kept blocking trace points and As there were and we add new trace points using open tracing api our overhead cpu and memory was less than 1 percent And our discovery was less than 60 megabyte per second Thank you Is this an open source project? Is it possible for me to Take what you've done and put it on a seph cluster and then run it? We are planning to have it upstream and yes, it's good. It's it's open source project because But aren't all the bits already available somewhere on github? They are available in my github, but yeah, but it's not an upstream code of seph I mean I can share the repository with you Yeah, it's all available. Thank you Did you actually make the modifications to yeager or was it all using? You know adapting to the api We only adapting to yeager api, but for adding synchronization point We need to have modification to yeager because currently yeager Yeager is not able to capture synchronization points, but We want to add this feature to yeager as well Because it's very important for us You've mentioned that context propagation is hard and it's slow. It sounds like it slowed you down That was resolved. How much faster would it be if it's already a w3c trace? Spec open and I suspect that's going to get approved probably by the end of october Would that make this easier in the future? By context propagation For context propagation There is one thing that is that was so hard in Seph actually the way that they implement blocking and have instrumentation point in seph and it was They modify seph api and I think They modify seph api and it makes things much harder to get rid of those parts And what we are thinking to use is that actually this is available for yeager We can use thread local storage to keep the metadata information In local thread and then retrieve those information whenever we want to create an span instead of modifying all over the code And change every api in the code Because currently this is the way that they it's implemented. It's not very good And one more question in general Distributing tracing is typically not mainstream today because it's very hard to add it to your code This is kind of a first example I've seen of someone that had tracing and tried to move to a different tracing Which seems like an even additional layer of complexity, right? Because you're kind of rewriting something you've already done Are there any tips or lessons learned from that going forward that may be useful for others? The actually the good thing about open tracing is that open tracing actually open tracing is Is the way to answer this question And it provides a standard api for any tracing infrastructure that you want to have underneath For example, you have a specific function and that's api and you use that api for for tracing your System then there is no matter what tracing infrastructure you are using you can use yeager you can use another tracing infrastructure But since you have the same api you can have different modules on the system different software on your system to be instrumented with the same Using the same api and you you don't have the difficulty to going from this yeager Going from this tracing infrastructure to another tracing infrastructure Where does the sampling happen like it in one of the pictures? It looked like you were doing all the trace points and then they were just getting stripped apart so was less storage space or Actually yeager provides several type of sampling You can have you can one of them is the remote sampling that is imposed by the collector And collector decides which no and collector decides on the sampling rate And this is what we have evaluated on our system, but there is another Approaches and there is another types of sampling that they provided through yeager that enables each agent to sample by itself And the different sampling again Well, so but what I mean is when you're doing sampling does that mean that only 20 of the trace points actually get turned on Or do all they got turned on and the data they generate gets thrown away or Where's the cpu if we if we enable it if we enable it on the agent side we for example take One of take every tens of the if we define it as 10 percent We get every tens of the trace points and then propagate those Generate actually trace trace points for them and propagate them But if it's enabled by the collector as the way that we evaluate our system The trace the agent collects all trace points and then sample them and send them back to you to the yeager agent So the key to getting this really efficient is to do the sampling at the agent. How come you couldn't do that? Sampling at the agent. Yeah, we decide which node which actually we decide on the Request to be sampled and then send send the metadata It is sampled on why didn't you use that's the key to getting really low cpu overhead right is too So how come you didn't use that approach? I actually I I didn't have much time to run more experiments because this is the only so is it feasible Yeah, it's feasible. It's just a configuration file that I have to specify I see, okay. It's different configuration that we have to impose to our system I'd like to can you fit this into the broader context of Why you're doing this like is it's just for an alternative debugging thing or What's the use cases for collecting tracing information? Collecting information tracing information used for many things one of them one of them is debugging The other thing is that performance evaluation of the system for example if we want to evaluate What is the what is the if we change if we have an optimization on the system If we have traces from previous execution and if we know how traces was executed previously and with our modification with our Optimization how traces was how how request was executed and what is that traces now? Then we have a better comparison between the two and we can evaluate our optimization better Yeah Any other question Sorry, I couldn't hear you Any more questions altered. Thank you very much. Mania. That was great talk Hi everyone, can you guys hear me at the back awesome? uh, so Welcome to the systems engineering track. No, which track is this? This is soccer quality Welcome to this and ensuring software quality track at dev conf us. Thank you everyone for being here The next talk is property based testing in python and rust by ann mulhorn and i'll hand it over to ann now Thank you. Okay, thanks All right, so you i've been introduced And um now i'm going to talk about who I am I'm actually primarily a software engineer Am I not on I am on Okay, good. I am primarily a software engineer, but um I don't have a naive confidence in the constant correctness of everything I write And so I care a lot about using the tools that'll help me Catch my mistakes design flaws and so forth. And so if i'm serious If i'm writing it. Oh now there if i'm writing a serious piece of software I'll have some sort of continuous integration thing running and then I care a lot about deploying static and dynamic tools That'll help me get stuff right essentially and um So static tools can be many things and dynamic tools are often testing And that's what i'm going to talk about today specifically a particular kind of testing commonly called property based Okay So why do I think i'm qualified to talk about property based testing? Well, I discovered it a long time ago because I was using the language haskell And haskell has the very first implementation of this idea called quickcheck And I was really Impressed by this idea. It seemed very powerful But as everybody here knows haskell has not taken over the world And so when I stopped using haskell I left this really cool idea behind And in fact, I didn't trouble myself much about that until I was working in python in 2015 And I had a library that I was constructing that I knew was absolutely riddled with bugs and that it screamed for property based testing And with I typed quickcheck python Into the google search box And a few seconds later. I was looking at something that looked really really promising And a few hours later. I was running a bunch of tests using this library And the bugs that I had known were there were flying out everywhere okay, and that that's a Somewhat sad story in its way, but I was very impressed with hypothesis I went from not knowing it existed to using it in a few hours and getting really good results Then I sank into the open source quagmire, which open source advocates like so much to talk about I seized to be a user Entirely and also became a contributor So at first it was the usual things their docs were Maybe a little lagging their development or something and I wanted to contribute to that and then I had some actual needs for hypothesis strategy as I'll mention later and I So and I enhanced that and so forth and so on and time went by And I gave this is now my third talk about this subject that I've given and sometime in 2017 I was invited by the hypothesis developer To become a co-maintainer I was invited by the hypothesis developer to become a co-maintainer and I was actually quite proud and happy to become this co-maintainer and From then on I really haven't contributed to hypothesis at all and that's Sad for me a bit because there's a lot of interesting work going on in hypothesis but the reason for that was because I was working much more in rust And with a little more optimism than formerly when I started working in rust I looked for a property-based testing library And I found one and it was pretty nice. It was called quick check and we used it and Got some benefit from it very clearly but the one I'm going to talk about is Prop test which is younger newer and which is Hypothesis alike explicitly hypothesis alike And from my point of view a much more friendly and easy thing to work with Okay Okay, so you know that I'm a hypothesis booster here The question might be do I actually use this stuff and the answer is yes, I do Okay, so I've written a bunch of these little python libraries here And with all of them I've deployed hypothesis and you can see over here That my code coverage statistics can be quite good And other places somewhat inexplicably to me They're not so good and um some places I don't know So payudev is something that I took over from a previous developer and um I seem never to have really followed up to get the coverage statistics I'll talk a little bit more about the influence of payudev And this one here the one below the double line Is in rust unlike all the others which are in python above the double line And getting coverage statistics in rust is necessarily harder than it is in python And so we just don't know that yet although I guarantee you that it is less than 100 um a point I want to make here is that Um in the open source world I often hear people talking about code coverage is just kind of a metric So people will say, you know, it goes up. It goes down. We feel better if the number is higher We feel worse if the number is lower Um I tend to think it's something more than a metric especially if you can have a well known number A well known coverage number 100 is the most convenient but others are good too And the reason for this is that when you're deploying And there's property-based testing that I'm going to tell you about It gets this coverage in a Very useful and interesting way so that if suddenly your coverage drops That doesn't just mean that your tests are maybe not so good That's a hint to you that right now you should look at that Find out why it's missing that spot that it used to hit You've introduced something new In your code and it could be a bug Or it could be a Nice event you've introduced an invariant that actually causes some code to be dead And if it is dead now is it's good to time as any to remove it in my opinion and simplify whatever you've got So I think that this coverage stuff is nice And useful not just as a metric But as a sort of debugging aid Okay, so what what am I going to give you here? I'm extremely simple code examples so that They will be so dead simple that you will hardly notice them so that I can give you So I can actually convey what I'm trying to convey with them so that the code examples shouldn't distract you at all I'm going to talk a little bit about things that I learned And also I'm going to refer to other things because this whole property-based testing world is Really big and getting bigger and I can't really Cover I can only cover a smidge of what this is all about in a time allotted here I'm an extremely simple example This is something that's going to convert take a natural number which is actually An integer which is non-zero in python and given a base. It's going to convert it to a String of digits that represent that number in that base Okay And so here's the old way the so-called example-based testing to contrast it with the property-based testing And what I do is I think of good examples and predict What they should do and I write all that down and then I run My tests and hopefully they all succeed So what you see here is checking that it does the right thing for zero Which I've defined as gives you the empty string and other things explained here And there's the ellipses here Which means and I would do a whole lot more To cover as much as possible. Okay, so that's example-based testing Doctrineer I think that when that's what you can write. That's what you should write. I think that that's better Than nothing By a margin which is considerable So What is good about these tests? Well, they catch bugs and that's good and we want to do that and they catch Surprises when you do something new somewhere else and that breaks Something that your old code previously relied on that's good too And they are Partial specification of the code they test They give some idea Of what the person who wrote this or the tester thinks it should do But they're implicit because you have to look at these examples I just went back to the previous side You have to look at these examples and try to figure out what the person who wrote the code was thinking their code should do So they're implicit Here's the property-based testing that I'm advocating And the idea with the property-based testing is you say something you express something that should be true About your function and what I've done is I've actually stated What should be true about my function as a theorem? So I happen to have a convert an extra function called convert to NAT And all I'm saying here is that if I run convert from NAT on my natural number with a base And then I converted that to convert to NAT. I'll come back to the same value that should happen Okay, so I happen to have this extra convert to NAT Function and these are logical symbols that say for every value b that's greater than or equal to zero That's all the natural numbers. I'm implicitly assuming here that they're whole numbers And for all bases that are greater than or equal to two because one base one is uninteresting This will hold true So that's my statement about this function And now here what I've done is I've copied it into hypothesis I've used the hypothesis Functions and decorators and some so a fourth to get this And what you should see here Is that it really just is a copy. I took a mathematical expression And I changed it Into an expression written in the code. It's more verbose. It's less nicely typeset, but it's almost the same thing We can see that here V and value are the same V and base are the same And I'm using this hypothesis strategy That gets python integers And exerting that I'm only interested in ones that are greater than zero for values or greater than two for base And then this is just a hypothesis function Which asserts Exactly what I'm saying up here Okay, it's as straightforward as that Okay, and here's another example, which is simply a repetition of the same ideas. So Just shows that you can turn them out pretty regularly in this one. I'm just saying that The strings I generate Will not have leading zeros. That's how I've defined it I like it that way It's nice and consistent and so I Didn't write out a mathematical expression for that But this is how I say it in python Okay, so just to show you that there's variety even in this extremely simple example And it's in a file and so forth And then the question is what benefit do I get From having written this thing And the benefit is very nice. We assume that I'm running this code under um in some Python testing framework either py test or unit test. Those are the ones that hypothesis is compatible And so hypothesis will go ahead and it will test That specification it will test that that specification holds true over and over and over again So hypothesis will do the work of selecting arguments for the value in the base It will make up those arguments randomly More or less It will run that function that test function. I had It will check whether it failed or it succeeded And if it succeeded it will keep on going Coming up with more possible inputs and it'll keep on going Until that assertion fails in which case it'll stop and report that it's found a counter example Or Alternatively It will stop whenever and whenever whenever your configuration parameters that you've said say it should Okay, it'll keep trying Until it's told to stop or it finds a counter example And then obligingly if something failed It'll hang on to that information Okay So as I said, I'm not doctrinaire. I think example based testing when that's all you've got is a good thing But I think the property based testing Is really great So if you write hypothesis tests, you still only have a partial specification of your code You may not have been able to say every single thing That your code should do But you've converted it from an implicit To an explicit statement About what you believe your code should do So the reader of your test doesn't have to intuit from a bunch of examples What you're thinking it's right there in the code and that's good So also you get more real tests With less code these are not the same they in a sense. They are the same tests over and over again But because they vary the inputs They do something different So you write less code To get the same amount of tests which are arguably just as good And for me personally, that's a nice benefit because I I'm one of those early testers I'm testing as I'm writing And with example based tests Maybe you've generated quite a few and then you realize that you want to Change the arguments to your function that you had somewhat misdesigned it and Practically speaking it's annoying to change 50 example based tests once you've just discovered this thing but with With property based testing if you've just written that one specification, then that's one thing to change Tidy and quick Another thing you get from this is you employ code reuse in your choice of examples And that's really important because Generally if things are going well, you can use a hypothesis strategy And those hypothesis strategies have been written by people who actually know about the values That they're fine that they're those random values that they're selecting so they know Often more than you So for for my first example, I don't know why I was using this but I was using um Python's decimal class And so the first time I ever used hypothesis The strategy started producing random instances of decimal And it turned out that that exposed A naive idea in my head about what decimal was and actually how it worked. So the The strategy since it was written by a person who knew more about decimal than I did tried certain varieties of decimal number that I didn't even know existed and Broke things all over the place, which was good, which was what I wanted So that kind of code reuse is very nice. You hand off all the work to some expert in Decimal or floating point or whatever and you don't have to Do that, okay And then the other thing is this database of failures that hypothesis keeps around That's really nice too because Now it has a property that it wants to check But the next time you run after you fail it goes straight to the thing that failed last time And tries that first So you can actually be Less clever than you would want and perhaps want to be with um, then you might have to be with example-based tasks where you probably Do something to make sure that the example that was interesting somehow got to the top hypothesis does that for you. So that's great And in fact hypothesis does a huge amount of other stuff, which I'm Going to sort of mention at the end and let you decide if there's any particular thing You want me to talk about more Okay, so so I've been talking about how great property-based testing is and the advantages you get But it's not actually easy To use it.  You can always cheat with the example-based tests And people are tempted to do that. So you write an example-based test. You think oh, these are some interesting values So you stick them in then you stick some Random value in the expected result and then you run it and then you see what the real value is and you're like Oh, yeah, that's what my function does and you plug that in and you have a test and My point here is that property-based testing Kind of makes it impossible for you to cheat in that way and that's good Some things are just going to be harder to express using hypothesis Um, the whole idea is that we have these sort of universally quantified statements And then we can say a lot of stuff because python is very expressive about what should happen but some things are difficult Or sometimes, um, you can sometimes just use hypothesis in a silly way And I I think I've done that a couple times because I was already using hypothesis for something And it was cheaper to use all the infrastructure of hypothesis to get a string That didn't even have to be random Because randomness didn't affect the quality of my test, but somehow it ended up there Sometimes time is a problem and I've certainly encountered that with more numerical things where you want to test something But you can really see that hypothesis is cranking along and it's taking a long time Um, to actually search even a few examples because it's just computationally expensive Or you might you might run into a practical difficulty Where you want random values and hypothesis Doesn't have a way to generate those particular random values for you hypothesis contains many strategies for generating typical python values And also, um, it contains many higher order strategies Which allow you to build An assemble strategies between those higher order strategies and its internal strategies You can build More powerful strategies and that works great and I know that I checked this it took me Closer to two years than to one year of Did well no it took me 18 months is one and a half years It took me 18 months to get from the point where I was using hypothesis I started using hypothesis regularly to the point where I needed to actually write My own from scratch Strategy before that I could always just synthesize strategies using hypothesis higher order strategies So, um That's another I think vote or a demonstration of the power of hypothesis as it is I'm also going to mention that I had a prejudice About property based testing which I lost due to the offices kind offices of a person I never actually met So because I was first introduced to property based testing by Haskell and anybody who's written in Haskell knows that Haskell is the the pure functional language and Usually if you're writing in Haskell the demonstrations involve abstract syntax trees and fun things like that and it's all kind of It's got a nice functional feel and so I I associated property based testing with pure functional stuff But I was really wrong about that and I found that out when I took over this high udev Library, which is just a python wrapper for libudet at this point Many of you could be forgiven if you're not storage people for not knowing what libudet is And so I'm just going to mention briefly What it is so that you understand what I mean by not really a functional thing You all know about devices we have devices on our computers and they're like the mouse and the keyboard and the hard drive And there's this device abstraction Which unix introduced and which is very useful And besides all those familiar devices that I talked to you about just now that we all think of when we think of device There are lots of subterranean devices that support those visible devices that we care about And also if you're a storage person like me, they're also Super devices that are built from the devices that people normally think about so there's lots of devices cooperating And udev That dev part is about devices udev is just essentially A database not in the sense of a relational database, but just a table udev is a table you identify the device and then there are some properties of the device What does two mean? two minutes okay And then questions all right so I'm actually going to stop talking about pi udev and you can ask me about that later But the point was it's all about devices and The original developer had actually had invented his own system of property based testing And I went ahead and hypothesized it up for various reasons and because of the combination of hypothesis And also And also the fact that I was running on funky machines I found more bugs bugs in my udev bugs in lib udev And a bug in the actual udev database it couldn't agree with itself and that bug. I think is permanently stuck there But I found it so There you go. Okay, python to rust. Do I have one minute? Okay, thanks Yeah, so as I mentioned before I did quick check at first because That was really what was available. Prop test is the newer one And the difference between quick check and prop test which you can find out about because they had many cordial discussions about this Is it's about how counter examples are minimized and you'll notice that in this short time I didn't even talk about counter examples and minimization But what I have to say is that prop test just works much more nicely for me Prop test is good, but it's not as mature as hypothesis And what I did is I just reworked The first example that was in python Into prop test and you notice there's a macro here Instead of a generator because rust supports macros. There's this little test annotation and then I've specified my values a little bit differently But it's really the same thing just in a completely different language And done in a completely different way, but the same thing and that Should get me to the end And I'm saying now you can ask me questions and you should feel free to contact me about this topic Because I would like to help you if I can get if you're interested in getting moving In many ways about this. Okay, so Questions Prefab I can't answer that definitely but they talk about numpy a lot Um, so that's that's not really my area, which is why I can't but um, they may Um and another thing that I didn't mention is that um There's And the hypothesis site there are external hypothesis related activity um, and Well, I've actually contributed to that to writing my own strategy and if you were absolutely desperate And this didn't exist You could write your own and put it up there and it would be it would be featured And can you just repeat though? Oh the question was about numerically heavy algebraically heavily specific things And um does hypothesis have strategies for that kind of stuff and I couldn't answer that directly But I do see numpy floating by so there's hope, but I can't be specific Yes Okay, here's how to run it says if it's for a random space Say sorry say that a little bit more You're running it like in continuous integration. Yes Failure that maybe didn't rip one. Maybe you're running something subtly differently. Is there some sort of output that Get so that you can instead So doing the whole space you can start with the thing that causes the problem in the other environment Right, so that's a good question. I think in the old longer slides I explained that that that's a problem Because it is hard To figure out how to reproduce precisely how you got there hypothesis It's not so bad because hypothesis will actually Tell you the failing examples as its output And so that could get you a reasonable amount of way there So so in fact, I should be more positive hypothesis will say these are the arguments to the test function Which caused this to fail and that will all be visible in the ci. Yeah How did you deal with the service inherent complexity sometimes in writing specifications for things do you? Learned the situations where the tests themselves have bugs because of the nature of writing that and how do you deal with that? Yes, um, so the question is well, what if your tests have bugs? And that's a good question because your tests will have bugs on occasion Right. Yeah. Um, and that's that's an interesting rabbit hole and I I do not Say I have any advice. I once told somebody you have to write your test You have to test your test, you know, you have to have this infinite regression But obviously that's not practical advice. Yeah, so, um There are two things that can happen one is your tests Let bad stuff through And that's unpleasant that can definitely happen. The other one is that your tests fail because They catch good stuff essentially you wrote them in a way such that they fail because they're They're perhaps They're either you either wrote them And they're a bad match for what you meant to write or else You were wrong and you wrote them and those those are those are two different things But both of those are less troubling Than the other one when you just let the bugs through Yes all right So sure Should I just start yanking clothes out? Oh, yeah, you can. Okay. I don't want to cause any trouble Uh, excuse me guys, it's just a couple of announcements. So, uh, if you are not aware we are having DevCon party in the evening if you wish to attend you can pick up the tickets at the registration desk and Tomorrow morning for the DevCon keynote. We are having the red hat CDO and the founder of code code Newbie speaking that will be at 9 30 in the big room. So please do attend that things And I Know that one is you guys this one is mine. Thank you Thanks Oh Oh, it's over here, do you want to use this one? Do you want to use this one? Yeah, I mean, this goes out of range. So if you plan to walk or even stand here? Yeah, I mean, it's very good. Yeah, it's good. Do you want this one specifically or? Yeah, I mean, this one also works. No, no, this is working. This one. So whoever plans to stand here can use this. If you're planning to stand away, then you should use that. Okay, so not this one. All three are active. All three are active. So yeah, you can use that if you're more confident. That's the clicker. Yeah, if you want to use the clicker, you can just put this in. Check, check. Cool. Awesome. Check, check. Check, check. Yeah. Right. Yeah. Can you close that? I think that's the issue. That's why we can't see the slides. No. Oh, it's on the next person. It's just not. Yeah. Go back. How can you point here? I wonder if I can just click it physically. It's there a power button. How's it off? It's off. Oh yeah, that would make sense. Oh, there we go. You guys have a minute. Yeah, you have a minute. Just come to the side. Okay. Do you want me to introduce you guys or? I can't, sure. I mean, I can just give like a welcome to Tom. Sure. Hello, everyone. Thank you for coming to the second day of the ensuring software quality track at DevCon to us. The next talk is open shift on open stack by Eric and Emilia and I'll hand them over to them. Hey, guys, my name is Eric. And my name is Emilio. This summer we worked in Boston on the open shift on open stack team and this is our presentation on overcoming development challenges. So show of hands who here is familiar with open shift? Cool. All right. So for those of you who aren't open shift is an application platform that runs at Kubernetes containers. And it can be used dynamically scale applications as well as update your back and on the fly. So it's a really cool tool. And then another show of hands who's familiar with open stack? Cool. So open stack is cloud infrastructure management platform and it's used to manage and lease resources and hardware and stuff. All right. Cool. So for our work this summer, we worked on the open shift on open stack team and open shift can be run on open stack and that's something that we are trying to accomplish. But why you might be wondering do we want to do this? So basically when you run open shift on open stack, it makes it a lot easier to deploy, manage, and scale your open shift cluster. And you can do this by taking advantage of the compute and the networking and storage services in open stack. Now developing in an environment that's sitting between two really large technological projects can be pretty complicated. And this is something that we've been taking on for a little while. But there are working versions of this. For example, if any of you are familiar with the MOC, that is an open shift on open stack environment. But anyway, so our team works on code related to deploying open shift on open stack and we are trying to standardize and optimize that process. So throughout the course of this summer, we faced several challenges. The first one being adjusting to a new development workflow. That's something that most developers will have when starting a new project. More specific to an integration project such as open shift on open stack was making technology to meet in the middle. And so we're going to dive into each of these one at a time and use our summer projects as examples for each. So first off, we're going to talk about getting adjusted to a new development workflow. And we're going to talk about how the tooling and the technology that you get exposed to are going to come into play here. So first of all, the first thing that you have to do before you get started on any project is obviously communicate with your team. And in terms of the open shift and open stack team, that can be a little bit more challenging than others because our team is actually distributed across the world. And so in order to get started, you're going to have to use a tool called IRC. And how many of you are familiar with IRC? Wow, that's shocking. Well, good for you guys. And then so the second fun part about that is since everyone's all over the world, they're not on your nine to five schedule. And so everything that you want to do has to be built around that time zone.  And so as you move along, obviously you're going to have to eventually hit a point where you ask for help. And the first thing that you have to learn about using IRC is you can't be afraid of the chat. And I think that's something that everybody inherently has.  But eventually you reach a point where you realize, there's going to be someone who wants to help you. And if you post in all chat, you're more likely to get that help. So do that. Second of all, people might always be willing to help you with one problem. And that's fine. But don't just ask for help. Ask for resources. A lot of the times people will be able to link your documentation or things that you didn't know exist to help you solve your problems on your own. And that could just make the development process easier for everybody. Okay. So then there comes DevStack. And this is another part of developing for OpenShift on OpenStack. If you develop on OpenShift in general, or rather OpenStack, you're probably going to work with DevStack at some point in your life. And basically what DevStack does is it builds a little miniature version of OpenStack in your own local environment. And it's kind of neat because you can build from your own source code and deploy it however you like. But just as anything that's software related, there are obviously some complications that come along with that, which Eric's going to go over. So have any of you guys ever set out DevStack before? All right, for those of you who haven't, buckle in. First thing you're going to need to do to set up DevStack is stacking.  It takes about half an hour. Works fine. It's great. But if you want to include services such as Octavia, which is OpenStack's load balancer, you're going to have to set that up and you can fake file. So you're going to have to do that and then stack it again. That's going to take another half hour. So you add these services. You re-stack it. It should be fine. Now, something's not going to work. As me and Emilio experienced, it's like I don't want to exaggerate, but we spent six or seven years setting up DevStack. And so we spent a lot of time debugging. Then finally re-stacking again. And this process continued and continued and continued. But we got it. And DevStack is a great tool. All right. So we're going to do the sub-project within OpenShift and OpenStack that both of our projects were related to. And that is career. So first and foremost, the most important part of the slide is this guy. And that is the career platypus. He's unofficially named Carlton. Don't remember that. I didn't tell you that. So basically, what does career do? Career enables OpenShift to use the networking packages and services that are already in OpenStack in place of its own software defined network. And so, why do we want this? Basically, both OpenShift and OpenStack have their own software defined networking solutions. And the problem there is that both of them are going to inject their own checksums and their own headers into their networking packages. And that makes them very slow to decrypt. So by going around that, you can really speed up the networking for OpenShift on OpenStack. Now, career as a project can be configured to run either in VMs or in Kubernetes. And that allows it to be kind of isolated from your OpenStack environment. But it's not like isolated from the software itself, just physically isolated. And there's quite a bit of a tech stack that you're going to have to get used to to work with it. So my project in specific this summer had to do with scaling out the career controller. And so the career controller is a component of career that is responsible for listening for Kubernetes API events in OpenShift and then translating those into the appropriate OpenStack API calls in order to service different networking events that needed resources to be created, modified or destroyed. And so what that entails is the following. Basically what we have right now is a high availability mode and it's an active passive mode. So if anyone's familiar with what that means basically it means that one node is going to be active and then a series of nodes are going to be passive or idle and they're going to wait for that active node to fail and then one of them will take its place. And this is good. It's high availability, but the issue is that it doesn't scale out. And when you start having bigger data sets and bigger data centers it's going to start getting stressed under load and we don't want that. So we are working instead to develop an active active high availability solution which means that all of the components are able to handle jobs and pick up the slack of any servers that might have died. And overall the benefit here is that not only is it highly available but it also scales horizontally with it scales its performance when you scale horizontally which is, you know, obviously better. And in order to work with it, actually let me go back real quick. You see these guys are in little boxes that's because in order to run it in high availability mode you actually have to run it in Kubernetes and so you're going to have to run the career controller each in its own pod composed of two different containers and that's where you start running into fun stuff with Kubernetes. So first of all because both there are two containers running in your pod it makes debugging a little bit of an interesting situation and so first of all when you build a container I don't know if you guys have built containers before it's not the fastest process in the world so don't make the mistake I did at first you want to be pretty vigorous and test your code before you build because it will slow down your development cycle a ton. So first of all read your code and second of all take advantage of unit tests the third thing that I want to mention which is kind of maybe not for everybody because I know some people don't use Docker but a lot of people don't always realize that you can actually use the Docker tools as well as Kubernetes tools but if you're having a hard time with Kubernetes remember that you can use Docker's tools and they have a pretty rich toolset that can help you out a lot but anyway back to the unit test slide because that's pretty important if you work an open stack then you're going to have to work with docs and I'm sure at least everyone's going to test but just to go over them at a high level unit tests are really good for isolating different parts of your code and just making sure that they work the way that you expect them to and again what's nice about the unit tests is that you don't need to build your containers or your final product even to use them to test your code and so if you take advantage of them and you build unit tests to test your code along the way it's going to make your development process a lot faster there is however one part of talks that I think about unit tests in open stack that I think a lot of people won't be familiar with and that's Pep 8 so Pep 8 is this fun little test that tests your style basically tells you whether or not you suck at programming and News Flash the first time you use it you're going to suck at programming and I'm not talking about like this I'm talking like like that many errors so what is Pep 8 what does it do basically it enforces a really strict programming style and it's kind of used to just kind of keep everyone in check and make sure that all of your code within a project is uniform to some degree and it tries to make sure that it's all readable and at the end of the day it's going to make your code a lot easier to understand whether you like it or not and you know it's kind of a necessary evil now how many of you are familiar with github alright how many of you are familiar with Garrett alright right so uh same yeah so you're also going to have to learn Garrett if you want to work on open shift on open stack and the you know the fun part about this is Garrett you would think oh it's a git tool you know how different can it be actually it's like really different um so in Garrett first of all you don't follow the standard open source workflow like you do in github you don't fork it you don't you don't make your own branch you don't take out pull requests you actually just clone the main project and then you use something else called git review and then you push your code up to Garrett um and this could be a little tricky to learn for the first time uh and there's a lot of little things that go on underneath the hood that you don't really realize um once another thing you might not notice or you might not know at first is that um when you submit your code to check it into Garrett and you don't want a code review you actually have to go in and give yourself a bad code review and then basically give yourself a minus one to tell everybody don't review this I didn't know that I got code reviewed it didn't go well um but uh the upside of Garrett is actually it's kind of neat um to it's a really neat way to manage uh a lot of projects that are taking or a lot of people that are working on one project at a time and I think in comparison to git personally I like it better as a way to um see the various stages in your code and to see either the reviews that people left what changed and ultimately just to see what projects are going on in general and we'll just take a look here so this is what a standard um project uh commit would look like in Garrett and you can see here your history and these would be the changes that you've made along the way and each of those are expandable which is pretty nice um but when you look at the project view like this right you can see all of the projects that are being worked on under one large project and all of their progress which is I think a lot more organized than git either way though I can't speak for the open stack team I don't know why they chose to use Garrett over github but um whether you like it or not you're gonna have to use it so deal so anyway the grand takeaways here are um don't be afraid to ask for help um and if you do ask for help also remember ask for documentation every now and then um always try to take advantage of all the tools that you have at your disposal and you know ultimately you're gonna bump into some stuff you might not like and definitely won't be used to but there's probably a reason that it's there and it's usually there to help you so you know get used to it so now I want to talk to you guys about making technologies meet in the middle so being a member of the open stack team the obvious open shifts and open stack we had to make them meet in the middle but there's also services within them that we had to adjust to meet them so this summer my project was called watch endpoints as a service or the acronym WES for short um basically what it is is a service to speed up career by listening for networking events so I'm gonna talk about what that means in a sec so why do we want this tool so this is a diagram of the interaction between a bunch of services you only really need to look at that part right there between the career controller and the neutron which is open stack networking tool and basically you can see that the career controller creates a load balancer and then will continuously pull neutron using the open stack API to show the load balancer until it becomes active so the way I've been describing this through an analogy is imagine you're a chef in the kitchen and you want to take you're gonna have to preheat the oven the current system in place the chef is gonna be going back to that oven every 3 seconds until the oven has been preheated what my tool does is just have the oven alert the chef once ready so you can just see by that it's a lot less work and it's a lot more efficient so how do I do this basically there's this messaging queue between neutron and octavia which is the load balancer service and rabbitmq is getting a queue of all the messages as it is a messaging queue so when the career controller tries to create a load balancer eventually when it becomes active octavia will send a message to neutron saying that it hasn't become active my tool will sit and listen to the messaging queue and wait for that event once it sees that event happen it will tell the career controller so this makes too much sense this is the obvious solution so why isn't this currently in place basically the career guys ask the neutron guys to add this BAPI they said thou shalt not take events that came not from rabbitmq for that would be duplication and we're running shorter on maintenance so no so what's the end game if they already rejected this idea why am I doing this basically we're testing it with career communities and if it behaves as we expected to being faster and more efficient they're going to propose it to open stack as a career self project and if it goes well it will be integrated so this brings me to the challenge of making two technologies meet in the middle my listener would not hear any octavia load balancer events you would hear the four of events find from neutron but it wouldn't hear anything from octavia so naturally because it was working with with neutron and octavia I figured the problem must have been within my listener so my train of thoughts first of all I assume that neutron and octavia events behave the same way because they're both open stack services so why would they not secondly when I would use an open stack load balancer command it would generate events but they appeared to look like port events like the event type in the jsock return would be port dot something so at the time I assumed that these were octavia events because they were generated by a load balancer problem so based on these assumptions there was no way to distinguish these octavia events from the actual port events and so this was an issue because if this tool can't sort the events in that manner it kind of defeats the purpose of it because then you're going to be sending back hundreds upon hundreds of messages back to your career of control so the explanation of all this we figured out after talking to the load balancer guys that octavia doesn't actually emit events like neutron does so why is this and why do we assume this basically on the career side of things we just assumed that most open stack services emit events why wouldn't octavia on the octavia team they thought why make it emit events ironic which is open stack bare metal phishing tools didn't do it and it was never a requirement so it was never enabled which makes sense so both sides had reasonable expectations there was just this miscommunication so meeting in the middle how did we fix this issue temporarily over here we have career expectations capabilities in the middle there's load balancer as a service version 2 so if this satisfies, if this behaves how we want it to why is this the final solution why aren't we using this well for two reasons the current open stack deployment uses octavia and also lbos v2 is deprecated so right now we're going to talk to the octavia team to get them to reconfigure octavia to emit events but for now we're using this for test purposes and this isn't the first time ship stack teams have to meet in the middle in the past openshift requires that nodes, a node being an open stack server can talk to each other through their host names as opposed to VIP addresses whereas open stack doesn't need at least support this so the solution they found is called neutron DNS and it's actually a tool that they've had built into open stack for a while now nobody really knew about it though so just digging deeper help them find a solution to this so less of a gain sometimes you have to dig deep to find a common ground so throughout this summer we dealt with a lot of challenges but we really learned the teamwork and collaboration between teams is crucial there's so many different teams so many different resources out there especially via IRC that it really gives you to make use of them to take advantage of your resources and renad obviously is all about open source so open source means being part of the community so you really want to take advantage of the whole community because you have that support thank you you can keep the questions oh okay oh here do you want to take pass this around um what does he does he's asking about Garrett wait what did you ask about Garrett hang on microphone you can use this one one okay when you say Garrett it sounds like it's like a replacement of github but you I think when I use gyrus you need to clone the repo and then I can gyrus is like an extra tool for github that you can use on your computer so I'm just confused we'd be able to just talk more because I was confused about gyrus as well I think Garrett actually works with github you can host your code up on github and I think Garrett is really more for managing different commits and code review so instead of using like the traditional workflow like I think for most open source projects in general what people normally do is you'll fork github repo and then you'll work within your own fork you'll make a branch and then you'll take a pull request out against the upstream but when you use Garrett you don't do that basically what happens is you just clone down the I feel like I'm too close to this basically what happens is you just clone the upstream and you can make whatever changes you want and when you submit it you use a tool called git review and git review is going to I don't know all the details but essentially it tags your work with something called a change ID and then all of the changes that you make under that change ID get submitted into one general repository and then eventually if it's reviewed and accepted then those changes can be pushed back up to the master to the upstream github yeah definitely we use Garrett a lot in Red Hat to explain it to you in the simplest of terms when you use github and say you submit so you commit something and there are changes requested you send a v2 with a new commit ID so in your history you have a commit ID then a v2 commit ID or v3 commit ID when you use Garrett with git it allows you to edit a single commit so you can do because it uses something called change ID so say you commit something get it tags it with a change ID say your team request changes made a mistake you don't change the commit ID you keep the same commit ID you keep the same change ID and you can make changes to the same commit so when you look back you have a cleaner history and in one commit you can see all the comments v1, v2, v3 just for the same commit so get it allowed you to do that in github it's like just more of a keep sending commit sort of a thing it's granular but it's not the same it gives you more granular control yeah it's very true although it's it's fair to say that github is starting to develop more Garrett like workflow tools because they're recognizing that the workflow component of it is actually really important I had a question I apologize I missed the first few minutes did you all did you talk about what is the well two things the rationale for the work in the first place like what problem are you trying to solve if you talked about that then ignore me because I know I just want to make sure that everybody else heard it the other thing I'm curious about is what you see coming in the next few months for the shift on stack work in the next you know next version or whatever that is so man do you all feel about for the future let's enter the oh I don't want you to do it oh okay well in terms of the first part of the question we did go over a little bit just the concept of what the shift on stack team is trying to accomplish and how career is part of that goal that being said if you want to add to that feel free to if anyone has any questions about it please ask in terms of future though for my project at least we're going to containerize it for triple open stack on open stack and I'm not sure what the next steps after that will be yeah I can I'm pretty sure my project is not ready to get pushed on it was a lot more research focused that being said I think some of the changes that were done are going to be kept and I think some of the changes serve as an example of what we should not be doing in the future which is fine that's part of the process and yeah so I think yeah also I do know for a fact that in terms of just the active active I know that the open shift on open stack team has decided to kind of put that aside by now for right now it's going to be coming I don't know if it's going to be in the next release but it's on the menu just it's not a high priority since the money was worth saying that the open stack project there's a lot of directions that that the team is going in on one side is just supporting new features like making sure that we support things like staff and stuff like that another part of that is like standardizing the process so that other people can just play open shift on the stack easily and without having to do a lot of customizations the third part which I think is what you guys kind of concentrate on is optimization which is making sure that bits of the code are optimized in this case networking so that's what you guys were concentrating on but we have we're going in multiple directions with open shift on the stack if anyone is curious talk to me outdoors and also just to give you guys an idea of like what open shift on open stack like why we are interested in developing this we think of it as it's kind of our hybrid cloud offering because right now open shift can run on almost any cloud provider including open stack and we want customers who are thinking of building clouds on prem or in the cloud to be thinking about using open stack as their under cloud solution so yeah another question I know that this container module for open stack I forgot the name is it something where it's like it's like a container module for open stack but I just can't remember the name but what's the difference if you know what I'm talking about between using open shift and the one check that one so this is okay this is very ancient history there used to be a project called um project solemn S-O-L-U-M which was about essentially making using open stack as a container scheduling system that's long since been obsolete by Kubernetes um there is another sort of interesting effort in open stack land around called Cata containers Cata containers are actually a fully free and open source implementation of Intel's clear containers which are really VMs that you can spin up very, very quickly. They're super minified VMs that boot almost as fast as a container and there is some movement within open stack to make Cata containers sort of a first class citizen but you just at this point there's no competing with Kubernetes momentum in the container space. We don't think anyway that answers your question Any other questions? Alright, well thanks guys um so ladies and gentlemen next up we have Xiaofeng Wang senior QE engineer at Red Hat and he's going to talk about performance test methodology so welcoming him on stage Good afternoon everyone I'm very excited to be here at DevCon this year. My name is Xiaofeng Wang. I work in the virtualization QE team of Red Hat Today for my presentation I'm going to introduce you a new network performance test methodology I was running performance test earlier this year with widely used performance test methodology My job was to do some network performance comparison among three different hypervisors like KVM ESX from WMWARE and HyperRE from Microsoft so during my testing some issues blocked my test so I had to fix them to finish my performance test when all the issues were resolved I felt that the solution to the to the problem can be summarized into a new performance test methodology I will give you a detailed introduction on these issues and the corresponding solutions but first sorry that's not work I will show you how I run performance test before that means with old performance test methodology the first step is always run the test tools in my case I choose the hyper so you can use the hyper for some package generators and mouse so I run hyper dash s on the server side and hyper dash c on the client side with some hyper options when the test is finished the hyper will show you your performance result like this one so the final performance result will be in the last two lines so in this case the performance will be 21 gigabit the third step is run more tests and get an average value as final result so I guess we always do that the last step generate the performance test report based on the average result so this is the how I run the performance test before with the old performance methodology what's the problem here maybe you cannot find any problems I will show you the problem I just found the first one there's no step by step traffic just send as much traffic as possible so what's the meaning what's the step traffic like this chart it's a step step step step for example if I want to test 10 gigabit NSE network adapters I will use the traffic should be start from 1 gigabit to the 10 gigabit with 10 steps one step will increase 1 gigabit so it's called step by step traffic so what's the problem here without the step by step traffic because the IProof or netproof just send as much as much traffic as possible to against your DOT means the device and the test in my case it's a network adapters so sometimes the system will be broken instantly by exiting the max traffic at the beginning of the test so at this time I cannot get the performance result of the system so I have to do some setup or some things to make the test running sometimes it's not running because the system broken this is the first problem the second problem is about max value I find nevertheless the IProof netproof or something else they just give you the max value that means the max performance in my case the network adapter can support but it is just a max value so the network adapter in my case cannot work well at the max performance for a long time just the max not the working value have you heard about the Intel has a technology called Turbo Boost have you heard that it's a little bit like this concept like I just said the Turbo Boost you can use the Turbo Boost not for a long time just at some very special scenarios for example one minute or just a short time that's the best max performance of that CPU but at the normal scenarios you just use a lower frequency of the CPU the concept is the same as I just said this so all the performance test tools just provide you a max value so the max value is able to reach the performance but the stability cannot get granted with the max value that's the second issue the third issue the test tools like Apple 3 use mean mean to calculate the final result why not medium there might be a result different between the different algorithm like mean or medium so what's the best one I will give you an example and more explanation on this on the solution section so the mean and the medium is different algorithm to get the average value so you choose different algorithm you can get a different result this is the third one the fourth one is there's no way to find a spike from the current test result log have you heard the spike in the performance that means you can find in some scenarios performance drop to a deep or increase very high so if you just use the iproof you cannot find this you just got a final result the last issue the report is sorry the report is not user friendly as I expected just to give you a number no chart, no tables just a number so that's all of the problems so what's my solution for the first issues we need a step load that means a step by step load to avoid the max or let the iproof or netproof send traffic as much as possible we give it a limitation tell the iproof or netproof how many traffic you should send this time how many traffic you should send next time so in my solution in this case I will run the iproof multiple times each time I will give it a limitation so the iproof will send that traffic as I expected that's the step by step load so I use the step by step load to increase the load until the network or network adapter reaches a point where the performance is significantly significantly as the load increase the network will be able to keep up until it runs out of the resource that means when if your network adapters cannot support that traffic the chart will be drop or keep on that on a line not increase that make sense yeah how to do that is very simple because iproof has a dash v options which is able to set the bandwidth it just needs you run the iproof multi times and run it will be very easy because you can write some write a loop to set up a read about different bandwidth and run the iproof multi times so this is the solution of my first problem make sense? yeah for the second issues I use some mathematics mathematics to solve that issues I use correlation and r squared that's true mathematics algorithm so I will explain this chart later I will give you a brief introduction about what is correlation and r squared for the correlation correlation is a statistical management relationship between two variables possible correlations range from one to minus one and a zero correlation indicates that there's no relationship between variables and if the correlation of minus one negative one indicates a perfect negative correlation that means this one for what indicates there's a perfect positive correlation meaning both variables move in the same direction I have an example to explain this I have these tables about two variables temperature and ice cream shelves you can find when the temperature goes up the ice cream cells goes up as well so these two variables have relationship they have relationship but what's relationship you have to do more things I will do that later we can easily see the warmer weather and higher cells go together the correlation in this case is about 95% therefore the relationship is good but not perfect high positive correlation yes because I did some modification on it so what's the R squared the R squared is a statistical measure of how closely a train line matches the date so if in this example I got the point in the chart I can get a train line across the date so the R squared will tell you how close the date with the train line with the R squared I can find how the train line can represent this date train so if it's very closely the R squared will be 100% that means the higher R squared the better mode fits your date I will go back to this chart I will explain this this chart for for the X axis represent the load like I just this this is the load for the Y axis is the PPI means the packet that means how many packets the network adapter can process, can handle in one second so when the load increases the PPI should be increased as well right but when the network adapter cannot handle the light amount of traffic they will go to the Y line so the load and the PPI they will be two variables they have relationships so I can calculate the correlation value between these two variables so the blue dot line indicates the correlations so you can find the correlation is always 100% but at this at this time they will drop because they cannot increase the PPI cannot increase so I believe the correlation a number 90% is my accepted number so in this case I can find here there is a cross two lines cross here so I can find the value in the X for the load right and based on this the blue line I have got a liner the liner will be Y equals AX cross B so if I find an X value I can get the Y value right in my case the max value should be which one maybe this one the max value but in my case I don't want that value I just want a value that can work for a long time so the value will be here or there should be this one idea it's a little bit difficult for me to explain it very clearly but let me explain it again we have two variables load and PPS when the load increase when the load increase the PPS should be increased as well so this line the blue line indicate that these two variables relationship they always increase but at this time they always increase so the correlation value will drop here but I set an acceptable value for the correlations I set it to 90% so there will be a cross point two lines cross point B here this is the correlation the dog line indicate the correlation so I can get the cross line with 90% here so I can get the X one and I can get a liner based on the blue line Y equals AX plus B like this one you can get it by some in my case I use the google shape so you can get it it will be generated automatically so now you have the X value so you can get the Y value that means you get what you want it is not the max value that means the maybe lower than the max value but it can work for a long time just in my case further the third issue is about the mean and the median the mean is the average you have used where you add up all the numbers and then divided by the number of numbers in this example the mean will be 9.73 it is called average in the google shape google shape has a function called average to calculate the mean value and the google shape has another function called median it will calculate the median value median means the middle value of the list of numbers is 9 the difference between the mean and the median is the mean value is calculated this value you cannot find the value from the list that means in the performance result the mean value will not in your performance performance number generated by the iprof it is just calculated but the median is this value is in the array of your performance numbers so that's the difference which one is the best it totally depends on your requirement so your test result and your requirement I prefer the median in my case because median result always exists but mean result does not in this case maybe the difference is not too much but in some case it will be very large the fourth one is about how I can find the spike in my performance test I use another mathematics it's called sand deviations that means the sand deviation is a measure of the distribution of a set of data from it mean perfect use the tool to find the spike you know the deviation is how you can find a mean value the actual value how far the real value from the mean value that means if there is a spike if there is a spike the sand deviation result will be higher or very high or very low so you can find it because you can from your result array you can get the max sand deviation value and the median deviation value if the three value is totally different that means there must be a spike in your test it makes sense for the last one it's about the result demonstration for my perspective I like a test report with with a table it's a table report with some chart the table report should be clean and simple and easy to understand and the table report should include most important performance indicator it has to be some chart intended to supplement the table report I will give you examples this is an example I run the performance test on the ESX this is the report table I have the RXPPS the TXPPS throughput average latency max latency this is not the final one I will add more into this we can find the standard deviation with this value if there is a very high or very low value that means there must be a spike in my result it should include some CPU memory usage in this report so this is a table report for the chart you can find there is some chart for you to reference for example this is the chart do you familiar with this step by step yes this is the result from iprv I press the iprv result and put it into the Google Sheet will give you a line to indicate your performance so you can find give you overview of your performance result a lot of spikes yes a lot of spikes so yes could you please repeat the statement you made just a minute ago about the standard derivation if the number was too low or too high you didn't have the table chart oh yes you were mentioning you didn't use it or you threw it out oh the standard deviation this standard deviation indicated the slide right but you can now find a very low or very high value here or did you not include them here yes but this but this table is not the date behind this table is not the same as the chart so they are different I'm sorry because so if I draw the picture of the date behind this table it's very small no slide oh go back so this is the last problem and my my resolution so that's all about my presentations if you are very interested in my topic you can find it from the conference page but this page is just for the Red Hat internal so maybe you cannot get it but if you are interested you can find me and give you your email address I will make them to the pdf and send you right that's all ok questions is the expert tool is that available so is the expert tool is that available to people outside of Red Hat or is it only Red Hat employees available at this time or expert or is that sorry the expert is the name of my project I just call this message but just give it a name called expert it's just a name got you no other meanings sorry for confusing ok questions when you're doing your testing have you looked at percentiles instead of doing what you know average and mean median to be honest you might have I didn't find too much difference between the mean and the median but in some my colleague gave me their test result I found that but I really don't know what the scenarios what the scenarios they use so but I think the the median one is is more reasonable from my perspective but it's not the it's not a regulation here it just depends on your requirement or your test purpose but in terms of knowing what the throughput of what you're doing what you want to strive for is say 80% throughput or 80% of the time you're going to get through or 90% of the time this is where percentiles will come in handy versus just doing an average or a median hmm oh so from from all of the test test value the interval value is from the median and all of the numbers comes from the median I use median all through my test not part of mean and part of median median thank you just an announcement we'll be having a party in the evening on the lounge if you haven't collected your tickets you can collect it at the registration desk and we'll also have a keynote speech tomorrow so if you're if you check the schedule you should be there at Metcalf Large and thank you for being here I I I I I'm I'm I I I and to and testing framework for react so we simply use the Java with Selenium Web Driver to test react but if you're not thinking about end then you left right left right let's see if this actually works hey it does awesome all right cool wow more people showed up than I expected I wasn't sure anyone would want to talk and come to a talk that just focuses on what we do with the community so thank you for being here I appreciate it we do a lot and we're terrible about telling people about it so I'm gonna get up here and tell people about it all right I am showing that it is it is 430 so we'll get started hello everybody my name is Thomas Cameron I'm a senior principal cloud engineer in the engineering business unit at Red Hat my contact information is up here I am Thomas at Red Hat.com I've been here for long enough that yes I have the Thomas at Red Hat.com email address and you can follow me on Twitter at Thomas D Cameron what I want to talk about today like I said is we we don't Red Hat does not like to you know thump our chest and talk about how awesome we are and what we do and and so I think we don't talk enough about and explain enough about what we do with the various open source communities that we work with so I want to spend a little bit of time today doing just that I'll talk a little bit about where we came from where Red Hat came from where we are today I want to actually also interject something about what we do from a business perspective because a lot of people think of us as just another software company and I want to explain a little bit about why that's not the case seg into what our mission is because our mission actually calls out community our mission statement actually calls out community it is ingrained in our DNA I'll talk a little bit about some of the projects we work with I'll talk about some of the acquisitions and how that has affected communities and then I'll talk about some legal issues that Red Hat has been involved in on behalf of communities some of the community programs that we help sponsor the developer programs and why we do what we do so first off where we came from a lot of folks it's really funny I still read articles that talk about Red Hat as you know the scrappy newcomer to the IT industry and I'm like I I'm not kidding we got a new what was it we got a best you know young company or something like that like just a few years ago and I was like we started in 93 like we're old but yeah we were started in 93 by a hacker by the name of Mark Ewing and a businessman by the name of Bob young Bob sold he had a catalog business he sold computers and media with free software on it out of his bookstore and Mark built a Linux distribution called Red Hat Linux Bob sold so many copies of it that he approached Mark and said I want to buy your organization I want to buy your company and they adopted the name Red Hat software and went forward as a going concern and in 99 we did well enough that we were one of the most successful I think at that point were the seventh largest IPO to date so went from nothing to huge IPO in just a few years and 99 we really changed the way that we did business because previous to 99 we were just a distribution we we packaged up software from you know various projects upstream created the distribution sold it and we're successful at it but in 99 that really changed when we acquired Cygnus software Cygnus software it was a hardcore engineering company that did a lot of kernel stuff and did a lot of toolchain stuff that allowed open-source software to be ported to different hardware architectures that changed the way that Red Hat worked at that point we went from just being a distribution to actually doing a lot of hardcore engineering and contributing a lot of code upstream we had contributed before but with the Cygnus acquisition that really led to us being a powerhouse in the open-source community because we were able to actually do a lot of really good engineering work and contribute code upstream so from very very early in our days upstream was really important to us and I'll explain why in a little while in 2002 we changed from a consumer based model where we were selling box sets at you know Best Buy and CompUSA and praying that people would buy our software and augmenting our business or augmenting our revenues with like t-shirt and ball cap sales to an enterprise model with Red Hat Enterprise Linux and that actually was a pretty controversial move there were a lot of folks in the community who had gotten free distributions from us Red Hat Linux and when we turned around and said hey we're not gonna do free distributions anymore we're gonna sell subscriptions to software there was a lot of bad a lot of animus from some some parts of the community and so from 2002 through today there are still some people who are you know a little chapped at Red Hack is because we don't give our flagship distribution away for free we we sell subscriptions to it this was a massive change this was a huge change and companies like Dell and HP and IBM and even Oracle threw in their support behind us we were arguably the first to market an enterprise open source and I think it's pretty fair to say that we have been the most successful open source company to date today we're approaching three billion dollars in revenues we've got over 80 offices in something like 40 countries around the globe and we have about 12,000 people that would work at Red Hat so there are a ton of folks that are doing work around the software and the technology that we sell subscriptions to and also committing code upstream now as I said earlier a lot of people think oh Red Hat is just another software company but let's be real clear we don't sell software licenses the license under which you use the software that we that we support and sell subscriptions to is an open source license it's the GPL one of the versions of GPL or LGPL or patchy software license or MIT license or whatever there are a ton of licenses that are included in our technologies but we don't sell software licenses when you get a subscription to one of the Red Hat technologies that we sell subscriptions to it's not just about the bits the bits are certainly part of it because we deliver the bits through our content delivery network but it's about our technical leadership it's about accessing the bits from the from our network it's about knowledge base access and all the solutions we have there a big one is it's about a 10-year life cycle for open source technologies now if you work with open source technologies you know that in a lot of cases you know a six month old package is completely out of date but in an enterprise environment you don't want to be upgrading your infrastructure every six months so you know having that long life cycle is important product documentation how to tips and tricks things like that a big one in enterprise space is security we have a huge security team that's global that reacts to post-release security incidents and then also hardens our software to make sure at least to try to make sure that we don't have any security issues when it goes out of the gate it allows access the subscription allows access to the labs on our portal so you can do things like pre configure various services and pull down config files and it's access to the solution engine support solution engine is when you type in a problem it'll actually use AI to go and search through our knowledge base to see if there's something that matches it support up to an including 24-hour follow the Sun a big one software compatibility you know we test our software with our partners like you know SAP and Oracle and IBM and things like that so when you install that software it's a certified solution it's access to education and certification you know you've heard of folks who are Red Hat certified engineers for instance legal protection which I'll talk more about in a little while this is important to a lot of our enterprise customers and that partner ecosystem that partner partner ecosystem of ISVs and IH sorry independent software vendors and independent hardware vendors and vars and consulting groups and things like that and then certification on common and popular hardware platforms from you know Dell and HP and Lenovo and so on so it's not that we're selling bits the bits are only a tiny part of what the subscription brings so when you hear people talking about oh software you know Red Hat's just another software company company it's really not just it's not that if you look at Red Hat's mission statement online this is our published mission statement it says our mission is to be the catalyst in communities of customers contributors and partners creating better technology the open source way this is in our DNA you will hear me say that over and over and over again we are totally dedicated to working in communities whether it's communities of customers or upstream communities and I'll give some examples of that so what projects does Red Hat work with I think before I go any further what is the first thing you think of when you think of Red Hat Red Hat Linux right I mean for the most part people think of Red Hat as a Linux company which is fantastic because that's certainly where we came from that was where we started was in the Linux space but there's a whole lot of other stuff that goes on at Red Hat Red Hat Enterprise Linux is one product out of all of these technologies that Red Hat sells subscriptions to so there's Red Hat virtualization there's Red Hat open stack Red Hat storage which is Gluster and stuff Red Hat satellite for systems management Red Hat J Boss middleware which is like I'm going to show you some of the things that we do in middleware and it's like multiple pages of technologies Red Hat mobile application development management a lot of people don't think of that with Red Hat Red Hat open shift for container management Red Hat core OS core OS is now a Red Hat property manage IQ for cloud management Ansible is now a Red Hat property we acquired them not too long ago and for every one of these products that we sell subscriptions to there's an upstream project that we sponsor and maintain and contribute to so you know RDO and overt.org in the Fedora project Centos project Centos is also Red Hat technology now you know the upstream core OS open shift origin etc etc etc now that's a whole lot of communities right there but the argument can be made well you know that's just upstream for your products right you know that's not real community I disagree because we do everything in those upstream communities nothing makes it into Red Hat Enterprise Linux or almost nothing makes it into Red Hat Enterprise Linux I'll say nothing because there's always going to be some weird corner case but but generally nothing makes it into upstream into Red Hat Enterprise Linux that didn't come through the Fedora project we put everything out in the community freely distributable downloadable open source technologies the same is true for over.org the same is true for open shift origin the same is true for manage IQ Gluster.org stuff etc etc etc those are all communities but then there are I don't know about a million other projects out there that we are aware of and try to keep an eye on that we work with like at the Apache and OpenStack and things like that so there's a ton of work that we do around communities upstream but it goes a lot deeper than just what you see on the screen these are projects that we are involved in that are not productized they may be part of a product but these are these are things that we are involved with from a foundation perspective we're with the cloud native we're part of the cloud native computing foundation the Libre office foundation Linux foundation open NFV Python blah blah blah I mean you can see that these are all foundations free software foundations that we are members of to some level or another so we do a ton of work around standards that have that are not you know necessarily product related and let's be real clear the truth of the matter is by working with these upstream projects and making sure that there are standards which are repeatable across different communities and different technologies the truth of the matter is this makes it easier not harder to migrate off of our commercial offerings we make sure that things are standardized across communities so that they work no matter what now we don't want you to obviously but but the reality is by being members of these foundations and standardizing things we may get a lot easier for you to do other stuff from an operating system perspective I know this is kind of an I chart for those of you in the back I apologize but you know the Centos project Fedora Cygnus and upstream GNU and Linux projects and all of these utilities all of these technologies which are yes certainly part of red had enterprise Linux in a lot of cases but also part of Fedora and a lot of these technologies have been adopted by you know canonical and by Suza and by Slackware and and so on and that's part of the deal that's part of the deal we know that a strong vibrant upstream community you know rising tide raises all ships and so it's a good thing that the stuff that we're involved with like upstream Linux and SC Linux and system D and so on it's a good thing that there is a large and vibrant set of communities around these technologies if you look at containerization we're part of you know cockpit and cryo and Docker and flannel and Kubernetes and atomic and rocket and so on because again having a strong upstream community makes all technologies better including ours certainly there is a there is a component of we want to have the best technology in the market but we participate in those upstream communities by contributing code by contributing governance by contributing bodies and so on so and then from a desktop perspective again there's a ton of work that we do around GNOME there's a ton of work we do around Libre office and a whole bunch of other upstream technologies for the desktop like xorg and things like that so a lot of work that we do in those upstream communities to make the desktop better and everybody wins all the distributions use those technologies from a middleware and identity management standpoint got an interesting story about our identity management platform that I'll tell you in just a few minutes but if you look at what we do upstream with the Apache web server for instance what we do with the various jboss.org projects around application services and data virtualization and memory in memory caching of big applications and things like that there are a ton of projects and communities in upstream that we contribute to from an operations perspective you know again upstream over to an open shift and open stack and and things like this for cloud technologies all of these are open source projects that we work in the upstream with and anybody can have access to and from a storage perspective we acquired Gluster for NAS style storage software to find storage using the Linux operating system and then the storage layer on top of that and then we also acquired ink tank so Ceph is part of the Red Hat family as well for block storage so a ton of upstream work being done there and then some really cool stuff that we're doing like with the condor project for things like high performance computing or high throughput computing for computational clusters and things like that we contribute a bunch to upstream technologies like NFS and you know GFS or should say GFS too since that's the current version but there are a ton of projects that we work on there that again get adopted by a bunch of different distributions and then from a developer tool standpoint real quick how many folks are how many folks think of themselves as more developers than operations folks okay how many folks are more operations than developers okay good we got a really good mix okay so how many folks have used Eclipse yeah okay we do a ton of upstream development in the Eclipse with the Eclipse Foundation and and a bunch of other technologies on here there are technologies there are developer tools on here for you know sort of typical middleware or application development all the way down to kernel level and C programming so a ton of stuff there as well so did y'all know that Red Hat was contributor would contributed to all of those upstream projects couple of you did well you work here I don't you better know so Red Hat has acquired a bunch of companies just I've been I've been at Red Hat coming up on 13 years and just in my tenure here we've acquired a bunch of companies and it's been really fascinating to see what happens when we do acquire a company what we do around making that technology open source and building communities around it since we were founded we've acquired over 30 companies some of them have been very successful and frankly some of them have not we made acquisitions that turned out didn't work out real well and you know that that's just kind of the nature of doing business but in every case where we have acquired a company or a technology if it's been successful we have either created or at least maintained and invested in existing communities around those technologies and interestingly a lot of the acquisitions that we made were actually closed source so we had to acquire something that was closed source in a lot of cases we had to go through a bunch of legal you know rigmarole because of license technology a lot of technical rigmarole where we had to replace that license technology with open source technology and we have spent a ton of time effort energy and money in making sure that those technologies are accessible by upstream communities and then building communities around those technologies so you know if you look back at some of our early acquisitions Sustina was a closed source company that was doing HA clustering and a shared clustered file system for Linux we bought them we turned around we made it open source and and it's available out there on the internet for anybody to download we acquired a lot of the Netscape assets from AOL when you all remember when AOL bought Netscape our identity management platform is actually based on the old Netscape identity manager code now it's been so long that there's no code left anymore but that that was absolutely closed source we bought that from from AOL and had to jump through a lot of hoops to make that open source uh meta matrix for data virtualization kumranet okay no red headers are allowed to answer this does anyone know what kumranet is responsible for yes kvm kvm spice and spice that's right so kumranet was a little company that we acquired out of israel that uh that was the the the founders of and the inventors of kvm which is now used in open stack and you know red hat virtualization it's a it's a super popular hypervisor technology on the Linux kernel um that was closed source when we bought it the the kernel module the kvm kernel module is open source but like the management interfaces and uh red hat uh what they called at the time uh what we called at the time red hat enterprise virtualization hell when we first bought it it was based on windows that went over like a ton of ricks um but uh but yeah so we bought that turned around rewrote it made it so that it could be deployed in uh with open source technologies and now red hat virtualization for instance is awesome technology uh managed iq was uh cloud forms our our multi cloud management platform ansible when we acquired ansible tower was closed source it's not anymore so we've done a lot to make sure that uh that that we have made those technologies accessible by anybody who wants so just just out of curiosity how much do you guys think red hat has spent in acquiring technologies and then turning around and either maintaining them as open source or making them open source who said that you're close now it's it's we spent have you tallied all the two point the ones that we have records on oh okay but a lot of them were private transactions because they weren't publicly held companies uh and so there are some of them i mean there are a lot of them that we actually don't have dollar values on or that at least they're not publicly released so on the low end about two point four billion dollars on the low end for technologies that we bought either made open source or maintained as open source and made sure that the communities had access to them red hat has also been involved in a whole bunch of legal fights on behalf of the open source communities that we participate with um we've had red hat executives and evangelists who have testified uh at the US congress they've testified in front of the European Union Parliament uh they we have written amicus briefs for the United States Supreme Court uh and various lower courts around the world uh to fight things like patents uh software patents to fight things like patent trolls uh to fight things like anti competitive practices in the industry you know we spent a lot of again time effort energy and money making sure that we were in the front of those fights to protect the smaller communities that didn't have representation we've kicked a snot out of patent trolls and that's been awesome um there's been a ton of articles there have been a ton of things that we have done where we have either beaten them in court or we've even purchased patents when it was at at risk we was just like you know what we'll buy it we'll you win this round and then we turn around and make it available for free to the open source communities so we uh we have done a ton of work around protecting intellectual property and trying to fight patent software patents um we also offer a patent promise that says that if you're using our technology and that technology is found to infringe on a patent we will indemnify you and we will also write code to replace whatever patent and covered software there is so we're very very anti software patents uh we found it we were one of the founding members of the open innovation network which is a collection of companies like it's us uh Google IBM NEC Phillips Sony SUSE and Toyota of all things Toyota um and anybody who's working on open source uh any Linux related software can join Open Innovation Network uh and we have a shared defensive patent pool so if uh if somebody tries to mess with us it's almost guaranteed that we will have a patent to fight them back with yes what is Microsoft going to join that's a damn good question I don't I can't presume to speak for them yes sir very small question open invention network no open innovation did I fat finger it you did oh you didn't see that look away right after the statement dang it we haven't you haven't both ways yeah it's it's innovation right next to each other really yes wow okay I'm a moron it's been a long week this whatever today all right so we're also involved in a lot of community programs from a supporting perspective um we have a group inside of Red Hat called open source and standards or OSAS um I don't even know how many people are on OSAS now it's going to be a very very large team uh they're technical folks they're administrative and organizational folks and community management folks even like marketing and and artistic folks in OSAS who help communities organize so if you're a member of a community and you're struggling with community management or if you're struggling with like how to form the community and how to get it vibrant and active we can help go to community.redhat.com and we will absolutely provide assistance uh we have done things you know everything from helping with governance issues and organizational issues and and community management all the way up to the simple stuff like yeah we'll host your website or we'll we'll provide email services for your community so we're very invested in making that happen if you go to community.redhat.com you can see there's a ton of information there and and I just did this screenshot today you know dev.com for us is on there we talk about activities in the community we've got an events calendar uh we've got a knowledge base there's a ton of stuff there that are resources for community managers uh and and if you need additional help you can always email us and we'll help you there so redhat also has some developer programs that are that are absolutely focused on uh thank you on uh community members so redhat software is available at developers.redhat.com and this was actually something that I and a bunch of other people but but I was probably one of the most obnoxious about it um and I don't mind being obnoxious internally it happens all the time um but we fought really hard to have a developer program where folks could download redhat software and not do the 30-day eval or not do the $99 developer subscription that we had for years and years and years there were a bunch of us internally who said we cannot do this if we want developers to be able to participate in the redhat community not just upstream we've got to make it available for free and so they did so the business units made it available for free if you go to developers.redhat.com and click on linux um got a cool video about linux and scroll down a little bit you can get a subscription to redhat enterprise linux the developer version which is the exact same version but we also enable all the add-ons for like ha clustering and uh clustered file system and uh gfs I mean a bunch of stuff we add on to it and it's zero dollars so if we get the box too no unfortunately we don't ship physical media anymore 1999 called they want their technology back I know I know right I would love to have that I saw that I was like oh no okay dang it but uh but and what's funny is every once in a while someone will email and say hey I've got this whole box set of redhat enterprise linux 4 and I'm like I'll take that just ship it up my wife's like please don't so but anyway so and over and above that if you go to the middleware side you can do the same thing with the jboss stack too so all of our all of our je middleware stuff um and and the entire developer suite you can get started with the middleware developers kit as well zero dollars hello so we're knocking on the door don't tell him we're in here okay thank you so why why does red hat do this we're a commercial company we're publicly traded I'm not gonna even pretend to say that it's not very important to us to be revenue positive right we got 12 000 people that we gotta feed and and shelter and you know make their families happy and stuff like that uh but when I came here in 2005 I came to red hat because I am an open source person um I am a red hat I'm an open source advocate you know I uh my background my background is kind of bizarre if you already know the answer to this please don't answer but I'm just curious does anybody want to guess what I did before I got into IT are you really it's locked I mean there was a life I know it's like let me in I know we can't it's locked anyway so so anyone want to guess what I did before I transfer before I changed into IT back in 1993 no and no car salesman oh come on damn shots fired shots fired no what's that professional water polo player I don't know where that came from no I was a police officer what I was a police officer when I was when I was 16 years old I volunteered through the police explorer program at my local police department when I was 18 I became a corrections officer when I was 21 and can get commissioned I became a police officer when I was 24 I was like I can't afford to do this but one of the things that drew me to open source because I started back in 93 worked on you know novel network to date myself and then on windows because that was the next big thing when I when I got into Linux when I got into open source and I realized that there was this incredible community where you could get on IRC or you could get on at the time it was a use net you know comp dot os dot Linux dot blah blah blah you know you get on these things and you could ask questions and people would answer and it's like that's the guy who wrote the code like holy cow and then I was like hey that guy's asking a question or that gal's asking a question and I know the answer I can help them and that was kind of my introduction to open source was I can participate in this community like I can help other people that's why I became a cop and I learned that I can do it in the community as well and I'm pretty typical of red hatters most of us who work here are doing this because the communities that we participate in are awesome and we get help and we can give help and it's it's a really cool sort of karma thing it's amazing when I came to red hat we had 1200 employees now we have over 12,000 employees but I have I have worried over the years like how do we keep the culture how do we keep that spirit alive but myself and several other folks are actually involved in new hire and new manager training programs where we teach the culture and we teach the history of red hat so that as folks come on board maybe they didn't come from the community maybe they don't understand but by the time they go through our new hire orientation by the time they get into the trenches with us they do understand how critically important it is that we participate in these communities. Most of us are very passionate about open source sure there are some who just you know it's a job right but for the most part we're really good about hiring folks who are passionate about open source we're passionate about helping folks we're passionate about enabling people to do things they never could have done if they had to rely on proprietary expensive closed source software you know when we did the one laptop per child program you know that was one of the things that I was most proud of up to that point and I keep getting more and more proud of the things we're involved in like we put technology into the hands of people who could never ever afford it if if it weren't for the free nature the free is in beer nature free software that's cool man I'm a tiny tiny tiny little part of it but I am part of it and that is incredible to me and everybody at Red Hat most everyone at Red Hat is the same way and over and above that we recognize that we are beholden to those communities that that we get technologies from that go into our products you know we are responsible to to make sure that those communities are vibrant you know we wouldn't be successful if it weren't for you being here today and participating so we're grateful for that we recognize that we owe a debt of gratitude to the communities we would not be successful if it weren't for what the upstream maintainers and developers have done and it's important for us to give because we have been given so much you know I got I got a wife and two kids at home and if it weren't for open source if it weren't for these communities I wouldn't be able to put food on the table or I'd have a job that I hated you know but instead I get to do this I get to come and talk to you I get to participate in this incredible thing and that's that's a pretty pervasive attitude in Red Hat and you know I love it I when I came to Red Hat I thought I was going to be there for two years I thought I'm going to go two years I'm going to get some street cred I'm going to be a consultant I'm going to that was 13 years ago you know I love this place and I love that we are geared to being of service to upstream communities and and I'm not unique in that at all so we are honored we are absolutely honored to be a part of your community and we will we will do our best to be good stewards and to contribute and we are thrilled to participate with you so thank you very much if there's any questions feel free to ask yes so I just want to say I feel like Red Hat needs to do a much better job explaining the technical aspects of upstream first how so I found a blog post from Dave Neary in 2015 on community at redhat.com blog upstream first turning open stack into an NFV platform where it contrasted upstream first with the vendor branch model okay at all the other vendors are following and it explains how it's harder to get all your patches accepted upstream and then backport them to your product your product basically being like a legacy branch and how other companies say oh it's usually just to develop all the features on our branch yeah and and how that provides temptation to go proprietary in the future and similarly like I contributed the X to go project and X to go product for like a decade relying no machines open core and uh for for a while it looks like and and then showing off the next version of no machine was all proprietary and now we're you have to develop the open core I feel like Red Hat needs to focus on the technical aspects of upstream first I will I will do so the next time I present this thank you any other questions so it's all well and good that you are such a heavy participant in these uh communities and it's very clearly felt in a lot of different places but sometimes it also feels like um when Red Hat is participating in communities uh it feels a little weird because like the reasonings or when the discussions are happening it's like we get half answers or things like that so I'm involved in Fedora and a number of other communities as a community person and I interact with Red Hat there's a lot um and a lot of times it winds up being where I feel like I'm getting half of the answer or a quarter of the answer and with the part I'm given I don't think it's good or smart or anything like that but like I feel like if I had the whole answer we'd have a better discussion and maybe have a better result at the end of it yeah yeah that's a fair criticism um I'm not justifying it because I don't know the specifics of of you know what you've experienced um I will say it is an unfortunate reality that you know like I said we are we are a commercial company there are things that we have to do from a product perspective you know that that are going to um dictate the way that we do some upstream work and there are going to be times where we've got things that are going on that we can't disclose for financial reasons or for you know disclosure legal disclosure I mean there's a ton of things like that maybe that's what happened I don't know because I don't know the particulars what you're what you're doing or what you're talking about I will say though you know if that happens call it out and I mean don't be a jerk about it and I don't I know you well enough know that you wouldn't but call it out you know call the person out privately and say I know I'm getting a half answer here like this doesn't make sense is there something going on internally that you can't talk about like is there you know what's going on um we will do our best and there is an internal policy I mean and it's not even a policy it's an ethos of transparency and disclosure where we can okay and you know I mean if you run across something like that again I don't have any juice really but tell you know you've got Thomas at redhead.com man send me an email I'll help out if I can yes sir hold on wait for the mic so that we get you on video you're being recorded for posterity you talked about red hat being good with upstreams and I'm wondering if this is in opposition to being good with downstreams as a person who is a free software advocate and a person who works independently on free software projects I find red hat projects to be the most difficult to cooperate with in terms of as an individual coming into a project okay and as a downstream so like if I'm a user of say gtk right interacting with gtk folks is actually kind of hard and it gets harder the more red hat they are so like the more employee they are it gets harder they get harder to talk to harder to contact and so on and I don't think this is necessarily a malicious thing or like a problem but it definitely does seem like my experience with the debon 2 community back in the golden days they focused a lot more heavily on downstreams maybe the to the to their detriment on their upstreams but I'm wondering if there is a balance to be struck between upstream focus and downstream for focus yeah actually Steven wants to answer that would you hand him the mic so as Thomas mentioned some of this red hat has grown very fast in a fairly short amount of time and there has been a certain amount of we have seats that need filling and sometimes they end up with people who are looking at red hat as just a job and less of less of the passion that some of us old timers tend to have and like Thomas said there with with osas and with several internal programs we're trying to express to individuals more about why the community is so valuable and yeah there are times and there are some projects that are very heavily red hat employee-based that need still need help getting it that that is what they should be doing so there yeah there there are times when we don't do a great job of this call it out when you see it if you need to talk to me escalagetredhat.com I usually try to take those people and have a meeting and you know explain the light to them so make it quick I really literally have to go to the airport like right this minute okay this can answer can be done by any of the red hatters in this room or perhaps not in this room I am an ISV VAR partner of red hat how do I work to make sure that my business is noticed by red hat in that we do not have a cloud product and we're not the buzzword thing but if it wasn't important then I wouldn't be in this business thank you yeah we do have to go