We're all here to welcome you and to welcome our two speakers, we'll spend 40 minutes with you and their names are Jan Chilohan and Jaroslav Praček. Jan is a developer and team leader of the NP Package Manager State and Jaroslav is the also developer of the NP Package Manager State. So please formally welcome them on stage. Thank you for introduction, I hope you hear me. Welcome to the presentation Developers to QE's of themselves. It will be mainly targeted for upstream developers who don't have any QE's and would like to deliver stable releases with minimal effort. We will explain how the continuous integration look like for DNF Package Manager. During the presentation, do not hesitate to ask about anything. If you have any questions, please just raise your hand. Or if you are shy, you can ask something afterwards to DNF team. This talk will be split into two parts. The first part will be covered by me and is about how the whole process of the testing workflow looks like and which technologies we have used and why. The last part will be about integration testing, which is the subpart of the whole process. At first I would like to ask you or do some research. Please raise your hand to everybody who is developing some upstream project. Perfect, it's about half of the people. So I'm right at the developer conference. And the next question, who of you who are developers have continuous integration tool integrated to your project? That's about one-third of the developers. So that's enough. So those of you who already have some continuous integration tool, you can learn about new concepts and tools, which we use. And those of you who don't have continuous integration already, you can adapt our tool. And the last question, who ever released the broken application to the customers? It's about the same amount that has continuous integration tool. I don't think that's a coincidence. We already did that and it's kind of sad that it happens to package manager that should update your system. How can you upgrade a package manager to the new release if you can upgrade at all? So there's the motivation to prevent broken releases. When you have more contributors to your project, you spend more time to testing these pull requests. But you can rather spend that time on developing cool features of your application itself. So invest on automated tasks. At this slide, we can see the whole process. Everything begins when someone, some contributor, does pull request at GitHub or anybody pushes a new comment to the upstream. There, the package build is triggered. Why we are building rpm package inside our continuous integration is the reason is that then we are sure that we can do another release immediately and that the package build pass. As a side product of this build, we get the unit test checked. And once we have the rpm packages, we can put it into our integration test system. This is, this is a minute by black box testing the whole application. The last of the test is that the code checker that checks for some redundancies or some violations of the code so you can persist the maintainable code. If this process happened by pushing some new patch to the upstream, we also stored that rpm packages and served them to the users as night labels. In this slide, there's described the whole technologies we used. The core component is Jenkins. That actually is continuous integration tool that automates the task. We have our projects from the NF ecosystem on GitHub and five of them are integrated into continuous integration tool. So everything is new for request triggered. It runs the Jenkins job and recursively rebuilds the other packages. With regard to package build, we are using Tito definition. That's the tool that automates the releases. With that, you can buy one single same command line, build the rpm packages of any of these projects. You have to just have one folder inside each of the project. I forget to mention that we use a Jenkins GitHub pull request builder plugin. You need to have this to have your pull request checked. Jenkins has many of these plugins. And unlike, for example, Travis CI, Jenkins is not a service. You need to host it somewhere on some server and we use OpenStack for that. Okay, let's move to the lower part, which is dedicated to the testing. We run Python unit test with some mocking, but we realized we need also some integration tests. And for that, we are using Docker technology to have a level reproducible environment. For study code analysis, we are using Web 8 and PyFlakes, which can find any violations against Web 8 standard in Python. And PyFlakes can find some redundancies of the code or some coding that leads to error-prone code base. So that's for the technologies we have used. I would like you to take two ideas from this presentation. Do not neglect testing and invest your time in a manually task that you do repeatedly, and invest time to automate this task. It will pay off in the future. You can reuse our whole DNF testing framework for your own project. There will be link at the end of the slides. Now, I would like to invite Yaroslav Marchek, who is the creator of DNF integration test. Okay, hi, everyone. Thank you, Hunza, for interacting with me. And I would like to... I have a great pleasure to present here something about functional testing. Well, in general, just if you think about the testing of your project, I think you start to think about, you know, to test each commit separately after when you merge it, try to test each nightly build, or to make a test before you make a release. Think about, is this really all what you want to do? Probably you can just have a dream that you can ask contributors, you know, to pass all the tests before they make a pull request. That will be a really dream, what we can dream all about. But if you want to wish to have a dream that will become true, somehow you have to pay for it, you have to offer them. You have to offer them something, what they will say, okay, I mean, not offer like, that they cannot refuse, but offer that they will accept because they want. Yeah, therefore, you have to start to think what they will like. You like that they will make a locally test. Okay, okay, they probably like that they will have local builds. Wow, automatic local builds.  Wow, I will try to cover in my talk both sides. The first I will just repeat what's going now workflow by a little bit different way. Okay, when the pull request appear on our server, we run Jenkins, it's triggered the job, it downloads packages after it makes a source RPM build, it sends to the copper, copper run all the tests, and after that you have RPMs, and after when the RPMs are ready on the copper server, okay, the all functional tests start and then it reports back to the Jenkins and the jackets to the GitHub. Yeah, that's the whole magic. Wow, all functional tests are based on the three, let's say technologies. Docker we use as a testing environment. I think you heard quite a lot during this conference about the Docker. We try to describe our tests in behave and also we use the Apache HTTP server to emulate the web repository inside the Docker image. You know, we try to do as much as possible to just emulate the normal environment in our testing procedure. Wow, our DDT test, I found that, well, it's become pretty famous. In the newspaper I found that, okay, they found the DDT already at the Antarctic coast. That's amazing, I think. If you start to think about Docker, mostly you heard the motto, which is build, ship, and run any application anywhere. That means that, okay, you built some working application, you distribute it, and that's all the job. Our approach with the Docker is a little bit different. We tried to build Docker image every time when the task is triggered. I don't know what's happened. Sorry. Well, that's a little bit different approach how to handle the Docker. Most of you or some of you already heard about the behave. Again, we choose to describe the testing environment. Because the behave uses something like they say that it use test written in the nature language style back at my Python code. It has an advantage that not only developers will understand the test procedure. Like, yeah, when we read the part of our test, yeah, when I execute bash command, DNF minus Y install test A with the success, then the package test A test B should be installed, and the package test C should be absent. I think it's quite understandable, yeah? It doesn't make much. It recognizes what to do, which variable has to be used, and that's just, it's executed, and you will then see the result. If we come to our functional test, what it does, yeah? The DNF, you know, it's used by the user from command light interface. Therefore, our test procedure is also used the same way. Therefore, it should the command to the command line, and we try to analyze what happened then. The test is run, we can say, in the three parts. The first part is the preparation. That's mostly mean the building of the, build of the Docker image. That's mean the installed or required component, upgrade components, and of course, install the tested version of the component which we run the test, and all of these tests has to be ensured. It's done properly. Building for the build, Docker used something that's called Docker file. Just it starts that, okay, it tells you from which image to start, what to run, you can copy it inside the directory or add some file, and you can describe what to do when you enter the container. This is not only way how to build the image, yeah? You can also commit the container as an image, and later on use it like this, yeah? Sometimes it's necessary because not all steps can be done during the Docker image build, yeah? For example, our unit tests requires TTY, and while doing the Docker build, you cannot use it, but you can use it during the Docker run. Therefore, you make a Docker run of your unfinished build of the image, and after when it's correctly finished, you just commit as a new image. In the second part, this takes place the run of our tests. Therefore, the old test description are inside the container, and we just say, okay, run this test with the Python 2 or Python 3 DNF version, yeah? Well, after when the old test passed, we came to the third part where, okay, you want to have a summary of the old test, what was okay, what was wrong, and of course, this is quite important part, yeah? You have to clean all images or containers because you can be quite surprised how easily your disk or system disk will be depleted by the running of such a test. Now we come to the second part. How to use our tool as for the local build? Okay, you want to run it locally. Therefore, okay, well, don't use a copper. Therefore, we just make a directly build of RPM. We did it into container. Therefore, inside this container is installed at all requirements, yeah? And when this is finished, we can start with functional tests. This procedure has some advantages. For example, what will be on the server side a little bit more difficult for the Jenkins? Yeah, it could be done easily and locally, like you have two components like in our case we have a plug-in score and DNF and you make changes in both projects and they are tied together. Therefore, you want to make a test of both projects in one container and this can be done locally. Therefore, you first build inside the first project install and the second project which requires the first one. Therefore, you have after that the test which can really work with the server project at one time. And there is another option that you can keep the image at the end. Therefore, you can easily enter this Docker image or run the container as a testing environment. You can see how if your changes really does what you want to do and doesn't make any mess. And that's what we can offer our contributors. The testing environment inside the Docker. If some mess happens, it doesn't interact with your host machine. It's inside the Docker. If you do anything wrong, you just delete the container. And that's all. You can forget about cleaning and maintaining the container. You just delete it. That's all. At this place, you can file our project which we use for the continuous English test. And well, we came to the end of my talk and I just want to ask you, please think about our philosophy of the testing. Please think about functional tests. Please use our work. Use our code because we live in open source world. Thank you very much. Okay, if there is some question, Onza, come. Yes? So if I want to run this test suite, what are the requirements that I... You know, what do I need to have on my system? Is it just Docker or something else? Yes. Yeah. Well, yeah. That's plan. No. No. Titer is installed inside the Docker. It used to be. Yes? Yes. Well, on the server side, there is the one part. Well, you send it to the copper. Yeah, that's... Yes, what? Well, it could take five minutes. It could take hours. That depends on how it's... It could take five minutes. Well, together it could take 15 minutes. Oh, no. Well, you know, this is server somewhere else and... Is it even projects or... No, no, no. The copper is the... This is the service which you call outside. And, okay, they have different, let's say... Yeah, sometimes the build is shorter. Sometimes it's the longer... It's not controlling it. No. No, you don't control it. But, okay, if we start to think about the local build, you can switch, if you don't like the copper, to the build always inside your Docker image. Yeah? Yeah, that's actually the component I forget to mention inside into copper you would just source package and it would build the binary packages for some federal distributions and architectures. All right, and then it will run some kind of unit test. Yes. Yeah, like, inside the spec file there's... Yeah, it's the script and it should run the unit test. And actually, in the database, it's a real progressive loop and we're currently rolling out... We're placing our work on an OpenStack Zoo project. It's part of OpenStack Intra. So, why can't I really recommend the... You can use that overall thing, is that you can write your test flow into so-called pipelines. So, in a check pipeline, you can put a really fast test in the fasted back loop and you can get the pipeline and you can load actually these costly ones. And the reason for that, so developer creates PR, then the reviewer, assuming the unit is fast, who plays a global merchant with a pickup, for example, then who picks it up in a Google as it's like a mediator between a pickup server and a Jenkins. And you don't need no pickup worker who learn more in serenity than orchestrated jobs. And the reason for that, so the code is still not merged in well-costly tests running. And then, assuming they're green, do merge all the stuff automatically and push to rural and French. So, it works really for this, like, huge project as well because you parallelize this testing of open-print PRs and you pass it into the code. Thank you for some good suggestions. Will you create a new test after somebody falls new back to cover that problem? Sure. I think that's the normal procedure that, okay, if you... So, are you still... Yes. For example, now I prepared the test for InstaRoot. At least the behavior in InstaRoot was changed. Therefore, I tried to, you know, to discuss or to check or it's checked, yeah. But to describe the tests that we do this next time automatically. If you spend all creating new tests or if you don't have many bugs... Well... Because I think for this, you need to queue it for creating new tests. Yeah, it's called queue it by itself, yeah. Eva, I think you doesn't need to cover by the test all of the bugs and so on. Yeah, I think the main... You doesn't need to test all possibilities, combination of the commands, options and so on. Yeah, it will be fine, but it's not necessary. It's good if you cover the most user cases, the first option, the second one, the combination of the options, like that they are in general handled correctly. Yeah, I think there is nearly no test that will really test all of the feature, yeah. But of course, I think the future will be that you could create by the pull request also the test procedure that will describe, no, not describe, it will prove that your commit, do what you really want to do. Do also create new tests for... I think that would be nice, but we will see how the test will work. You can have my pull request without this combination. No. Especially from you. No, no, sorry, I'm joking. Okay, let's distribute. Okay, thank you very much. We'll ask. Okay. I don't know what you're talking about. I'm sorry, but who doesn't have a test for us at the moment? Well, it's just... I don't know what you're talking about. I don't know what you're talking about. At the beginning it was that each of us had a test for us. I think that's what I'm talking about. I don't know anything about it. I don't know anything about it. I don't know anything about it. Well, of course. But at the beginning it was that each of us had a test for us at the moment. It was that each of you had a test for us at the moment, but at the beginning it was that each of us had a test for us at the moment. I understand again, two children. You are my husband? Yes. And the youngest one is still flying. We still have some kids coming. What are you doing there? What are you doing there? They are already there. It's been a long time. We took pictures in the evening. I have to get ready. They are not on the other side. They are not on the other side. What happened? Did you get some sleep yesterday? I got a good child's sleep. Are you in a good mood today? Yes, really. Let's have some fun. Do you want a bite? I have the red card. Here it is. It's my present. Yes, yes, yes. And you wanted to copy it? Yes, I did. Ah! Yes, yes, yes! Thanks. Thank you. Your welcome. I'm honored to be here. It was great. It was great. You're welcome. So, what do you want? I'm sorry, but I'm not here... You're welcome. I'm here for the first time. I'm here for the first time. You're welcome. I'm here for the first time. I'm here for the first time. You're welcome. You're welcome. It was great. We're waiting for you, we're waiting for you. I'm waiting for you. We're waiting for you. I'm going to go, if anything. I'm going to go. It's ok. Yes, I'm going to go. I'm gonna go to the bathroom. Yes, ok. You're welcome. We're waiting for you. We're waiting for you. No problem. Do you know what happens at the toilet? No. I've been to the toilet before.  Yes. You used to stay at my place. And I'm going to stay at my place. I'm going to go. I'm going to the toilet then. Yes, I'm going to the toilet. In the end, I think it's time to say goodbye. Thank you. I hope you are having a great evening. Bye. Bye. I'll see you in two weeks. Bye. Bye. Bye. Bye. Go ahead. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. One, two, one, two.