Hi everyone, really happy to be here and it seems that we're now having some conversations with this projection technology, which may or may not work. So I'm Audrey Tang, talent's digital ministry in charge of social innovation. I'm really happy to be here and share with you some thoughts around social innovation and especially participatory civil IOT as we call it. What you're interested in the materials, all the materials are CI.PAIWA in CI.Taiwan at GOV.TW. It seems that it's stuck for some reason. Can we switch to HDMI or something? Anyway, here I go. So that's the website and as you can see we have put all the relevant information related to the CI project, the civil IOT project into this unified website address. And we do this not just for the civil IOT but also for AI Taiwan, for SI Taiwan, which is social innovation, and also for smart Taiwan and also for bio Taiwan and so on. And this shows a very different approach of policymaking context because it is not just the central administration or the municipalities. This is not just about one ministry or some other agency. This is a single website that outlines all the efforts including the municipalities and the various ministries involved and everybody can update their latest development on this bulletin board and for everybody to share. And so I'll be mostly explaining the policymaking approach, the collaborative policymaking approach that we have done in the past couple of years leading to the civil IOT Taiwan project which is a shared evidence-based platform for everybody to contribute. Mostly it's for areas and the municipal agencies have already shared brilliantly about how this enables an open API-based connection for everybody who are working in the front lines who want to have an edit lines report of where exactly is the way they're going, how the roads has been broken, and what kind of ideas or information are on the few reports there are. These are all aggregated in this corner. But in the background there's also ongoing efforts to collect air information, to collect water quality information, as well as earthquake prevention data. And all of this is aggregated into the National High Speed Computing Center using a cloud infrastructure that allows for people to contribute their insights and the resources that they have to the same platform. But my main topic for this talk is not exactly the features now which will only increase and you can check it all out on the CI website anyway, but on how those ideas can be and how we incorporated the civil society and the social sector's innovations using drones, using blockchain, using the cutting-edge technologies. Frankly speaking, the administration is maybe only a late adopter and waiting for the civil society to surface their ideas and can only be done in a way if it's fast and if we make the civil society feel that they are welcome to share their latest ideas in both the civil IoT, the disaster recovery, and also in the substantive data sharing. And so this is literally my office. This is near the Chamber of Commerce, near the Central Diamond Park in Taipei. It's called the Social Innovation Lab. It used to be a Taiwan Air Force base, but it's converted into this playground-ish office and it's co-created by hundreds of social innovators. And this particular soccer field is drawn by people with Down syndrome. It turns out that they're brilliant artists who see the world through a different lens. And so it conveys this atmosphere that everybody is able to contribute. Now, there are people working on various cutting-edge technology there and we provide the sandbox in which that they can share their latest innovations and think of use of public good along with other people. For example, a few months ago, there was some self-driving tricycles here and they just lived with people and interacted with people and they drive kind of slowly so it doesn't hurt anyone if they crash into anything. But it gets people accustomed with the idea of what we call open innovation because these creatures from the MIT down lab, they're not just open stores but open hardware so anyone who know anything about programming or electric engineering can change the mood light into a face or something like that to show how they're feeling toward our world. But also the data is released as open data so we can collectively form a new norm to interact with these creatures in the peaceful times. Like during strolling through the temple flower market, many people attending the hackathon thought of it as a way of helping you carry the pots of orchids or whatever as you stroll past the temple flower market and by the end you can hop on it and it can drive you home. But in times of disaster, in times of fire, in times of various other ways, they can be instantly repurposed and reused as ways to do scouting, as ways to distribute goods in a way that doesn't harm the people on the ground. And so all those innovations are there for people to add upon so they don't have to control or in other ways colonize the technology if they rather open up the technology for everybody to see. And so this space, our Air Force space, perhaps fittingly, became one of the premier training ground for drone operators for unmanned vehicles to test various interactions and possible drills that they have around maybe an explosion accident or maybe an earthquake and things like that and using good carrying drones. People here just rehearsed various ways of unmanned vehicles in a way that shows the people how exactly that works in a simulated fashion. And so more incidents about the Apple TV. This is totally not an advertisement by Apple even though I did work with Apple for six years before joining the cabinet. Anyway, yeah, seems it's working again. Can we switch to HDMI please? Just a second while we switch to more reliable technology. One of the wireless engineers I know prefers wired connection because there is something more reliable. You know what, let's just forget about these slides. It's obviously not working.  So the idea of regulatory co-creation or the sandbox underlies the formation of the Civil IoT project. The sandbox system which is installed in sandbox.org.tw is our main way of engaging with the civil society when they think of a new drone or a new invention that is once countered to the current regulation or the current law system. And on sandbox.org.tw you can see many applications, people who come up with innovations to solve interesting public issues that are currently held back by laws or regulations. Oh, here we go, it's working. Oh yeah, alright. So then simply by declaring that this invention is currently breaking some regulations or some laws, it could be around AI, it could be around unmanned vehicles, it could be using a part of 5G or experimental broadband that is currently not allowed for industry or commercial use. Anything can be challenged on sandbox.org.tw and there's hundreds of cases where the government, the central administration responds to the real needs of the municipalities who really want to relax some of the existing regulations that can be done in a way that shows everybody that this is actually safe and it actually increases the common good. And so the sandbox.org.tw is the front end to all applications, but in the back end we have various different ministries in charge of various regulations and they are designed to be broken. So, for example, the platform economy and financial technology are the NDC, the National Development Council, and the Ministry of Finance respectively. And now before the end of this year we're working with legislation to pass the sandbox of the unmanned vehicles or UVs. And I like in every other country where the Ministry of Transportation or Communication takes care of those unmanned vehicles. Here in Taiwan the Ministry of Economy takes these ideas of unmanned vehicles. So because of that we don't distinguish between drones and automated vehicles that are on the road or in the sea. You can have hybrid vehicles that drive and then vertically take off or ships that convert into a bus that are self-driving applying to the sandboxes. And we are already receiving a lot of interest from the new Taipei City, Taijong and Gaocheng City about deploying these innovative vehicles in a way that serves the public good and especially around disaster recovery. And the way it works is that anyone can apply to break a law or a regulation that is owned by the ministry or challenge any regulation from any other ministry by submitting a proposal and after a multi-stakeholder conversation if it is potentially to the public good then they are given one year to experiment and to show people how it works in the real world and not as a programming. So anyway, here we go. So the idea here is that everybody gets a year after the initial proposal period to operate under a law or regulation through the innovators' liking and with the blessing of the municipal support. And after a year of testing if it's not a good idea then we thank the investors for paying the tuition for everyone. In exchange we get a data sharing and a risk assessment so that the new innovators can try a different angle. But if this is deemed a good idea by the civil society or by the citizens then they are given maybe one extra idea to test on another municipal area or expand their testing range or speed. And up to two years of testing will result in the final verdict of whether we incorporate this innovation in the civil IoT or other support system or not. If this is blessed by the people saying this really is a good idea then all the regulations that the challenges must be readjusted to the version of the innovators' rights the new regulation. Of course if this is a law challenge if this is a law change then the legislators in the parliament need to have a substantial discussion which can last up to four years. But during those four years the innovator essentially gets a monopoly by allowing them to continue the innovation including the business model under the new version of the law to their liking during the sandbox. And once the legislators are okay with this new spectrum, with this new field with this new kind of eco and new kind of use and things like that of course competitors enter the field and we then successfully incorporate regulations with innovators. Now the key to make this work and if you are interested in taking this back to your country to make this work the key to this is the regional innovation system because people cannot if they cannot see those creatures or these new inventions in action then people cannot make sense of what exactly how it looks like. And so every Wednesday from 10 am to 10 pm I'm in the social innovation lab for everybody to meet me in my office hour to have a substantial discussion even a demonstration of their technologies that I then record and publish a transcript for everybody including the municipalities and investors and whoever else to see and then every other Tuesday or so I also tour around Taiwan to all the different potential counties or cities or municipalities that are willing to experiment with those cutting-edge technologies that are willing to make full use of the open data API we offer on the civil IoT platform and meet with the social innovators I think the left one is Hualien and when I do that people can dial in from say Taidong or any other corner and they meet together in a kind of conference room to actually see those innovations in action and always the 12 ministries related to social innovation are invited to the social innovation lab in a kind of large room they connect through projectors and see those local social innovators through my eyes so it's just me who travels but 12 different ministries are in the social innovation lab where we have some like good drink and good food we have a resident chef at 11 p.m. every day so it creates a kind of ideation atmosphere, a fun atmosphere for people to collaborate and they know that the ministry will always absorb any risk if they propose something innovative and risky and it goes wrong it's on my fault but if there is some good ideas and is to everybody's credit because all the transcript is published online and in this environment maybe to one bureau or another or to one ministry or the other it's very potentially likely for the public service to say but I need to consult with some other ministry and then they will send another text and then another text and then throughout all those documents not only is the time takes that is longer than the innovators expected but also contextual information that you lost during the transmission but 12 ministries into the same room literally and the people they want to consult are sitting next to them literally it's very difficult to delay or postpone the ideation process so whenever anyone from the local such innovation says but now we need automated vehicles to help agriculture delivery transportation disaster recovery then all the relevant ministries are there and they just poke around themselves and very quickly the answer which we require at most two weeks after each regional innovation meeting so that I think is the trick of making this work and that then adds back to the collective evidence gathering that is the civil IOT system and finally I would like to show you a picture which is part of the civil IOT system also this is the air box which is people around Taiwan donating their school or whatever and purchase those very cheap less than 100 US dollars sensors that detects air quality and as you can see by now more than 2,000 points in Taiwan and they all donate these numbers back to the government to the national high computing center but of course people if they have participated in such citizen science before it actually challenges the legitimacy of our environmental protection agency so this is kind of rare in Asia because many Asian countries will see this as a threat to legitimacy but in Taiwan when we see efforts like this we decide we cannot beat them so we join them and so what we do in a central administration is using partnering with people who own for example blockchain technology to take a snapshot of those numbers before they are stored on our national high-speed computing center so that people can know we will not change the day before the election so they feel much more confident in contributing such information and second we punt in with the research institute to make the measurement devices more precise while keeping them as inexpensive as before and finally as you can see this is maybe a map of Taiwan's digital divide so the environmental protection agency set up measurement devices in places where there's less citizen scientists our latest plan is to set up an air box here which is in the middle of the Taiwan Strait I think no citizen scientists will build air but because of offshore wind turbines we actually have something that generates electricity in that point and can report whether the air pollution comes from outside of Taiwan or is domestic in nature so in conclusion I think this kind of open innovation of using sandbox to collect people's ideas and sharing data innovation with the whole world in a way that anyone anywhere in the world can just download it from github to put those self-driving vehicles together to join the civil IOT data gathering platform I think this is one of Taiwan's main offer to the world by solving not only our domestic social environmental issues but documenting precisely how and why of we're doing this through the regional innovation system and also sharing through the sensor things API and other international standards with people who are like-minded around the world and who can all join Taiwan in bettering the environment and the society so thank you very much for listening


It's nice to give them a chance to win, but they're confident in how I'm moving on the mound and just able to get out. So I would have loved to go back out. Pitch count was in a good spot, but I'll wait maybe not understand the moves. So, keep it up. You mentioned earlier in this year, like setting a new routine between starts and things like that. How much do you attribute that to helping you sort of get on this run? Like what do you think has allowed you to really settle into the season, too? That routine really just helps me get comfortable. I know what I'm going to be doing pre-game on each certain day. And it really helps me kind of lock in what I need to work on each day. Next to Matthew Robertson. Jordan, coming to the ballpark today, knowing that Alcantara is pitching for the Marlins, do you feel like you have to bring it a little more, knowing that the other team is doing one of their best guys? No. I'm just worried about their hitters. I was a little worried about getting in the box, facing 99, but I knew my guy was going to get him.  I mean, I haven't hit, I think I hit 17 against Nova. So he's done heavy sinkers, but not 97, 99. So it's definitely eye-opening how hard hitting is. Thank you. Anything go next to Eric Bowland? Jordan, I know you guys don't necessarily think about it when you're out there, but does the lack of run support ever frustrate you? No. I feel like I run. I know they're trying hardest. They're playing great defense for me and really trying to put together good ABs. We're down the picture. We got them tonight, but I've always got their back. I'm not ever going to frustrate it at all.


It's not possible. So it's why this talk is more of a combination of Python stuff and I think I will make my point clear towards the end of talk. So this is it for you. Those who have a lot of people think that Python is very quick and still so much production ready that you can just make your idea and have something working and you can still deliver to have some value. So this is a mix of DevOps plus Python stuff. And just to give a highlight, what I'm going to talk about here, it's a big subject. I think a lot of people would be attracted to Django and related stuff. So basically what I wanted to highlight is Python is really a good choice if you are building stuff very quickly and it integrates to everything else very, very, very easily. So this is what I'm trying to help you out here and it will have some non-technical stuff also, which is the phase one, which we are going to talk about is basically, actually this is my intro. We can skip to it if we get some time. Basically you can meet me at your friend room if you wanted to have some discussion or if we are running out of time. So we are going into three phases, first phase, when you have an idea how would you formulate it to have a product. And then phase two will include Python related stuff and what you can do with Python, basically Django and how to integrate with the stuff mix, which works very easy. Phase three is definitely how to make it into production and why Django is one of the first choice when you are going for rapid development production and continuous improvement. So let's have the phase one. So you have an idea and now you have, it's abstracting how you make it concrete. So that's, okay, so all of this, what I'm going to talk about is a real product that we tried out and it's into production right now and we did it into a few weeks. So how we started off is something already something called hypothesis. So you have to have your ideas written down in this format, what, who and how. So I'll be sharing links and this presentation is already live on Google presentation. So you can actually click on the links and get a lot of details out of it. So basically what we are focusing here is how would you come up with your ideas and how would you make it more actionable. So this is how you should document our idea. I think you can read through it. So basically what you have is a business model hypothesis. You will like that, you know, okay, what you have to build. Okay, and then we kind of try to use stuff which actually gave us the extension items, the actual business objective that we have to build. So there's something on business model canvas. How many of you have heard of business model canvas? Okay, so this is a simpler version of business model canvas, lean business model, lean stack from lean stack. And this is how you actually start writing your problems and then you'll reach your solutions and which has some revenue streams also. So I highly recommend this before writing a single line. Of course, if you are into serious development, if you wanted to earn something, this is the way to go. Again, this is not a single stuff. You won't meet this once. You will meet this for all of your ideas, all of your problems. And one of the good thing about this is you constrain yourself to top three problems and top three solutions. So actually you try to visualize your product as lean as possible and that's the only way you could deliver some value within very short time. So I think this is a very top level thing that I would recommend if you are... So how many of you here are from a product background? Like, you know, they are building into products kind of, yeah. Okay, okay. So I think they will be really fun to live in this. So rules and rules are this, you have to iterate multiple times. You have to iterate in parallel. So like if you have two, two or three, you know, you should do it in parallel and then come up with something which the whole team agrees upon. And again, you don't need to use any commercial tool or don't need to use any high-five stuff to have this thing.  So that's it. So we have it now. Then what to do? How to execute it? So basically, we want the actions which can be measurable because until if you do something that's not measurable, you won't be able to improve on it. So next step is we use Trello. This is the real board that we use. Trello, I think how many of you are fan of Trello boards? Okay, cool. So I think no introduction to Trello required. Basically, I just wanted to highlight you how we use two boards and to manage stuff. And I think it's from user voice guys and we kind of improved upon it or maybe modified it. So you need a planning board and when you put down your ideas and features at very top level, then you have things which are going on right now. So, you know, the current team, you wanted to walk up on some things from it. And then we obviously had this is not in the original user voice board. But we had the infrastructure requirement which was continuously going on because we were in a always to production mode. So, okay, one, two days we have something will be there to fill the Amazon and the actual production site. So we try to do this. So we already we have a infrastructure requirement going on. They have bugs. So anybody or even customer reports something we put it down here first and then we prioritize. So that's it. And everything else is going to be for must be moved to a different institution before going into next board. There's the last thing this board. It's the kind of archive stuff, but we call it post evaluation. So we put down ideas there which we are not going to do right now, that kind of stuff. So what we put it there for reference, you know, the guys could actually, you know, pick it up and if they can bring it back. So it is a simple stuff. We write out this is really helpful to prioritize what you are planning. Okay, so this is it. And then we use the another general board. So from that up next, we actually come here, sorry, from last year and for execution team will come here up next. And here the devs and the other team members will pick this task. And it's like a simple to doing and done thing. But what we did out of this is next up doing and something called launchpad. So I'll discuss more about this why we are not having a done thing. Why it's called not long spread because we are following something called master base development and so long spread seems empty right now because we need a release. So all of this individual things are released. Okay, so each product registration what we did is this. So all of this stuff what we are in planning and then come here and finally we have a release. So this is actually a release. So this is what we sent out to customers. So that's about the execution and that's it. This is the whole code flow. Only two more. That's it. Nothing more, nothing less. So this concludes the plan, the execution and all that part. And so I'm going to talk about a lot of small stuff that you already love about and how we use it. So get everybody. So no explanation about it. So get and get branches. Everybody are aware about branches and all that stuff. So we just try to simplify stuff as we use master base development. That means master is exactly always ready for production. So I think it goes into master. It's exactly what the normal get flow is where master is unstable or developed right. But we generally only do price maybe for developer actually working branch and master. And so what we had a very simple rule is each brand must represent fully working release. So else don't touch. Okay, so if you are going to deliver it then only do it. That's the other way to look at it that don't write or don't do anything which is not going to go into production. Because this is what we are talking about daily and not twice a day release and that's into production. So that's it. So sometimes there are a lot of questions about if you are working on something big and how would you come at it? How would you actually do it without affecting other things? So feature flag, feature flag are like normal if condition. Okay, if there is a setting x equal to your enable feature one yes, then you have some four that's not. It's a very simple stuff, very simple idea, very powerful idea. So that's it. So this is how we try to use skip. And it's definitely the real mid-jango. So I have around eight years of experience with Java Enterprise Technologies and we had to do this project due to NLP requirement. And I think the guy just discussed those sentiment analysis and there were some few energy features we had to use. And that's why our product was actually based on jango, based off of the new. And so we found it very, very effective in coming from the background of Java Enterprise application. That is one of the best framework which provide out of production readiness. So Django people understand that yes, this is not a toy. This is not something that you use it for thrower projects. This is serious stuff. So I'll just highlight few of those things which is the Django is definitely awesome. But I'm just going to highlight very, very minor things, few of the things which help us to do a very quick integration with the things. So, sorry, I'm not going to Django a lot of features. One of the very important part we found out about Django is it has a lot of extension points at each and every stop. So first point was and whenever we have a feature which is we, that's a simple mode to make it work first and then make it correct. And Django allows you to do this if you were into some other thing, but you might have to relate a lot of stuff. So that's what we like about Django at very high level. Okay, so as I told you, I'm only going to highlight very few items which I directed us. It's one of the best thing is data migration and Django is seriously serious about data migration and which obviously have that whole previously known as Django South. And now it's part, kind of a part of the code from Django 1.7 is Django Migrations where we can actually go to full manual mode. So I like this, I think not clear to the table, but it's a simple migration method. Those who are not aware with Django Migrations, it is a simple class with two methods for migration and backward migration. And you can actually write literal SQL DDLs. It's that simple if you want to. That's the point I wanted to highlight it. It has a full auto mode. You'll never have to write any SQL statement ever for migration. Yes, but when you wanted to make it correct, when you want to have the power, that flexibility, it still allows. It's that simple. Okay, anybody wants to go into Django Migrations? Okay, no, okay, so we can take it up later. So that's one thing we wanted to highlight. Okay, and so I'm talking about now going to talk about a lot of things which integrates very easily with Python and specifically Django. So PostgreSQL, we are from Django Mig data background. I was part of a particular team for HBs at Y. And when we came to this land, you know, from the last two years, I'm throwing this out as a word that used blue elephant before trying out yellow elephant. Anybody knows yellow elephant? I do. And blue elephant? So yeah, I think yellow elephant is a mascot of Hadoop and blue elephant is a PostgreSQL. That's the icon, okay, if you see. So it's a simple thing that a lot of friends, you actually don't need to scale that much. Okay, you think that you need that much big thing, but you don't. Okay, that's the saying that you hit a business scalability fast before you hit a scale. So that's how you start with. So start with yellow elephant, start with PostgreSQL. Again, PostgreSQL is a very, very, very giant stuff. And although it's a very easy to integrate with Python and Django, I just wanted to highlight one feature. It's a full text search. You can see that this is one of the powerful stuff which you can do with just a few cricks, a few command line instructions, and you have your whole current Django PostgreSQL server serving you the full text search engine. Okay, you don't need solar or any complex stuff, service running. You don't need to learn API. You don't need to build index. You don't need to maintain index. Everything has been taken care. So that's what I wanted to highlight that. Full text search. There's a small 200 line of port which gives you text. This is Django ORM. It's so powerful and so extensive of that. This 200 line of port actually allows you to integrate full text searching and indexing with your normal RDVMS tables. So if you have a table, let's say here we have a table and there's a text field passed with the text. And so just few configuration options. And you have a full text search going on onto all of this. Automatically, you don't have to do any line of port even by injection or things. On top of it, when you wanted to search. How many of you work with new scene or something for full text search? Full text search? Anybody? Okay. So that's a lot of you need to learn API. There's a REST API call you have to make. And here you will write normal like SQL query likes. And that actually does the full text search. So if you wanted to go into detail when to use it. There's a full detail discussion that I can use. You can go and read when to use, when not to use. So this is again PostgreSQL's big stuff but it's a very quick integration. You can use right away in your app. Just 10 line of configuration and one line of configuration for models. That's it. One of the cool stuff which we are. This is one of the things which picked up in our organization very, very quickly. Is the interval. Okay. How many of you work with configuration management, deployment, automation, pop-back? Okay. So I think this is the kind of simplified answer to most of the questions which Puppet tries to answer. It's as cool as it technically uses inside Python inside. It's based on Python. But you don't need to look into the box. It's so simple that it's simple as a search. Okay. We have to provision a lot of services around a lot of machines. So you have to say, okay, Django is always here. You already have a service there. You have Engine X running somewhere and all that stuff. So at least you have, if you go to production, you will at least have more than one machine. At least one more than one machine. So how do you make sure that when you deploy something or you want to do rapid development, rapid deployment. So how would it help? So it's so easy to integrate with your normal Django application. Why I wanted to highlight the folder section. Okay. How many Django is here? Django. Okay. Cool. So if you are aware, like in Django, if you are going to build modules, it's a folder section. You have to follow some patterns. And if you use .py and models.py, there are filenames and folder section. If you learn once, it's very easy to make modules. And that is the same stuff if you look here. The roles is actually modules. How would you want to set up your services? If you see very popular, I have DPE back kind of modules. And there are tasks which has to be done. So the whole structure is, if you just spend like 20 minutes, if you are Django guy and simple folder structure is exactly kind of very similar. Okay. So what would you do is, one thing about this is it's agent class. You don't need to set up anything on your machines. You have 50 production machine or 2 production machine. There is zero installation required on actual machine. You start with a blank machine, blank AWS machine. And you point out that it's well from your laptop to make it fully deployed. All of your application with database, with migration, with web servers, everything done just via one SSH connection. Done. Okay. So that's it. One of the powerful things about what we require. Like Django has a setting.xml file and you need to customize for environment and all that stuff. Right. So how do you do it? So they have a simple markdown template, which is a Jane Inger tool, some template, but it's as it is markdown, just like you're editing it pages. It's very simple. And yeah, I already highlighted it's a modelized, it's the same as modelized Django. Okay. So, okay. And so this is just a showcase how easy it is to let's install and configure a database web server to allow database connections. So you have host file, you have variable file and HP. This is the host grace configuration file. If you see I have, you can write down a very simple follow to customize your configuration file. It automatically does all this variables in place and all that stuff. So, you can come down to me later on if you want more detail on this, but it's just a showcase how easy it is even to do complex configuration like this. Next thing what we use very, very frequently with Django is, Django is sometimes you need to run some external services. So it's a hope most of you guys know Django CLI commands. It's a, you know, Django supports a CLI and you can run your custom logic. And so you will say Django manage.pun, your command and your custom code will run. So you can basically integrate all those CLI commands or your application. Let's say we had a use case that we have to run a Java service and Django app has to interact with that service. That's fine. You can make it very easy, but how do you ensure that the Java service is running properly and how do you make sure that when you deploy the things are latest into the Java service also? That's a simple thing. It's a simple supervised configuration, one configuration file and a few lines of Ansible code and you are done with. So this is the Django integration with non Python stuff. How do you do it? Okay. And then we have phd. So this is where we are integrating. We are targeting that. Okay. We try to do a lot of stuff repeatedly daily and we still have a running system. So technically speaking there is no phase three. You have to have same execution tool and design most support continuous integration just like Django. If you don't have a proper migration module, database migration, you have to do a lot of stuff manually. So you pick up a tool which actually allows you to do continuous improvement. So most of the tools what we discussed already provides that. So the learning is that you know separate phase three. When you are doing phase two execution, you have to make sure that you use right tools which allows you to do rapid things. Which allows you to do repetitive things, repetitive elements, continuous deployment. And so, but still I'll just give you two hints which mostly falls into monitoring part. Not technically measurements but you can use a very simple Google Python tracker. So we wanted to have all of the Django application internal APA performance metrics collected. So don't need to write a lot of stuff there. There is a hundred line of Python tracker code which you can pick up and by that you can actually measure each of your Python method performance with some metadata. You can actually log some metadata also. And you can just create a fancy report in Google Analytics or some other reporting services. It's static. So that is one thing we use heavily to measure the APA performance in Python. And one of the last tool is, this is for monitoring your life services, load average, open file network stack. This base and our favorite stuff is tailing and searching logs. So you have a production machine, you've got some issues or services that are slow. How would you find out that, you know, what's the state in that machine? You can just point to that machine, have the PSS installed. PSS is a simple Python tool which runs in the background. We configure it with super wizarding so it always runs. And, you know, whenever you have issues you can go to Python process Django. You can go be with the PSS process, see the open file, network state, everything. And some extra things to log files, everything from your browser. So, good one. And finally, covering some quality analysis and testing parts. So basically, definitely you have to focus on things when you deliver something. You have to test and make sure that things are working fine across releases. If you want to do multi-day release, you need this. So basically, what we did is we created a simple. This is a single page AngularJS app. AngularJS, anybody have tried out AngularJS, you're cool. So, this is a simple HTML page with some AngularJS goodies, which creates a fancy chart and reporting. Basically, it is all open so you can pick it up right now. And the good thing, what we wanted to have is you need a simple shell of Python existing test case or existing some message file with you. And it should produce a JSON test output. That's it. That's it. Test one, pass test two, fill some description, that thing. So, it's a very, very simple JSON there. And if you do this, you will have this fancy report. So, we do this on a release basis and we just keep it there just to make sure that everything is not breaking and that kind of stuff. So, that's it. That's it for me. One or two quick questions? Yes. I think we have two, three minutes. Any questions? So, it seems so easy, but it's actually dangerous to put people who don't know what this means behind the scenes, right? Yes. Okay. Okay, thank you guys. Again, thanks a lot.


Thank you very much, Chris. I'm afraid we won't have time for jokes because I have 30 minutes and like 90 slides. So Hi, yeah, welcome to my talk Can you hear me? Yeah, you can there is no crazy game here, but it's okay So my name is Sebastian Witowski and today I want to talk with you about Continuous integration or to be more specific about how you can optimize your CI pipelines Setting up a CI pipeline is not the easiest thing to do Because unlike with a local development and now you have to debug things running on a server that you don't necessary control and on top of that there is this additional layer of complexity that You have to configure various services together using some kind of configuration format that your CI provider requires you to use But usually with some templates that we can find on the internet we can glue together some reasonable setup and That might work well when we start our project But as the complexity of the project grows also the complexity of the CI starts growing So we start adding more tasks and more tools We start building different versions of our release packages. We have more and more tests and It starts to get frustrating to wait for the CI Run to finish before we can merge your code or for example to Wait for half an hour and see that your pipeline has failed because you have like unused variable And that made your linter unhappy So in this talk, I want to take a look at a few different ideas that you can consider when it's time to improve your CI setup First we'll take a look at some improvements for Docker images Then we'll talk about running things faster for example by configuring jobs to not wait for other unrelated stuff Then we'll have a look at not running Unnecessary things and stopping them earlier and then finally I will share some miscellaneous tips and tricks depending on how much time we have left If we want to discuss continuous integration, we have to choose one of the existing implementations because Different CI providers like github actions githlabs CI circle CI and whatnot. They all Require you to use a different configuration setup. I mean the general idea is the same You write some kind of a config file, but the way you write this config file differs between different CI providers So you cannot take let's say github configuration move it to githlabs CI and expect it to work out of the box So for the purpose of this talk, I chose githlub Because this is the platform that I have been using most in my recent projects and also according to this Reddit survey It's still the most popular option But I am in no way related with github. I was not paid by them to come here I know that they have paid plans, but everything I will be showing here can be fun can be used with the free plan We also need some code that our CI will run on So I created a simple project that you can find under the link at the top I will have all the links to the slides and to the this project at the end. So don't worry This is a Django project with a simple to do up if you don't know Django, don't worry You don't have to know it to follow the stock I used Django for the simple reason that Django up by default is a bit more complex Than a bare-bone flask or fast API. So we will have some migrations to apply and We will also have like more files lying around. So it feels a bit more real world But we just need this project to have something that we can run our CI on so I'm not even going to explain you the code That's not important What is important is that I have for example a bunch of random dependencies that I'm installing to slow down the build process I Have some tests that are sleeping or performing some large mathematical operations. So also they are slow We also have a build process that uses docker and docker compose So docker compose will set up two services postgres database and a web container and The web container is built from a pretty standard docker file. So So we start so we start from a Basic image we just set some environments we copy the requirements run peep and then we start the server pretty pretty standard stuff and This initial setup has a pipeline that takes around six minutes to finish so Here we can see the six minutes and we have three jobs So first we build a docker image then we have a test job that runs the migration and runs the test and This stage is actually badly designed because the first docker run command here will actually build the docker image from scratch So I will I will fix it as we go with the talk and then finally we have a deploy stage That takes around 54 seconds and all it does is just prints this command so 54 seconds is the time it takes to just start the job container. So keep that number in mind And of course this example project is simple for the illustration purpose. It's not production great So maybe don't use it in production As I mentioned that my example project is using docker Docker or containers in general are now very well supported by most of the CI providers So if you can use docker use docker because it will make your Development CI and production setups much closer to each other And Even if you don't do develop development with docker, it's quite easy to wrap a simple application in a simple docker setup So if you do use docker The first step to improve your CI is to actually take a look at your docker file and make sure that you're not doing some Obvious mistakes like make sure you're using layer caching properly during the build time make sure you use tags So you know which images you're actually using in your setup And speaking of images which one of those two images is better So we have slim buster that is a smaller Debian based image And we have alpine which is a pretty bare bone Linux image. So who here thinks that alpine is a better image as a base image Raise your hands. We have around 10 hands up and who here thinks that slim buster is a better image Okay, more people The answer is it depends So using alpine image means that you have to install a lot of additional Linux Libraries yourself This will make your docker file a bit more complex and the build process will be longer But the final image will be smaller because alpine image is very small So it will be faster to push this this image back to the registry and pull it in all the other jobs On the other hand slim buster is twice the size of alpine Which is still not that bad because if I was using Buster that would be like 15 times the size of alpine But if you use slim buster the chances are that it has all the linux dependencies already installed So all you have to do is to run deep install and you're ready to go So yeah it will be a larger image but and It will take longer to download it between all the jobs But your build process will be much more simpler and in the end the choice should depend on whether your pipelines spend more time Building the docker image or actually pushing and pulling it between the registries But slim buster in general is a good choice. I would say So one way we can speed up our build time is to not build the docker image in each job But to actually build it in the build step and pull it in all the consecutive jobs And here we can see that even though we are building the docker image in the build stage In the test stage when we run this docker compose run command We will be rebuilding docker image from scratch because jobs are independent So the test job doesn't know that the build job already build our docker image So We can fix that by pushing our docker image to the registry at the end of the build job and simply pulling it at the beginning of some other jobs and Here we can see that the build job up stop it The build job build job is now Slower but the test job is faster and we are talking about like few seconds of difference So this might come from the fact that it just took longer to start job container But you're mileage mile my very so if your build process takes long and you have many jobs Then I would say it makes sense to pull the image from the registry But if you have a build process that is simple and fast Then maybe actually maybe it actually makes more sense to build the docker image at the beginning of each job Because the saving you would get from pulling the docker from the registry won't be that great. So you have to test different setups Another thing that you can do if you end up building massive docker images is to use a multi-stage build Multi-stage build means that you start a build in a separate image So you copy a bunch of files you install a bunch of Linux dependencies You run the build process that creates some cache files on temporary files and the size of that image grows Big but you don't really care because in the end you will just take the results of your build process and Move it to the separate image that is much smaller And the multi-stage build works much better in languages where the build step requires you to install a lot of additional Linux dependencies But the result of your build will be a single binary like for example in Rust In the Python world, it doesn't matter that much I mean if I were starting with Alpine Linux and I was installing a lot of Linux dependencies Then I could get some benefits, but I used SlimBuster that had all the dependencies already installed So in case of my particular example app, there will be no difference Let's leave docker for now and let's talk about CI pipelines. So how can we make them faster? So as your pipeline starts to grow and your stages starts to include more and more jobs You'll realize that maybe some jobs are unnecessarily waiting for other jobs to finish before they can start By the way, if you're curious, this is the CI setup of GitLab itself. So that's that's that's quite a big setup So you might want to take a look at the structure of your CI configuration and move things around For example, instead of running your jobs in separate stages one by one You can run some jobs in one stage in parallel So this setup and this pipeline is ultimately going to finish faster Except that what if one of the jobs in the stage fails Now we are wasting computing resources running other jobs in this stage Even though we no longer care about the results of static analysis or preparing the release because we have to go and fix test in the first place And there's actually an open issue about Canceling pending jobs if one of them fails in a in a stage But that issue is open since 2018. So don't get your hopes very high that it's going to be solved anytime soon So there are always some design considerations that you have to Consider when you're structuring your CI setup But in general, I would say that the time you waste waiting for a pipeline to finish is much more precious than build Than the than the cost of a build minutes that cost you a couple of bucks So I would say that if you can parallelize things you should parallelize things and There are actually two ways how you can move things between stages One is called directed a cyclic graph or DAG and you might know this concept from other tools So we've dug we can configure one job to start after another job finishes regardless of which stage they belong to For example, let's say you're you have a Python package and you're testing it under different Python versions and For some reason you don't want to use a tool like talks or knocks that will allow you to set up different Python environments You just build Docker image with Python 3.8 run tests there and run the release you build that Docker image with Python 3.9 run the test and so on So here we have a build stage that has to finish then we have a test stage that can start after the build is done And then finally we have a release sage In the ideal world all the build jobs will Run in parallel and finish in more or less the same amount of time in the real world They won't some of the jobs will take longer to finish and if you have a custom git lab runner setup You might actually have a limit on how many jobs can run in parallel So some of those might actually wait for their turn to start and if the image for Python 3.8 is already Prepared what's the point for the test for Python 3.8 to wait for this Python 3.10 to be built? We can start right away So this is where we can use the directed acyclic graph and connect some jobs with their dependencies regardless of what stage they belong to So here we can see that the corresponding test and release jobs are starting right after the previous job is finished And in terms of code all we have to do is to add this Needs keyword that specify which jobs have to finish successfully before this job can start And Direct data cyclic graphs are cool Yeah, but if you have a lot of jobs and you start connecting them throughout your whole configuration file It might be hard to follow what's going on So instead of doing that you can group some of your jobs together and create mini pipelines that can run as a whole So for example, let's say you have a project that uses a different text tag on the back end and different on the front end That could be like Django rest framework with react on the front end and then your back end code leaves in the back end folder you front end leaves in the front end folder and Your back end test probably don't depend on having your front end up and running and also You probably don't need to run all the back end test if you only change something in the front end So you might want to separate those two things So we can create two child pipelines and they are one is called front end And the other is back end They are pretty similar. So we'll focus on the front end. So under the trigger we say that configuration for this child pipeline is living in the front end GitLab CI and we also specify the strategy depend so If we don't specify the strategy and this child pipeline fails Then the parent pipeline will continue running but if we say that the parent pipeline depends on this child pipeline then if a child fails the parent pipeline will also fail and We also have the rules key here We which says that okay, we only want to run this job when something changes in the front end folder and Same here. We only run this pipeline when something changes in the back end What else can we parallelize? We can parallelize tests Most of you who used pytest are probably familiar with the pytest ex dist plugin So it will distribute your tests across multiple CPUs Which can give you some good speed improvement if you're running your test on a server that has a lot of CPUs But if you don't you can run your test across multiple runners So this is especially useful if you want to like dynamically spawn new runners instead of keeping one Large expensive multi CPU server up and running all the time. So here each runner can run in a separate VM And to do that we need to install the pytest test groups plugin and then specify the parallel option in your githlap config You also need to provide pytest with two configuration variables Test group count which specifies how many groups you're gonna have in total and test group which specifies the index of the current group So are we in a group one two three four or five out of five total groups? And this setup is nicely supported by githlap ci because we have environment variable for both of those things So all we see here is all the code that you need to run your test across five different runners in parallel And this is how it looks when we enable this feature So by the way my repo has different branches that corresponds to different things that I'm talking about So if we go to parallel pytest in groups, we can see that now We are down to five minutes instead of six in terms of the time But we basically used twice the amount of computing credits because I think previous time It was like six or seven computing credits now. It's 12 And here in the test we can see that we have six five jobs running in parallel So yeah It's faster, but it's more expensive because as we saw at the beginning just starting the job container takes around One minute so starting five job containers cost us five computing credits Other things that you should pay attention to is to make your make sure that your jobs are interruptible That is if you have a pipeline running, but you push some new code You want your pipeline to restart and actually run on the new code and there are actually two steps here Which is maybe not something that everyone is aware of so first make sure that in the setting of your project you select this on auto cancel redundant pipelines option and this will Restart the pipeline when there is a new code pushed to the to a given branch This option is enabled by default so you might not know it's there and take it for granted But if you have a good reason you might want to disable it But with this setting alone your pipelines cannot be interrupted They can they cannot stop in the middle of the job They have to finish the currently running job before they can be restarted and that can be a pain if your job They if your jobs takes a lot of time because let's say you have tests that are running for half an hour You have to wait for this half an hour to finish running tests even though you actually Want to stop and start running tests again? so you can mark your jobs with this interruptible true and This will make them interruptible So they can stop immediately when there is some new code and you probably want to have this option enabled for example for the build and test jobs, but you Don't want to have this job enabled for the deployment job because you don't want to end up with like Partially deployed code to your server so in in case of deployment You probably want to like finish the current deployment and then start a new one Another thing is to stop your job when it doesn't make sense to run it anymore For example, if your full test suit takes half an hour to run and already the first test failed And what's the point of running all the test if you know that you have to like run the test locally and fix those tests So you can run pytas with minus X this will stop the pytas to run after the first First failed test and then some next job can start You know what else can make your CI faster Not running things in the CI So one of the biggest revelations for me was that you don't have to run every possible check in every possible pipeline If you have a bunch of slow integration tests You can just run them on the main branches for example in my current project We have some integration tests that takes quite a bit of time But they check that all our apps are working nice together. So they are like testing different integrations But because of that our test suits takes around 45 minutes So we marked all of those jobs as slow and we moved it to a separate pipeline That is interruptible and that runs only on staging and on master branch And now the pipeline for a merge request takes five minutes to finish and that works fine I mean sure we don't detect all the bugs right away, but the Merge request pipeline finishes in five minutes and we eventually get the feedback from the full test full test run So that's fine. And you can also run some particularly slow jobs manually or during the night and Very quickly because I'm running out of time and we might actually have time for questions So I think I'm speaking faster than when I was practicing. I hope you guys can understand what I'm speaking My my talk will be on YouTube so you can play it with like those 075 percent speed So you probably know that you can cash things for example, you can cash the peep cash between jobs and that can Give you a bit of a speed improvement, but you can also specify the cash policy By default each job that uses cash will pull things from cash Run the steps that you define in your job and then push things back to the cash again But maybe for some reason you don't want to do that because let's say your job is doing something destructive to the cash So there is a policy keyword that you can use to either Disable pulling the cash at the beginning of the job or pulling or pushing the cash at the end of the job and Speaking of caching you can also select the fast zip compression method and that will allow you to specify the compression level for your artifacts or for your cash So here we can select for example the fastest method That will run very fast, but the resulting cash object will also be larger So it means that the caching will take less time, but like downloading this cash object in consecutive jobs will take a bit longer That works really nice with the cash controlling policy because let's say If you have if you have a pipeline setup where you Build your cash only once and then you push it But then you pull it in all the other jobs It actually makes sense to use the slowest compression method that will take a bit longer to build the cash, but the resulting object will be Smaller so it will be fast to put in all those next jobs If Docker is too slow to build your images consider using a different build system. There is build out There is canico. They have very similar commands. They actually have a bit of different features So maybe one of them is better for you You can also use your own runners And this is a very vast topic that could take a separate talk to talk about in in details But using your own runners first of all will save you cost because you're not paying for the computing credits of you For using gtlapci and also gives you much more flexibility For example in my current project We have to use runners because we have some proprietary code that we cannot really push to gtlap So instead of running tests on gtlap we set up runner on our server So the codes are running on the tests are running on our server And we only push the results back to gtlap and then gtlap handles all the displaying whether the job failed with all the logs And so on and just for fun. I checked How it is running a runner on my computer and I got actually some interesting results So the build job was faster and the deploy job was also faster. That's the build That's the deploy so deploy is twice as fast as it was on the gtlap VM But actually running test now took me like 10 minutes Which is super weird, especially that it was for those tests that were making a lot of mathematical operations So I would expect that my MacBook Pro with 16 gigs of RAM and 16 CPU cores Would be more powerful than a small VM in a cloud that gtlap CI is using but apparently Apple scum me But jokes aside this drives a point home that CI is a complex beast Especially if you don't do DevOps on a daily basis as I guess most of us don't do it because this is a Python conference So adding a custom runner gives you flexibility, but it also gives you another layer layer of complexity for HCI Okay, let's wrap it up So here are some key takeaways. I want you to remember from this talk learn concepts not tools Even though I was showing you how to use gtlap CI here. I think whatever I showed you here is universal if If you know how to use something it's just if you know that you can do something It's just a matter of figuring out how to do this in your CI setup There are no silver bullets in terms of a perfect CI setup Should you use alpine image and install linux dependencies yourself or should you use? Debian image and deal with the fact that the image is larger Should you pull your docker image in order all your jobs or should you build your docker image from scratch? You know all your jobs. I don't know it depends on the setup of your project Not every check has to run in every pipeline slow jobs can run manually or on the main branches if there is new code available interrupt and restart the job and Try to make your merge request pipelines fast and your main branch pipelines thorough And also if you think that you set up your CI once at the beginning of your project and that's the last time you touch it think again Not updating your CI is the same technical depth as any other technical depth in your code It will make your code review slower It will delay important feedback and it will make your developers more and more annoyed So give yourself a favor and check from time to time. What can be improved there? But overall a well-designed CI can be a great tool in your daily work. Thank you very much for listening. I Think we have a time for question or two Anybody wants to ask a question. There's microphones in the room and anybody can come and ask a question if anybody has a question online I'll So it And if you don't have questions now, you can always find me online here is the link to the repo here is the link to the slides Any questions on this court? Okay, I Knew I was talking so fast. Oh, there's a question Can you maybe speak to the microphone because I think it's recorded In terms of the multi-stage docker builds that you mentioned before do you find them useful? And what scenarios what problems are they solving for you specifically? That's a very good question. I Never use them. I mentioned them because I know that they exist and in some Particular cases where you have like a huge build step They would make sense as I said, I don't think in Python They made they made that much more sense when mostly like Java Rust and things like that But yeah, it's a viable option Thank you for dog. Is it possible that you mentioned the Customer-runner so was it running locally on your Macbook? So is it possible that you build a docker image for Intel or something and then ran it on on your Macbook? But it wasn't in a multi-platform build but from the What is it arm in my book the silicon so is it possible it was so slow because it was emulated and not native So that's a good question. I knew I should have like debug. What's the issue? I think I was building the docker image fully from scratch there So I think it was using whatever architecture I have But I know that the runners have like a lot of options whether you want to run things in shell whether you want to like run it in like docker-in-docker and things like that so Yeah, I really don't know what was the reason behind it I just check it I was surprised and then I moved on with my life Yeah, thanks I think it might be the case like it's it's super slow on the emulated when you build it on like Linux server And then you pull this image and run it locally on Mac It's super slow and it's emulated. Okay. I have to check it. Thanks. Thanks. I think that's that's it Thank you very much for coming. Enjoy the rest of the conference