Olisi koko ajanut, että ollaan koko ajanut. Joten mitä olemme lannut, ei ole vähävä ympäristöä, koska ne on vähäväuhtoja, jotka on tullut kaikille. Ei ole vähävä projektoja. Ei ole minun. Jotenkin olisi koko ajanut. Jotenkin voimme nähdä IoT-solutiona mitään machine learninga. Jotenkin olen Markku from Google. En tiedä, miten ovat toisiin nyt. Ei ole mitään, näetkö ne. Miten minä olen? Minä olen Viet-Aksent. Olen Fynlanda-säntöä. Se on niin koulut, että me tehdään tällaista asioita. Me jäsenä kenttämme kenttämme. Meillä on toisilla asioita kenttämme kenttämme. Meillä on myös kenttämme kenttämme kenttämme. Tämä on myös kenttämme kenttämme. Minä olin tullut Nokiin. Minä olin kenttämme kenttämme kenttämme kenttämme. Minä olin kenttämme kenttämme. Minä olin tullut Pus Cloud Computing 2009. Tämä on se, mitä meidän pitää tehdä 9 vuotta. Minä olin tullut Amazon Web Services. Tämä on teknologiikin avangelista. Minä olin Google Cloud. Minä olen kenttämme kenttämme kenttämme. Se on kenttämme. Se on kenttämme. Tässä on kenttämme. Tässä on kenttämme. IOT ja data. IOT-tevaisee on todella pieni. Täällä on kenttämme. Täällä on kenttämme. Voisin loppuissa osallistua. Tällä on kenttämme. truth generally in Italian devices are really really tiny things. Because there's a lot of them. There may be thousands of them. Or millions of them. They need to run from battery power, from maybe solar power or from a battery for a year. But we don't want to manage them, because there's literally thousands of them. So if you think of Singapore, we have drainage canals. säädä, ne olisivat vahvistuneet. Tällä tavalla, kun perustelijat ovat kentäriä, se ymmärtää, miten paljon vahvistuja on, mitä vahvistuja on. Nyt ajatellaan, jos pitäisi tehdä ympäristöryhmäkirjoitukset kentäriä niin kuin se. Ja miten voidaan käyttäämään, etc. Most of the IoT real use cases in the world are kind of like industrial internet or machine to machine, really tiny sensors. That's an ESP 8266, I think. That's about $3. Let's say $2 to $5, so the unit cost is super small. They are very weak. They can only do very simple things like reader-sensor reading and control of motor like open a door. But that's the beauty of it. maintained order of it. They don't really have an operating system most of them don't have. They just run a binary forever!" That makes them super-reliable. And because they are weak and because they are cheap, they don't consume a lot of power so they can actually run from batteries for months even years. They have things like deep sleep where you can play the sleep and oh, they wake up and take a sensor reading. Send it out. Simple reading. Simple reading, it's 32 degrees again in Singapore, and then sleep again, right? So there's a lot of data coming. Why there's a lot of data? Because you might have thousands of these devices. When they're all sending data, you already have big data. Now if you want to manage this data manually, it's kind of difficult, not that good, it takes you a lot of time. So it's much better to let robots handle it, automation. So this is one way of doing it. You connect your devices securely somewhere. Cloud is one way, you can have your own data center if you want, but then you have more hassle. Anyway, you connect your devices securely somewhere. Why? Because the devices can't handle the data, they can't do processing most of the time. The second thing is that you want to collect the data centrally somewhere, so you understand the whole picture. What's going on with all my devices? So that's why you need to, in most cases, get a big picture in the middle. So you collect the data securely, and then you do things like real-time analytics. It's not good enough to analyze tomorrow. Yeah, we know it was a bad situation. If you have real-time analytics, you can react. Like again the Singapore flooding canal, if the water level goes up, then you do some actions like, I don't know, they open some gates, and the water distributes somewhere else. For that you need real-time analytics. And then one of the best ways now to actually get value out of your data is the brain picture on the bottom, machine learning. With machine learning, you don't have to do anymore if then else kind of programming. In the old days, we always used to do if the value is this, then do that. If the value is something else, do this. It's only okay for a few use cases. Machine learning is great at finding signals in your data. Us humans, we cannot really understand weird numbers coming from millions of sensors. It's just, ah, it's just a random mess for us. Machine learning's best thing is it identifies patterns in the data. It's much better than traditional programming at identifying these patterns. It doesn't matter what is the pattern, it could be like a connected car where it can start noticing like gearbox temperatures going up and maybe the, I don't know, some friction on torque differences in the gearbox. That can be used for predictive analytics, predictive maintenance. Based on these signals in the data, these weird data, we predict that with certain confidence after a while your gearbox will break. That's a very common use case. So that's difficult to do with normal programming. The other nice thing about machine learning is that you don't have to program it in a traditional way. You feed it data and it learns. It starts to recognize the patterns. There's many ways of doing machine learning. You can do it on your laptop with things like TensorFlow that Google has open sourced. There are some accelerators available. You can stick them on your computer and you get like GPU or accelerated chips. Or you can run machine learning in the cloud. If you use Google Cloud for that, we have a service called CloudML where we have specialized hardware to run machine learning models. Your models, but they run in specialized hardware called TPUs, TensorFlow processing units, the same ones we use internally. They're just more efficient. They run it faster and cheaper. And then with TensorFlow, real experts can write your own custom models. I'm not one of them. I don't know anything about TensorFlow programming. I have to learn it. I am a simple, very simple developer. I can use these. Many providers or many software packages provide pre-built machine learning models. There's the inception, image analysis, machine learning model and YOLO and these. For example, in Google, we have some machine learning models that can give you value with a simple API call. Like translate this text or understand what this person is saying in the speech, change it to text or what is in my video feed? Are there cars? Are there trees? Are there animals? So they are very easy to use without programming. How do people use machine learning? This is the global phishing watch. Those light dots, they are phishing boats. So they do things like, well, big data analysis and visualization of phishing boats. But they actually use machine learning. So they track the boats and they have made machine learning models that detect, based on a movement pattern, what kind of a phishing boat this is. So that's pretty interesting. And then they can do things like that. We want to eat our own dog food or drink our own champagne at Google. So we have very large data centers. They have cooling and heating and whatever, a lot of things that are consuming energy. And then waste heat is just waste. The data centers should be very efficient so that they are green and everything. So we have been tuning them manually for years and years. Then we wanted to try how about machine learning. What if we feed all these sensor readings from our data centers into a machine learning model? Could it optimize the situation even better? Make our data centers more efficient? Well, yes. They did an experiment. And the machine learning model improved the efficiency of the data center by 40%. That was just shocking internally at Google. And you know self-driving cars like Waymo et cetera, those cute cars. They are basically IoT devices that use machine learning in real time to understand things like the LiDAR and all these things. Very crucial, of course. So I wanted to build one. So this is the first real demo I will show you. I don't know how to build a Waymo car. It's a bit beyond my capabilities. So I built this. It's a connected car prototype for one of our automotive customers. I wanted to show them an example. I wanted to have a modular powertrain. So I used Lego. It's modular. So it has a modular powertrain here, very simple. And it has an ESP8266 microcontroller here. So very, very simple $5 chip here. And then there's an accelerometer here. So it knows the position and G-forces in three axes, X, Y and Z. Its license plate is G00613, of course. I'm trying to do some lead speak here. So let's see how this works. What did I actually build? I have the car here in my hand. It's connected securely. In this case it goes through a gateway. It's raspberry by here. It's connected securely to the cloud. And it's got real-time analytics and stuff like that. Let's see how it actually works. So I've got the car here. And if we go to that gateway raspberry by here, then I will just show you one thing. We have way too many cables today. Just bear with me one second. I will show you how simple this is to configure these devices. So we take a connected car. And we plug it into my machine. And we go to my Arduino. So this car has been programmed with the Arduino IDE. And you can see here that I simply tell the car where is the gateway. Where am I sending the data? And I will show you what kind of data I'm sending. So now what I'm doing is I'm flashing the car with a new version of the software. So I'm compiling the binary software for the car. And after it's compiled into binary, it will be sent to the flash memory of the chip. And then the microcontroller will simply run this code forever in a loop. So now we are uploading, connecting to the car's controller. Flashing it. And all I did was I changed the address of where we send the messages. And then let me show you the code. So it's very, very simple. There's a Wi-Fi client. So this microcontroller connects to the internet. There's a PubSub client for sending simple MQTT messages. And then there's the Accelerometer library. So I'm basically reading things like acceleration. And jumping a little bit here in the code, this is the loop. It's literally called the loop. It's getting the Accelerometer values for x, y, and z-axis. And then publishing those and sleeping one second. And as you can see here, this is the Raspberry Pi gateway machine, if you will. You can see that now the car has started sending data. So if you can read there, it says license plate G00613, x, y, and z values for g-forces and acceleration. And timestamp and stuff like this. So now our car is connected. We start driving along. But what happens if we have an accident? Accident detected for vehicle license G00613. Oh dear, we crashed. This is a really simple example of streaming data. And then acting on a threshold. In this case, if the acceleration of the x-axis, like going that way, was very high, then that's bad. And the idea here is that the car would also have GPS or GLONASS positioning system. And then it would send, it's streaming the data, especially if there's an accident. So then let's say worst case scenario. You have an accident, you hit your head and you are passed out. Maybe you're bleeding. I don't want to sound very negative, but it can happen. In this scenario, we have a $3 chip here, sending data up to the cloud. And then we can alert emergency services and they will start going. They see the G-force of the crash, they see the license plate, they see when it happened and where it is on the map with the coordinates. And the ambulance starts coming. You can't call the ambulance because you are, well, passed out. Let's say that you topple over your car. You're driving alone and then, oops. Oh my. So the car toppled over. So we can have different kind of reactions based on what happened. So the car went sideways on its roof or on its nose or something. Yes. So that was a car. And how am I actually doing real-time analytics? I'm using a technology or open-source software called Apache Beam, which is one of the packages or packages. It's one of the open-source software that Google has open-sourced. So we open-sourced Beam. This is the same that we use internally for our real-time and batch streaming data handling. And you can run Apache Beam in Spark or on your laptop or Flink, different Apache open-source projects. If you want to run it in Google Cloud, we have a managed service called Dataflow. This is actually my buildings, this one here, real-time analytics pipeline running with Apache Beam. So you can do things like real-time analytics very easily. Okay. So that was a connected car. So it noticed accidents. It was a simple example, but that's the whole point. The hardware here is a Lego block and a $3 microcontroller. But it's a connected car. Let's move on. So I joined Google a year ago, just over a year ago. And I was really happy. Yay, I'm at Google. I will finally get yummy hardware. Like all the Googles I saw, literally that's what I thought. All, everybody has Google Glass. I will finally get the Google Glass augmented reality things too good. Or I will get some augmented reality tablets and then reality hit me, no. Literally nobody gave me any hardware like this. So then I remember at home I have funny stuff. I have raspberry pies with a camera. I have a laser projector. So what if I put these together? I'm an engineer, so let's get to work. So let's get to work. And I built this. It's a cat detector because you know the internet is built for cats. Internet is pipes full of cats, 70% of them. So we have a cat at home. So I built this. I built my Raspberry Pi. It has a camera here. I take a picture of our cat. And then I send the image of the cat to Vision API, which is a machine learning model. And it has the image of the cat. And it will send back, most likely, a label cat. So an English word cat. But because we are international, I take the English word cat and I send it to another machine learning API, translation API. And I get back, for example, Mao in Chinese or Kojani in Korean. And then I send it to GTTS, text to speech API, this word Mao, for example, in Chinese. And I get back an MP3 or streaming audio connection that has Mao as a spoken word. Then I use the laser projector. That's the augmented reality part here to project the metadata, like cat and Mao, somewhere in the real world, near the real object. So I like augmented reality. And my device will also say Mao like that. Okay, let's see if this works. I'll grab the speakers. And I will show you first a quick demonstration. So I needed to calibrate my system. So I'm projecting four phases of my manager on the wall. And then Google Vision API is able to detect if there's a person, where is the face. And then give me the bounding box coordinate. So in this case, with this way, I can actually align the camera image, what it sees, and the laser projector where it projects. So I can have alignment. So you inspired me with this idea, so I call this Oivind calibration. That's my manager. And you can see that these phases of Oivind are actually projected by the laser projector. So at home here, the Raspberry Pi is sitting on top of the laser projector and that block there is the camera. So when it's working, I will show you a live demo as well, but I'll show you this video first. When it's working, it looks like this. I start the app. After this, I will show you a live demo. We put an object in front of the camera. It takes a picture, sends it to Vision API, Translation API. Change the language. Because it's just one API call difference. And see the laser projector is projecting in English and in another language, and also speaking through the Raspberry Pi. This was the biggest problem. How do I write Hindi fonts? Honestly, the biggest programming challenge for me was finding the font. All right. And cat, yay. The real cat refused, of course, because they are the master rays. So I had to replace the real cat with the fake cat for that. So let's now see a live demo of this system. I have to speak a little bit here. Hopefully you can hear something. And this is the same Raspberry Pi as in the video just now. This one, I think. And now here, I don't, of course, have the laser projector. I'm connected to the venue projector here, but it looks the same. We start the app, and then let's do the hand first. So this is now a live feed through the Raspberry Pi. We can see the same as the camera. So that's my hand. We take a picture. And goes to Vision API, Translation API, GTTS, three APIs. Huh? But this is the best one in Chinese. Hand model. Yes. And this will be featured later. You may be wondering, this is getting very strange now, this presentation. So let's see. Let's have only the doll in the picture and take a picture. Let's see what happens. Oh, fashion model. It's a girl. And she has quite a long hair. Okay, maybe one more thing. Let's try the bottle, water bottle, very common household objects. Let's try this. Yes. All right, that works. So that was a super quick demonstration of using machine learning models. With a simple API call. So augmented reality at home in my system was the laser projector. I will quickly show you how fast it is to actually call machine learning APIs. So this is the software running on that Raspberry Pi. For that demo you just saw, it's 367 lines of code. And basically all we do is we take a photo with the Raspberry Pi camera. And so basically we get an image that's very easy. And after we take the photo, we look how short this code is. Vision client, label detection. We just call the pre-built machine learning model for vision, give it the image, and we get back the response. In this case a Python dictionary that contains the labels like bottle and hair and girl with a confidence value. How many percent confidence? So it's super simple. So that's the whole point that using pre-built machine learning models it's extremely simple and fast to integrate machine learning into things like IoT. But how about custom models? I have seven minutes left so I want to show you one more thing. What we saw earlier with the Raspberry Pi here was using pre-built machine learning models. That's great if you can if it works for your use case. But in many use cases you need to have your own custom model for your own business problem. Now traditionally that looks like this. On the left are the machine learning APIs that are pre-built. Super easy to use, even I can use it. You make one API call, you give it the data and you get back the response. It's called inference. You don't have to do anything, it's super easy to use. On the right are the experts. That's the expert zone. Custom models that you create using TensorFlow for example. Of course then you get a custom model but it's super involving and you need to know things like TensorFlow. What is the difference? If you use a standard API with a pre-built model you get back a common categorization. In case of images for example you would get cat. I know it's a cat but I want to know which cat, who is it? With a custom model you could train a machine learning model, a neural network to recognize between your different pets. That's Bob, it's not skippy or fluffy, it's Bob. So that's good but it's kind of like this. Pre-trained model is not good enough for you. I should learn TensorFlow maybe next year. Ta-da, how about AutoML? AutoML is something new. It's a new idea which is kind of a cheat code and I love cheat codes. With AutoML you get a custom model without having to know TensorFlow. I still recommend you learn and I learn TensorFlow but meanwhile we can use things like AutoML. AutoML is kind of inception. It's literally machine learning models creating machine learning models. So we have something called the meta learner that's like the AutoML itself that's trying to create a model for you. And it does this by spinning up baby models and then optimizing and tuning them and selecting the best performing model that's in this example, the one in the middle. So it does all the heavy work for you. It does this hyper parameter tuning which I don't even know what it means. You get as a result a custom model that you can use. And there are a couple of things to learn there. AutoML uses something called transfer learning. So if you think of images, we already have the image machine learning model that we used earlier. The image model already knows the difference between cats and dogs and trees and dolls and bottles. Your images are probably not that different. So what we can do with transfer learning is that we take the existing model, the deep neural network and replace the last layer. The last layer is the one that gives the categorization. Which one? Is it a bottle? Is it a girl? Is it long hair? You replace that layer with your custom labels, with your custom data. So that's a kind of cheat code as well. The lower levels of the model already know the difference between a cat and a dog and a tree. So we can use those. And then learning to learn means that there's the meta learner. Which is the big model spinning off these baby models and selecting the best one. And because it's done automatically, that makes your life easier. So, well, we have a service called Cloud Auto ML, but the concept is the same. This is a generic concept. You give this service example data. So basically, in case of images, for example, you give it a lot of examples of handbags. These are all shoes, and then these are all hats. You just feed it like label data, and it learns. And it learns to recognize between those. When it learns the recognition, you have your own model, and then you call your own custom model with your categories. And then you get back a response. Yes, that's a shoe, and that's a handbag. Without TensorFlow coding. So that's my last demo. You may be wondering about this. This is my smart-ish building. This is a smart building. So it's supposed to look like a skyscraper. It has a lot of sensors. It has temperature, humidity, UV, air quality. It has lights and a door. It has two brains. It has an Arduino, which is great for the analog thing. So many of the sensors are analog, and the lights and the door motor are analog. So Arduino is good for those. Then there's a Raspberry Pi, which is a generic computer. It's connected to the internet, to Google in this case. And the Raspberry Pi is also controlling the Arduino with a serial cable. Good old serial communication from way back then. And the architecture looks like this. We've got our building right here. The Raspberry Pi is the connection to the internet, secure connection. It does real-time analytics, et cetera, just like the car, actually very similar pipeline. But there's one more difference. There's a camera here. There's a Raspberry Pi camera. I wanted to see if I could make an entrance camera for myself, which recognizes my friends that, hey, your friend like Jan is visiting. Hey, Jan. So I wanted to try if I could use AutoML vision for that, because I wanted to learn how to use AutoML. So for my building, I wanted to build this entrance camera. And I will very briefly show you the AutoML. So it's super easy to use. The idea here is not to block the product, but it's to show you how AutoML as a concept works. You give it a lot of data. You give it training data. You import somehow the data. And then you specify your labels that you're interested in. So in my case, I have three labels. There's this one, Barbie. So I uploaded pictures of Barbie. My manager thinks this is somehow very creepy, but I think it's funny. Then there's another doll, Jenny. These are my girlfriends dolls. They are not my dolls. It's us to be clear. And then images of me. So these are the three labeled categories of images I uploaded. Then this is my own personal tensorflow for dummies. I just press a button that says train. Thank you. And I get a custom model. Literally that's all you do. And then you can evaluate the success of the model. It gives you some percentages, how confident it is. And then you can do things like query. So you can use the model. So let's try. Let's try first by uploading that picture of Barbie and that picture of me first to the user interface. So these two photos. And I click predict. I have my own custom model now in the cloud that I trained with those example images. And maybe the first request will take some time because the service is in alpha still. So it hibernates your models when you don't use them. So the first one might take a bit of time, but then the next one is fast. So what we are doing now is that, well, we are waking up my custom model. It's still in alpha. And then we are testing it first using the user interface here. The web interface. And if we are successful, so we are uploading two pictures. One of Barbie and one of me. So you have the photos again. This is sample image one, sample image two. We uploaded those. Yeah, it's hibernated. It will take a bit of time. And then it will tell us which one of those three categories it is. So literally that's what it does. There's one more category which is none of the above. So none of the above category or label is like it's not Barbie. It's not Jenny. It's not Marko. It's probably something else. And waiting, waiting. So the beauty of this is literally that using the concept of AutoML, you can create custom models really easily without actually knowing TensorFlow. So it's kind of a cheat code. So that's the first test image. It's 100% confident it's Barbie. That's the second one. It's 99% confident it's me. Okay, so looks okay. Now let's switch to the last demo. I know I'm slightly running out of time, but I will be brief with the demo. Now we switch to the building. So now the image will be coming from the building. You can see it's building there. We start the client app on my building. I move the sound to the building. And then we boot the building. Yes. So I start the Python client app on the building Raspberry Pi. Raspberry Pi is the brain of the building. Reboots the Arduino. So Raspberry Pi controls the Arduino for analog things. And then it connects to the cloud securely using its keys and certificates. Welcome to Google Cloud. So this is the user interface of that smart building. And now you can see it's reading things like air quality, air pressure, humidity and temperature. It's pretty warm here, 30.3 degrees. And UV is zero because there's no natural light here. But then there's a funny thing. There is a camera and there's a doorbell. So let's try like this. Let's say that you have a visitor for your house and she presses the doorbell. And I just wanted to see if I can recognize my friends. Who is coming? Okay. So first vision API gives me the bounding box and I crop the image. Then the cropped image goes to AutoML. And if we are lucky, maybe it's still hyperinated a bit. The first one might crash because my timeout programming is not correct in the building. This is a prototype. So if the first one crashes, which is 60% likely, I will start the app again. But I just want to show you that I actually use machine learning in two phases here. This is phase one where I use the standard vision API. It actually takes a really wide-angle picture. And normal vision API tells you that there is a person and it gives the coordinates of the phase. So then I simply take those coordinates, I crop the image. And then I send this cropped image to AutoML, which is currently sleeping. So we wait for it to wake up and then the next one will work. I have asked the service team to please not hibernate my model. So we don't have these demo effects. And there's the crash, yes. It's a predicted crash, so it's not that bad. Let's try again. So my Doppel is not very successful because she's still waiting downstairs. By the way, we are rebooting the building. So that's new. So maybe that's the future of smart buildings. You need to reboot your house. Go cloud. Now it should work. See, it's very persistent. She's still there. Well, how about if Papi comes with a friend, Jenny? This is the weirdest Google presentation you've probably ever seen. And then two girls are coming. Maybe it's a party. Let's see what happens. Oh, that was bad. See, my model is not the best in the world. But let's say that I come with Papi to the party. Papi and party. Let's try this option. Okay, so I have a temporary girlfriend. Yes. By the way, just one thing I want to mention. It's not facial recognition. It's image categorization. But in this case, the image is look like those people. I'm not sure if she's a person, but slum did like a person. Okay, I know I'm out of time, so thank you very much. I hope you enjoyed it.


Okay, everyone give Mike Miller a big hand. He's gonna talk to us about conda Thank you very much for the great introduction. I Would like to talk about conda easy installs and simple builds just a real quick race of fans Who of you knows what conda is? Third and who is actually using anac are using conda Just about 20% thank you very much conda is Quite a few things actually it's an installer similar to PIP, but I think a bit better It's an environment manager similar to a virtual and but still ever a bit better. It's cross-platform. There works on Windows Linux and Mac It's not limited to Python so you can use it for different languages Actually, there are people using for Ruby and follow languages And actually it comes from the scientific community, but I think it can be useful for every Python programmer So it's BC licensed and typically you install it you either install me conda or anaconda I would talk about this different condas here So there's what's called mini conda, and it's a kind of a small bootstrap like version includes Python itself conda and a bunch of dependencies like PIP and wheel and some other tools and It provides access to many hundred libraries so you can install a lot of libraries with conda, maybe even thousands And you just need to say conda install I will talk about this process a little bit and there's something called anaconda which is Different thing. It's a distribution similar to like if you like a Linux distribution so you get Python packaged up with a bunch of scientific applications and This includes of course Python conda conda install and it's about more or less like scientific 200 scientific packages could pop off here So supposed to be there but scientific package about 200 or so and if you install it it takes about two gigs of your hard drive But then I think a lot of scientists need a lot of these tools And you don't have to fiddle around installing the old separately They all come in one thing and then the value pay a little bit price was some this space But this space is cheap and programmers time is expensive And there's something called channel. So when you install something this conda conda looks for a channel and turn it channel is a place somewhere Somewhere on the internet was maybe even your own machine. That doesn't it's also possible That looks for packages And the default is anaconda server, so we don't say anything typically looks there But there's also something called conda forge, which is a channel There's a lot of open-source packages by the pricing community But you can also have private channels, so you can have a channel that's your own name These are company name whatever and then you can install from this channel by and say say install dash see My channel my channel is the name of the channel and then it will grab the package from this channel will be included There's also a lot of configuration options, which channel has precedence and which channels look first You can configure this in a configuration file Okay, a few basic tasks I want to introduce you to what you can do with it That's by no means a comprehensive introduction, but gives you some feeling what you can do so you can install packages you can create and administer environments and you also can create packages yourself so Writing is that a pie as you know and doing a lot of steps to create a package? The first thing you can search for something So you say conda search and you type in the name of the package You're looking for I use pandas pandas is very much used package and scientific library and then Goes out there and it's searching places So I added conda added a few channels you see every dot you see they're fetching packages dot dot dot dot And every dot means looking for a channel So I have by a bunch of channels it's looking at and then gives you a long display I just cut out of a few things and you see the three dots there It's cut out and it lists you those then the package so everything this search looks for everything has pandas in its name And shows you the version that's available Though all these versions are available is a normal one dot a dot here means it is available And it's downloaded though it when you're down when you install something it puts it in a local repository And you install from there. You don't have to download it every time and the one with the star here was asterix That's the one I have installed right now here and you see if it has a dependency on numpy was a scientific focus tells you I have numpy one dot Ten and pysons three five and build zero and it comes from defaults Which is for my naconda, but there's also a version that comes from quanta for sure It gives you the channel here that comes from but it's very interesting can be very long list And but it shows you what you have so it's easy a bit longer here and by the cut out quite a few It can be several pages long easily Okay, the search can be it more fancy so That's a problem. I have a pit when you search for something for church for jungle It just gives you I don't know how much very very very long listing this everything has a jungle inside here You can say for the full name and now it displays only the package It's called pandas not all the packages have pandas in their name and now I get only pandas It's still pretty long list, but it's much shorter already And I know it that's pandas and that that's a good feature and there's more so you can also Limits a search or say I want to have only Even though I did it on my Mac. I can search for all the available packages for Windows Win32 and I also can specify the version. I want now shows him all the available packages for Versions 0.181 and it tells me okay here now it offers everything starting This is numpy 110 Python 27 up to one pie numpy 111 Python 35 So there's no dots and last race because obviously If everyone make it doesn't need to download it here So that's it's not there, but this would be available and would download it and they all come from defaults and there's more options If you want there's quite a few switches you can use and can make the search even more specific pretty interesting feature Installing the package is very easy You just say content store and you name the package and it goes through your channels and you can at the config Fizer the channel that comes first attacks on this channel. See now it provides me with a name I want to install this package and typically tells I'm going to download here doesn't say because I've downloaded ready Typicalist, okay I want you to download these packages and then I install these ones like mkl's mask on a library all the dependencies typically if I have a properly packaged Application in a conduct package and all the dependencies are there installed all the dependencies and everything's binary So everything especially a scientific field. There's a lot of C and fort on extensions and Compiling these things typically works in Linux it might work on OS X and it's right like it doesn't work on Windows So that's why it's and still I think most people still Windows users are the most users of Python I think on Windows as According to all the statistics I've heard about and a lot of scientists and a lot of engineers So I teach a lot of engineers and most of the engineer companies are mostly Some of them have used Linux, but by far the biggest majority in Windows So you need to support Windows users and it's a pretty good way everything is binary and you can Do this okay create an environment So this is the first thing installing something but very often we don't want to install in your what's called the root environment You want to create environment? And that's what this is command up here. You can see conduct create Dash n or dash dash name. So I create a new environment which I call my pie There are three five and I can specify the Python version and I say I want to have Python 3.5 and You can also have if you wanted to could have a 3.4 or 2.7 or even 2.6 so far it supports 2.6 2.7 I'm not sure 3.3, but 3.4 3.5 and They are pretty quick as soon as 3.5 is out. I won't take along 3.6 3.6 will be supported typically a new NumPy version is supported within 24 hours or something pretty fast And that tells me what it's going to do. There's it never does something this out of asking And it tells me okay I need to download these packages and I need to install these packages or the dependencies and then if you say yes Which I did then it starts creating this package and gives you this The car cut off here, but it shows you how long it takes and how fast is doing stuff Okay, now it created environment to in to actually activate Now I can actually list environments. I created environments and I don't have any one I've just a small selection of the environments. I have you say condo and list for instance It's one of the commands that shows you what environments you have and you see an environment It's nothing but a directory in your home folder in my my user folder under condo under condo amps So if you install mini condo will be mini condo amps and here's another condo amps It's just a different directory and these directories contain all the information about environment and you see now the root environment is a The one that's active right now. It has this Okay, you see I gave the trailer the pie data Berlin here Then I made a new for the tutorial I made a new environment and I don't mess up with my route I can install what I like different versions and everything is isolated Okay, activate an environment So depending if you're on a unix kind of system linux in OS X then you have to say source activate name my environment on Windows is just activate so typing source activate is pretty long So I just had defined a short alias But that's to be how we how we activate an environment and just type deactivated deactivated again And then the prompt changes and then you see prompt and now when you list it again, you see now the environment will be Activated you always can see which one's active Okay, and then you can say list and Then it gives you a list of all the packages you have installed You see it tells me again the name of the package the version Here this dependencies of numpy or not the Channel and this one is red because this when you see it's installed this pip So you want you will find maybe a few hundred packages there You can install this conda directly if not you can always install this pip and conda will recognize this one So it's not a contradiction to work. I love this conda pip we can use both And that's what I typically do if it's not available this conda I use pip and install this pip but conda writes these these packages and you that's why I say it's installed this pip Yeah, so you have the version is installed this pip and you can also install builds There's no problem. So there's no contradiction in this and you have this pip installed there also Okay This is if you're a kind of a consumer of packages you can install it can can environments There are way more options. I just show a few but now the next thing is building a package So you want to build a package and there are essentially two ways You can build a package you can just take a package from pi PI and Convert it into a conda package with a few commands or you can start from scratch Bring it with your own package Okay, building from pi PI is pretty easy. There's a skeleton first. You need to install Conda build so conda builds if you have an account that's there already if you have mini conda need to install conda build Which is a command line tool that helps you with the package and then you say conda skeleton pi pi pi pi my package So this one goes to pi PI fetches The package we have there and then you say conda build my package and that's pretty much it So it's not very difficult and it's built a package and the package is a it's a par ball It's a car PC to in this case. It's just one file with everything inside And that's it and now we can install it and per default it creates it for the for the platform You're on so it's if I do this as you can see here's now score as x and You can now install it and that actually two different ways installing it right now I can say conda install use local my package And you just will go to the place because it knows that your fault place We are put it and installed it but you can also if you wanted to if you had somewhere else You could just specify the full pass with me for just file name. This is exactly the same here But it could be a different way just if you download one of these you can always install of this conda install from the command line You can also specify a Python version So even if you have a different Python you can say okay build this one for Python 3 4 and it builds it for 3 4 Yeah, so even if it's pure Python it always has a Python version inside and this is especially useful if you build extensions and see or Fortran, which is very common in scientific field and This one builds everything for this You can also convert to other platforms you see I can say conda convert platform all for instance and just specify my My file and say output directory And if it's a plain Python version it will create a version for each platform for Windows Linux and Mac If it's this extensions is not that easy then you need actually need the operating system because you need to compile it So then I'm I don't think if there's a cross-compile at least I didn't try it I don't know but So so far I compiled on Windows on Mac and I think in Linux is the same to be Linux is easiest to compile it anyway So if you compile extensions it works and I even compiled Fortran extensions Which is to be a bit more involved and see Simpson and the books nicely so far so you can create this extensions also and then if you like to you can upload and You need the Anaconda client and now I Uploaded to the Anaconda and you can also make your own channel if you like So I don't have it here, but you can also create a channel and you just say Anaconda upload and just give full pass of this file and it uploads This thing Anaconda and other people can install it just by saying condo install name of your package and If you put it in a different channels, they have to specify the channel when they install it Okay this is building From a PI PI package you can also do from scratch So if you want you can customize things and you need a few things you need a meta yaml So a yaml file that declaratively Specifies what you want to do and Then you need a build file and there's two different ones if you're on Windows You have a build dot BAT that file a batch file and then Linux and Mac You have a shell file and you just use a normal setup dot py so you probably familiar with it So you just use the normal setup dot py and then you have additional information in the meta yaml Which also has dependencies in it Let's have a look at this file So the meta yaml looks like this is just a yaml file and has a lot of different Sections in there. You don't need all of them So you have something about met information about the package then you have the sources So you can say it comes from some hits URL something like this You can have requirements and there are different requirements your violence for building things and provides for running For instance, if you use science and you might need science and to build it But you might not need to install on the client machine because you just have this to be the extensions you have different dependencies for the tests what you do with the tests and then you have normal information about who the also is what the licenses and Okay, this is just a short part of an Adventa yaml I used so I used just my package name here and you say, okay The source is just depends where all two directories up. There can be a relative part for instance. It doesn't have to be a URL Requirements of Python and setup tools and then for running I need Jupiter and see here This is something you can specify this one I need it only for Windows and the same way you can specify Linux and Mac. So this will be only installed For the Windows version, but not for the version of other operating systems. That's as well Very important and you can use the same syntax for and other places. So you make it dependent on the platform with one file Of course, then you have this MIT license, whatever you want to put in there Then you have this build files and they're typically if there's a skeleton made it just you set it up And you create files that look like this and they're just you see You just call Python and set up to py install and they just call your set up to py the setup device Just the normal one as you know from the normal installation process But you can also add other commands in here, which is very interesting So if something is not available as a contact package in here You can do a pip install something and this will install the package and this will go into this top all So everything you pip install the you will deliver the same package. I had this problem I had the library that was not available as a condo and difficult just installed it and she could distribute it alongside There's quite a few things you can do here in this build file You can customize things or something doesn't work typically have a way to do it in build file some way or the other Okay And then just build it is a conduct with my package And that's it and all the other steps are exactly the same as with the skeleton So you can do upload and all kind of stuff install from the local machine operations Good That's it already so conclusions I think condo is a very interesting package and it's very much used in scientific field But can be really useful for all Python programs because it's a pretty good installer in a very nice package manager You can manage environments and you can also build condo packages It works together with pip so it's not a contradiction You can still use pip if condo doesn't supply the package But as you can see you can always convert a pip installer package into a condo package if you like They might make the installation more smoother for it for your users then you might actually Just convert pip package to a condo package Very well known in scientific community if you go to a sci-fi your sci-fi conference pretty much everybody will know a bit about it I think But it can be really useful For all Python programmers and I encourage you to give it give it a try Thank you very much, and I'm here for questions Because and ask them for me right first hand went up here Thanks, I understand I can use condor for distribution and platform independent package installation, which is great now if I Have I am a maintainer of a tool and I have lots of different dependencies Outside pi pi and inside pi pi and with npm and whatever and it would be nice to be able to just Use condo install whatever for me Would I have to package all those npm pi pi packages to condo manually or is It possible to I don't know say condo just install this and install it from pip because you don't have the package yet Depends there's just some things you can do so if you install a package just that that is condo Then this package will install its dependencies So if you and say I say I depend on pandas pandas has a bunch of dependencies and install numpy for me And so I don't have to do this to some degree you can do pip so far you have to as I think you have to write it in the In this built script you have to say pip install pip install pip install But probably the this meta yaml will be improved That's what I just saw a ticket that you can then even say okay This is supposed to be come from condo This is supposed to be pip and this might be a different channel so it's still evolving and not everything works perfectly. There are some Problems there's no problem. There's a perfect but so far you you can always have a walk around by putting it in the build script I think So what would be your suggestion for this kind of situation? I'd put it in the build script. I had the same dependency thing and I put it put a pip install in the build script in this build SH script and it worked for me so that was was the easiest one the other thing It's just you can always convert the pip Packages into contact package and then you can list them in a meta yaml and just upload them as long as they are BST MLT lesson shouldn't be a problem and anything just upload them to Condo crowd and that's it. It is just you can automate it You just need to list all the packages and then you can update it if it's in a version Just check if the new version and just build new packages. Okay. Thanks Thanks for the talk. So one of the selling point at the beginning that is that not limited Not only to Python right but all the examples have seen all the pattern packages So what do you mean by not limited to Python? So you can use this everything is declarative So people use it for Ruby and other languages so you can use the same thing and you can also have Dependencies that are not necessary Python for instance one package is called pie tables is the interface to htf5 And this depends not only a numpy, but also in the htf5 library There has to be in a different version and all the things can be packaged in there So it you can have depend DLLs char libraries You can include them in your thing that makes it much easier if you have a good if you have a package at the end Then the user doesn't have to do anything about fiddling around this include parsing Oh, it's kind of tough suppose compiling htf5. It's not the simplest thing So I tried this once before condo and you can easily take two days out of your life and you still haven't All right, thanks. Oh, thanks for your talk very to the point Given that recently we have been adopting by PI binary wheels They seem to solve most of the issues that condo seems to solve Other than the fact that you have a single command to manage your install packages in virtual environments. That's good It's nice, but what's the big advantage now that binary wheels are well taking off You write wheels is it's a great step forward as I say that two things not everything is in a real format yet and you might not have the right version and the right platform in the in the wheel and Condo can install wheels no problem. So you say people install and install the wheel So that's not a contradiction this way at this at this as far as I know the story goes like this Travis Elephant he's a CEO of Continuum which is behind this it's open everything's open source But there's a company behind it and he he asked Peter van Rossum a while a few years ago About this packaging and because I just go ahead and do it And that's what it did So it's kind of a little bit a parallel to some degree a bit parallel to PIP and wheels and something like this but it's I think it's more comprehensive and It's better and many has a lot of nicer features and PIP and wheels for instance the search for name I couldn't I don't know maybe I'm not smart enough I couldn't get PIP into just showing me this package It shows me everything shows me hundreds of packages for Django and this is not really useful They have to run it through grab and do your own thing. Why not just say I want it I mean just customize search and those things aren't this the condo so condos nice But you as I said you can use everything that pip and wheels and pipe your eyes You don't exclude it you can still use it and I do it very often because it does only 200 packages Or maybe now 2000 out there, but not 80,000 or 90,000 whatever it's on All right any more interesting questions for Mike. Here we go. Yes Best question yet Okay, so is there like any Standardization effort like are you working with Python packaging authority or something or is it going anywhere? Or will it be forever separate like conda? It's great supposedly, but only in the scientific community and like Everyone else this is bad Great question, but I don't know the answer Would be very nice I would I would like to see it this way that both but actually everything is open source and both communities could use the code from the other Easily no no problem. I think everything is BSD or MIT license So there shouldn't be a big problem to use code and the I don't I'm not really sure why this is not a bit more coordinated efforts I'm I don't know it would be very nice to have one stop and Join forces and do it together For these things and I don't know why it's the case. There might be some reasons and haven't we found out yet More questions more questions keep the dream alive everyone. I Have a question. It's like a two-parter So presumably if I install anaconda it's gonna come with its own versions of Python as well as my system pythons that I might have installed separately And I once had this problem where or you know I don't know if anybody else seen the problem where you upgrade your operating system and you upgrade your system Python and all your Virtual lm's are now broken because something something something which I don't really understand And you have to delete all your old virtual lm's and recreate them is anaconda gonna magically fix that problem for me or sub question If I upgrade, you know, if there's a Python 3.5.1 and a Python 3.5.2 How does anaconda manage the upgrade process there?  And there shouldn't be a problem. So if you work with the environments those environments are pretty much isolated under totally Isolated sometimes if you use command line tools and they're installed like in the scripts directory is from there If not, it goes outside So if you say pile in something and you haven't by installed pile in the new environment It might take the wrong version and gives you a lot of syntax errors because it's a pile in for two seven And you and the three five environment But if you install everything it puts everything in the directory and there shouldn't be a problem and you update if you work with environments and You will see if you work for while you will because environments anyway all the time you will rare Actually, maybe the best approach just have the route just as a helper and then work in environments So or isolated of course you can always if you can always reproduce the environment You can freeze environment it just just specify what you have installed You get a big list of installation and then you can give this this file to somebody else and the other person can say okay Install from this file and install exactly the same versions of the ones you can reproduce the environments and they should solve the problem I don't know if that could be some you can you only find it out if you test it really and do it But theoretically there shouldn't be a problem practically. There might be one. I don't know Is that only me that's ever had that problem of changing the system Python and all the virtual labs are broken So so I guess we're thinking that Anaconda is like kind of even more isolated from system pythons than than virtual end managers if It never has that problem Yeah, there's some voodoo there further research required any more questions. Yes. Yes fantastic. All right No, he wants to hear my voice any longer than absolutely necessary Are the environment relocate table or is there any plan for that? Can you repeat a question? I didn't get it are the environment relocate table. So I Installed somewhere. Let's say archive the environment and Uncompress it somewhere like another Computer. Yeah, so I just said it you don't actually copy the environment You just copy all this every all the packages are installed as a version. So but I showed you this listings here So one of this earlier things shows you this Whatever is installed you have this List somewhere do have a list. Yeah, something like this so you can actually Freeze no, that's a search add to somewhere you can tell tell okay export all the versions I have in the file you just get a file with all the names Equal the version and then you can give this fight somebody else and somebody else can recreate exactly the same environment Exactly the same versions You should work it's just a directory The problem is in not everything is copied though if like I have this pandas version I have may I use it in ten different environments, but it's only once in my in my Repository and not directly in the file It's just a sim link and you have to make sure that you copied it along with visit But I think there should be an option to copy everything just sip the whole directory and put it somewhere else It should work as long as the same operating system But I no guarantee but theoretically should work can try it out But maybe recreating is it's the safe of thing So if you recreate you can also say okay, it freezes everything with except the same version But you can also go in the file and edit is okay It shouldn't be exact the same version should be at least the version or even update the version so you can go in File and customize it and then still everybody who has this file would create and this nice thing is would work I can freeze on my Mac and somebody else can create the same environment or windows I shouldn't be a problem if I sip the file and would only be for the same operating system All right anymore for anymore. It is now or there is one okay You have to make it a question you would talk very very fast and Mike will talk very very fast Okay, so let's say that I have a package Python package on Linux that has That needs separate Installed C library and at and let's say that this C library Expects another C library very so what will come that it will Download this library and all of its dependencies dependencies and put it in the condo repository locally or What will that different ways I had the same problem typically what I did I just included in the file So I had this you can you can specify the Dependency so if you compile something I had for trying dependencies some shared libraries somewhere and you included so I'm not read myself. I'm not sure how to the best way to do it, but that that different ways including Shared libraries in there and you can distribute the shared libraries along and they will put the shared libraries in this environment And they wouldn't clash with your shared libraries because you might have to share live them or maybe different versions Then it works for a while and until it doesn't work anymore because it's a different version something is different So you can distribute shared libraries with it. There's different ways of doing it It had you have you still have to look at there's a lot of recipes out there how to do it And probably you will find one that serves your purpose Okay, let's give Mike a mother big hand. Thank you very much for a great talk and do it break


What is going on everyone Sam here with late in sports cards? We are ripping 22 Panini National Treasures Collegiate football hobby 4 box case break number 19 Random division style here is a look again at your random We've got Thank you so much monsoir dude. You don't have to do I really appreciate that. Thank you, buddy. That's amazing That is amazing, man. I'm so excited for you to get that in Boppe That is so thank you monsoir. I'm that's awesome dude. Thank you so much. Thank you so so so much That's awesome. Wow. That's awesome monsoir. Thank you, man That's awesome You can always find these results in The link in chat as well as the results page on our website. Let's get it guys state monsoir I really have to say thank you so so much on that one. Let's do it Good luck and see collegiate so just so you guys know anything that is not a Not owned a majority by one team or is a player that's still in college will be random to either those teams or the Everybody if they're still in college So empty, let's do it. Good luck here guys. We're gonna start it here NFC North there you go Tim How about a Justin Fields three out of five on the emerald congrats man in Boppe was that was Monsoir that was Monsoir such a nasty card. We've got going back to the NFC North. That is Aidan Hutchinson There you go Tim wife's giving a little good mojo there. It's going on Tim and your entire family Hope you guys are doing well Tim and Tim's wife We've got going to be NFC East that is 17 out of 49 Ryan Robinson, Jr. Dude. Oh my god dude Tim Tim Tim are you ready? Oh my god. That's so nasty.   That is nuts. Congrats, Tim Tim that is so I'm so pumped for you man. That is so freaking cool Dude, these jumbo patches are absolutely insane This is going back to the AFC West again, too. That is gonna be Ronald Jones the second See there's there's Nathan Nathan goes I'm going to bed and Tim says he's a Bama fan. Oh man So Ronald Jones is still on the chiefs technically so this will go to the chiefs I believe this is NFC North, but let me take a look really quick. I think he's a Viking Yes, go into the NFC North. That is Tim B. Again, Luis Sine.  That is three out of 25 Kenny pick it inscribed H2P Hail to pit Dude, what a what a box. Oh my god. That is insane.  So that will go NFC North There you go, man. Congrats on that.  That is amazing. Let's go dude. Oh my god Thank you guys so much Heck yeah, I've got it. We gotta keep ringing the heat that man Yeah, unless it's like out of five or something or it's like a crazy bull patch. You're good. Just yeah, go ahead We've got 72 out of 75 that is Michael Vic Vic will probably be NFC South I would think I just don't know how long he played on the Eagles for yeah, I'm assuming I think he played for the Falcons longer though Yeah, this will be going to the NFC South Vic one of one and a same Hal and wow. Oh my god, that's insane Nathan We've got nice six out of 99 how about a Trayvon Walker? Oh my god That's who it's going to do Tim Tim by scratch all my god, man How about 21 out of 25 on the silhouette signature? Justin Herbert boom 21 out of 75, excuse me Dude that is unbelievable AFC West. There he is There he is Dude, that's wild.   We've got this is NFC East going to John M We've got Cadarius Tony 11 out of 39 Tim goes bring my wife every time now honestly Alright, so I believe this will be a random we've got So we've got team quad materials that is AFC South Chad AFC North is Benjamin NFC. Yeah, so this is gonna be a random to these four divisions at the end everybody's in a different division So Stingley Jr. Joe Burrow Justin Jefferson letter for now. What's up Dave? Hey, what's going on man? I haven't seen Dave in a while. That's awesome. What's going on Dave? I hope you've been well man. I Believe this will be a random as well AFC North and NFC East. Yeah, this will be a random to these two divisions We've got Brian Robinson Jr. and Najee Harris 74 out of 99 so it'll be a random to the NFC East and the AFC North. Haha. He's a Gator fan by love UGA We've got NFC South George D picking up a Chris O'Lave 52 out of 99 true RPA and Grats man. That's awesome And then I believe this is a random as well I think he's still in college if I'm not mistaken Yes, this will be a random to all at the end. We've got a Tyler van Dyke That is 66 or 68 out of 99 Yes, Dave flawless is up next flawless is up right after this Empty let's keep this case going man. This case is beyond fire that hard to The Tyler van Dyke that one's pretty cool honestly We've got that is 66 out of 99 Patrick Mahomes. That's AFC West. There you go again, Tim We've got going to the NFC South. How about a Tyler Algier 42 out of 49? Nice one there running back one in Atlanta right now There you go George We've got five out of 99 Matteo Durant That is going to the AFC North as he was most recently on the Steelers Let me double check this guy again. I'm pretty sure he's Colts, but Just want to make sure Yes, go into the AFC South. That is Chad W. We've got a Devante price and a lot of randoms coming up here Nice one there on the team logo signatures FIU 18 out of 49 We've got NFC South and NFC North. So yeah, this will be between George and Tim Drake London and Amon Ross St. Brown Got going to the NFC East on the three colored patch that is Nacobi Dean Nice one there. He is an Eagle. That is actually really sick 16 out of 25 nice Grats on that one John M next up. We've got 44 out of 99 Oh That's so sick Dude Damian Pierce going to the AFC South Chad W Dude, this is so nasty We've got going back to Chad One of one notable nicknames Traylon Birx Inscribed Trey Bob Dude, that's so cool.   Oh My god, congrats Chad man. That is awesome. Love is being spread around here, too. I love to see it And last box of the break See if we can keep it going guys All right, we've got John Elway. That is going to the AFC West 12 out of 99 AFC East with Victor J 3 out of 99 is sauce Gardner Nice one there on the rookie silhouette signatures We've got going to the NFC East that is Jalen Tolberg On the booklet auto South Alabama also, he is a what's it called? This is a Cowboy couldn't think of it so that goes NFC East I Appreciate that looking crazy man. It's just I love it man. It's so much fun We've got another booklet. Oh my god. That's nasty This is Trey McBride Go into the NFC West that is mon sore.  We've got this is a nasty patch dude. Go into the NFC South that is Matt Corral really awesome patches on this thing George D. Congrats George Six out of ten. We've got NFC South again. That is gonna be Desmond Ritter Six out of 25 Yes, yes, we do ship overseas free shipping out of the door.  I'm looking for responsible. Yeah responsible for any taxes or tariffs that are in import fees Basically, but yes, we ship overseas We've got 11 out of 75 that is going to the AFC East Breeze Hall nice one there Victor congrats Victor And really nice way to end it NFC North back to Tim why not Seven out of 25 Jameson Williams.   This was insane Let me do a couple of randoms really quick. I gotta get these set up We got four to do wow. It's actually a lot Hold on guys. Let me just get all these set up and then we'll do a recap and then we'll go into flawless We got Trojans duel That's gonna be NFC South I Think it was a FC South it was Chad doubly. I don't believe he's in chat though xiris We've got do that one last Alabama dual It is actually a very very good product I have to say Paul very good product that is LSU quad Yeah, that was unbelievable, dude, that was actually unbelievable And we've got the Tyler van Dyke And I random be a random to everybody. This will be all of the divisions All right guys, let me so we're gonna do the random for the Tyler van Dyke auto the LSU quad the Brian Robinson jr. Najee Harris duel and the Drake London Amon Ross St. Brown duel, so let me take you over to the random screen really quick So we're gonna start off with the USC Drake London Amon Ross St. Brown 55 out of 75 gonna roll the dice. This is gonna go for every single So we're gonna go seven times on each random, so we'll start with this one one Two three four five six and seven that is NFC North On the Trojans duel, so that'll go with Amon Ross Seven times on the Alabama dual relic that is Ryan Robinson jr. NFC East Najee Harris AFC North Good luck one Two three four five six and seven that's going NFC East with Brian Robinson That was Brian Robinson, right? Najee Harris and Brian Robinson Yeah, yeah, yeah, we're good. We're good I'm gonna do the LSU quad next that is Derek Stingley jr. AFC South Joe Burrow AFC North Justin Jefferson NFC North and Leonard for net NFC South I'm gonna roll this again seven times George goes he said north so I should go to London one two three four five six and seven NFC North so yes, it does technically go London and Gonna do the random for the Tyler van Dyke auto. He's still in college So it'll be random to everybody in the break seven times on that one two three four five six and seven AFC East on that one All right guys, let's do a quick recap on this one now. This is insane dude Absolutely insane We had the Trojans dual the Alabama dual the LSU quad Desmond Ritter NFC South that's Matt Corral NFC South Nicoby Dean NFC East. Oh You just got confused. I got you. Oh London I got you know, I thought you were uh, I thought you were saying like it should go London is in London, England I don't know why I thought of that, but no we just do when it's a random like that We just do it to everybody in the in the break So it's NFC North for Amonra St. Browns. He's plays on the Lions and NFC South for What's it called for? Drake London because he plays for the Falcons. So that's how we do those generally. Yeah, my bad. I'm no confusion there Aiden Hutchins at NFC North Tyler van Dyke Jameson Williams NFC North sauce Gardner and AFC East Devante price AFC South Mateo Durant AFC North Tyler Algiers NFC South Chris Olava NFC South now You're good George Kaderious Tony NFC East Trayvon Walker AFC South Louis Sine NFC North Ronald Jones AFC West and Brian Robinson NFC East my bad I just completely miss under I misinterpreted what you said. I apologize. That's on me Brie Saul AFC East Tray McBride NFC West Jalen Tolbert going to the NFC East Damian Pierce AFC South and Christian Watson going to the NFC North and our mag guards these were Absolutely insane. Oh my god. We ought to know where to start on this. We had a Justin Herbert 21 out of 75 going to the AFC West There you go Tim with the AFC West We had AFC North that is Kenny Pickett inscribed H2P hail to pit There you go Benjamin L We had one of one Zemir white on the SEC patch going to the AFC West from Georgia He is a raider and Then we had one of one Traylon Berks Inscribed Trey Bob notable nicknames go into the AFC South for Chad That will do it for the break guys. Thanks again. Next up. We got flawless


Felly dwi'n gweithio. Felly dwi'n gweithio ffawr o'r cwrwm. Mae naill yn Ddunkan Ngu. Mae oedd y tawdd yn cyflwyno sydd gyda'r cyflwyno mewn cyflwyno a'r cyflwyno a'r cyflwyno oherwydd ychydig iawn cyflwyno yn ystod o gyflwyno. Mae oedd y cwrwm a'r cyflwyno yn ystod o'r cyflwyno. Felly mae'r cyflwyno ac mae'r cyflwyno a'r cyflwyno yma ar y tawdd. Dwi'n gweithio i fyddi'r cyflwyno. Er mwyn i'n gweithio, I'm going to take you on my personal journey with Cloud Foundry. So I've been involved with Cloud Foundry for a number of years and I help companies install and configure and help companies get started with Cloud Foundry. I'm a consultant for Pivotal. I've worked both here in Amir and I currently work in the US. And throughout these engagements, especially with this getting started experience, I typically find I come across the same set of problems and challenges again and again. So I sat down, I started to write a book for O'Reilly just to capture the technical concerns and the technical considerations around installing and configuring Cloud Foundry. And I've got some early releases here at the end, if anyone's interested. This is kind of half the book, although it's almost finished. Someway into that journey of writing about Cloud Foundry at the 2014 CF Summit, I was introduced to this guy, Onsy. And Onsy took the Cloud Foundry community through some of the changes with the elastic runtime and some of the challenges with the old elastic runtime and how Diego fixed a number of these problems. Now, this presentation is still on YouTube. It's a really, really good presentation. So if anyone hasn't seen it, I'd strongly recommend that you go and have a look. But just to summarise what Onsy talked about, he talked about the tight coupling of the components and some of the orchestration challenges that the Cloud Controller had because the Cloud Controller had too much responsibility and it had too much control over what was going on. He also talked about these triangular dependencies and this poor separation of concerns between some of the components. And this led to a real challenge. I'm just going to switch. Does this work still? Well, cool. So he talked about this poor separation of concerns. And this led to a challenge that Cloud Foundry was hard to reason over and hard to test and also hard to manage the existing features. But more importantly at the time, it was hard to add new features and this was a real problem because back in 2014, Cloud Foundry was just about the app and only supported one type of workload. And it was very platform-specific, meaning it had Linux-specific code hardwired throughout. And so Diego was pitched as this ground-up rewrite of the elastic runtime in Go. In September last year in 2015, the first GA version of Diego was released. This was the first version that supported zero downtime deploys. Now Diego is fantastic for the Cloud Foundry community but as an author trying to write about Cloud Foundry, this was really problematic. It meant I had to go away and refactor 50 pages very quickly and then if that wasn't bad enough, Dimitri came out with Bosch 2.0 earlier this year and it meant I had to refactor another three chapters. So writing about stuff is hard but when the technology is changing as fast as Cloud Foundry, it becomes even more problematic. And as I was delving into the weeds of Cloud Foundry and really understanding the detail, I stepped back and asked myself this question, why should I care? Not about Cloud Foundry. I think Cloud Foundry is fantastic but the detail of Cloud Foundry, the detail of how Diego works, why should I care? And at the back of my mind, I had the developer and thinking as a developer, why should you care about this stuff? Why should you care about Cloud Foundry internals and its abstractions and how it works? Again, borrowing from ONSI, we have this fantastic developer mantra, here is my app, run it on the cloud for me, I don't care how. And by the developer, I'm not just talking about the person who cuts code, I'm talking about the entire team responsible for developing and running that application. They by and large think of Cloud Foundry as this black box. They push their application, Cloud Foundry does its thing and returns an endpoint to a routable endpoint. So they don't really need to peel back the covers and understand what's going on. And for anyone who's been working with Cloud Foundry for some time, this is the highest level of abstraction that we're used to, that developers think about their work, they think about their apps, their tasks, their services. They don't really need to be mindful of what container orchestration you're using or what version of Linux you're running on. All of those are the platform concerns and that undifferentiated heavy lifting is really what the platform is there to take care of for you. So it got me thinking back to this book. The book isn't really for developers. Cloud Foundry is so easy to use as a developer that if you were going to capture those developer concerns, a book like that would be really small. And so in true agile fashion, I decided to actually start small and I wrote such a book, so I've got some of these around as well if anyone's interested. This isn't really targeting the developer, but this is the developer experience of delivering agile software with velocity. So in my naivety, I thought that small book would be about a week, but six months later, I was back to the original book and I was able to frame my question much better. As an operator, why should I care about Cloud Foundry's abstractions? And this is a really important question to ask, not just for an author trying to understand his audience, but for the Cloud Foundry community as a whole, why should you care about Diego and the internals? And to answer this, you need to understand the context of another question, and that's why a distributed system is so hard in the first place. So to level set in case anyone's completely new to this stuff, distributed systems involve a number of components and they're networked or they're connected together. And because there's many moving parts, they become inherently complex because you need to orchestrate them. Halfway through Onze's original talk, he had this comment, it was almost like an off the cuff throwaway comment, but it really resonated with me, that within a distributed system, it's hard to have an accurate picture of the world at all times. And when you look at Cloud Foundry, it's a multi-user, multi-component environment, it has a number of microservices, things can fail, usage paths change, autoscaling can kick in, self-healing can kick in, all of these things can change state and they need to be tracked and potentially responded to. Moreover, there are just inherent challenges with any distributed system. So for example, heterogeneity, the ability to support multiple different environments. Cloud Foundry manages this really well. It has a Bosch CPI for supporting different IS layers like AWS or vSphere or Azure. We have bill packs to support that polyglot programming environment. So you can support Ruby, you can support Java or all the languages that run on the JVM and you can support Node and so on. And it also has this marketplace for the middleware services like Redis and RabbitMQ and MySQL. But all those components need to be extensible. You need to be able to extend the platform. And again, a great example of this is the route service or root service. Instead of the marketplace just being about the middleware components, you now have services for the route to application. And again, new CPIs are coming out all the time with Photon and GCP. And also, you can extend the bill packs and you have other bill packs like the Tommy E bill pack coming out. So extensibility is really key. You also need transparency. Now, a lot of people look at transparency as everything being open and available, but actually in a distributed system, I feel it actually really means the opposite. Instead of seeing all the complexity and all the gory details, what distributed systems need to do is they hide that level of complexity and make it transparent to the end user, and make it clear to the end user. So the end user has a really simple way of interacting with that system. Distributed systems also need stable APIs. They need well-defined APIs that are stable. An example of a good stable API is effectively like a USB. It's been around for ages. A bad API would be like the new iPhone, headphone jack.  So those APIs to be stable as much as possible. This gives each component the ability to be independent and decoupled, and that's really valuable because then you can plug and play those components. You can swap them out, replace them, and the system as a whole carries on working. So just to delve into the details for Cloud Foundry for a second, a great example of stable APIs is the cell. So if you look at a Linux cell, applications run on cells. In Linux, they run on runC backed containers. Containers are managed by Garden, and the cell itself is managed by the Rep, and the Rep talks to Diego. Now there's a decoupled interaction between Garden and the Rep, but there's a strong contract between them. Because Garden isn't participating as part of that wider distributed system, you can make breaking changes to the API, but because it's coupled to the Rep, it will also have to change the Rep. By keeping the Garden API stable, it's got really strong advantages. So for example, you can swap out the back end, it was Garden and now it's runC. If there's a CVE, all you have to do is upgrade Garden, and we make a promise to the community that if you're on Diego version X and above, all you need to do is redeploy Garden, and you get all the later security fixes. You don't have to do any changes to the Rep or any other component, because those APIs are really valuable. Distributed systems also need to support concurrency. When you have many different users, whether they're users of your application, or whether they're different developers, or competing to view and update the same piece of data, the system needs to handle that in a robust way, and things like dynamic routing and distributed locks really help with dealing with the concurrency challenges. So the list of complexity just keeps on growing. There's things like security and self-healing, and monitoring and logging and user management. When you're running a distributed system, especially in a cloud environment, there's many components that go into making that system stable and reliable. So, in essence, they're hard to build and complex in nature, and it's because distributed systems like Diego are complex, there's a lot of value in really understanding them. For example, the cloud foundry operator, there's at least two reasons why they should be concerned about the internals. The first reason you need to care is because you want to debug your app. If the user is accessing an endpoint or routing cloud foundry from an application, and for some reason that route goes away, there could be any number of failures to cause that to happen, and without understanding the system, you don't know where to debug. So, for example, your container could have died or your app may have crashed in the container. The rep on the cell may have died, and it looks like your cell is offline, or the router may have died, causing the dynamic routing table not to get updated, so any one of these failures could cause that effect, and it's important to understand the flow of communication between the components in the system to really understand what's going on. The second reason why you need to care about the internals is because you need to establish resiliency. Components like Consuel and XED and MySQL use a raft protocol, which means they need a quorum, and to have a quorum you need an odd number. If you just have a single node and that node dies, you've lost your service, if you have two nodes and a single node dies, you still don't have more than 50% remaining, and therefore the system as a whole will shut down and may need manual intervention to restart. So ideally you need three nodes, preferably in three different AZs, so if you lose a node or you lose an AZ, your system still has quorum and it still remains operational. Other components like the brain, you have multiple versions for resiliency, but they can only work in isolation so they need to establish things like distributed locks to ensure that they're not trampling over each other. So again, by understanding the system, it helps you build in the right level of resiliency. So how does Diego deal with some of these distributed system challenges? The more I looked at Diego, the more I realized it actually has a very elegant approach and I've loosely termed these as Diego abstractions and I'm going to take you through some of the high level ones. Diego itself is a subsystem to Cloud Foundry. So things like Cloud Foundry's Cloud Controller and Cloud Foundry's Logigator access Diego as a client. Diego is comprised of a number of independent components and each component hosts a set of microservices which is scoped to the boundary of that component. And as I mentioned, the important thing to understand is how work flow through the system. Actually understanding the technical implementation at the BBS and what backs of BBS is less important. The engineers can swap out XED for MySQL for example and by and large the system should operate in the same way. So Diego is a decoupled infrastructure and a number of different components are responsible for the orchestration. And it handles this through a separation of concerns between these components and it uses abstraction layers as work flow throughout the system. And this gives the benefit that work can be agnostic so Diego can now support a richer set of workloads and is also container agnostic both in the image format you deploy to Cloud Foundry and also on the back end as well in terms of how you run your applications. The first abstraction I want to cover is subsystems. And again, this may seem like I'm going off at a tangent but actually subsystems are really important to Cloud Foundry. Diego has been encapsulated in its own Bosch release so its own code base. And this is really valuable because it's become independently deployable as a standalone system. Standalone systems typically offer a pattern of reuse. I don't know anyone outside the Cloud Foundry system using Diego so why is this so important to have an independent code base? It gives a couple of things. When you start to decompose conceptually at least a monolith into subsystems it allows each individual subsystem to grow and evolve as it needs to. This increases the developer velocity but it also has a really rich side benefit of testability. So the release engineer or the platform operator can take a more granular approach to testing because they have a number of different subsystems. So for example, if you deploy the latest version of Diego but you back it with an old version of Postgres if you then migrate that version of Postgres to a new version will Diego still be able to interact with it? So in addition to just having backwards compatible APIs you can also now have a more granular approach to actually migrating disparate components so it gives a richer testing cycle. But in addition for the individual teams it allows for isolated testing so now just the owner of that subsystem can focus on their subsystem. They can have assurance that the teams which they depend on have done their due diligence and they can also not worry too much about the downstream teams it means that their view of the distributed system is way smaller which then automatically allows them to move a lot faster. The second abstraction is workload abstraction. So most people know that Cloud Foundry has moved beyond just supporting applications to this more generic term of long running processes and tasks. So a long running process is Diego's view of your application it's desired to just keep on running. Diego has a view conceptually of a desired LRP a long running process and it's an actual process running in a container known as your actual LRP. If you scale that process you can then have multiple actual LRPs for your LRP. Now LRPs have a life cycle I'm not going to dig too much into Diego's state machine but these life cycles pretty much map to the behaviour you'd expect so if you see your application running it's likely to be in running state, if you see it crashed it's likely to be in the crash phase of the life cycle but the reason why life cycles are important to understand is if you go in and query Diego's database of BBS and you see your applications in unclaimed state it could well be you haven't provisioned enough resource for yourselves so by understanding something about the life cycle state it can help you troubleshoot as well. So on to tasks, tasks are guaranteed to be run at once they're effectively a singleton they should always terminate and they should have a finite running time so tasks could be anything from a one-off script like a batch job or a con job or maybe you're doing stream processing using ELT applications themselves can call out and run one-off tasks so applications can now use this task abstraction but the other important feature of the task abstraction is that Cloud Foundry itself uses tasks as a end user you ask Cloud Foundry to run your application your application goes through the staging process and the staging request from Diego gets encapsulated as an internal task from Diego so in addition to tasks and LRPs applications or actual LRTs also have this app life cycle the app life cycle is responsible for building and running and keeping your app running so there's three distinct phases there's a builder which actually runs your application sorry the builder will drop it and build your application there's a launcher which runs your application there's also a health check that runs alongside the application in the container application life cycles aren't anything new they've been around for some time so in the old world they were packaged as build packs on the runner the DEA the VM that used to be responsible for running the component of the applications and you can start to get a sense of this tight coupling between what actually ran sorry I just got back one between what ran, how it worked and also where it ran so this tight coupling was one of the core challenges originally build packs still exist but they're part of this more generic execution so now they're not coupled with the cell meaning the cell's significantly lighter weight and also because they're decoupled you have this plugability you can have build pack life cycles docker life cycles, window life cycles if you have a different type of workload in the future you can just add that in the mix so the way it works is when you need to use something like a build pack it gets injected into the cell so it's decoupled and it's plugable so looking at containers and I've touched on this before in a previous talk but I still see a lot of confusion out there in terms of what actually can CloudFoundry support in terms of the file system CloudFoundry supports docker images and it also supports the ability for you to push your application and allow CloudFoundry to build it into a dropler plus a stack so the file system is what you actually run the thing responsible for building and running that isolated process, that container management is garden and garden can be backed by Linux or it can be backed currently by Windows but really we talk about containers anything which adheres to the garden API can be plugged in so if you want to run on run V or you want to run on bare metal conceptually all of that can be plugged into that garden API so Diego itself has what we loosely come action abstraction and this is fairly complicated but all the pieces within Diego are totally independent and that means that they can solve their task at hand in isolation they can choose to have unique expressions of the problem as it flows through the Diego system and it can choose the right expression depending on the problem being solved now there's a lot of information there so let's visualise that as you CF push as an end user you're apt to the cloud controller the cloud controller then passes it into this bridge layer called the CC bridge that then goes to Diego's database it goes on to an auction and ultimately that request ends up as a scheduled process running on a container as work flows through the system it starts really coarse grained as this big kind of bolder like analogy and it gets broken down into finally something very granular as a scheduled process now if everything within Diego was concerned with this schedule process it would be incredibly brittle and it would be very complicated for the end user to interact with it so by adopting the ability for each component to have its own view of the work and its own expressions of that work and abstractions it allows for this plug and play model where you can take things out and replace them but it also allows for this really easy transparent client interaction so as this workload gets kicked off as the cloud controller interacts with Diego it's imperative it can say run this and then it can leave Diego to do its thing Diego knows about the desired state and it tries to rectify the desired state with the actual state and run that workload so the cloud controller beyond that point doesn't need to be overly concerned with how Diego does its job so the cloud controller which is Cloud Foundry in Diego have two effectively different views of the world Diego's view is very generic and Cloud Foundry's view is very specific for Cloud Foundry and this bridge component in the middle allows the translation between the two for example when you stage your application Cloud Foundry knows that the app needs to be staged but Diego turns that into a generic set of staging tasks or LRPs and so on and the uploader as well Cloud Foundry knows that you should upload droplets to its blob store and download droplets from its blob store but the upload action in Diego is only concerned about upload a file to this URL it doesn't have to specify the URL as the blob store that's Cloud Foundry's concern to tell Diego about it so the BBS Diego's BBS which is effectively Diego's API uses something called Composable Actions and this is the next layer down from tasks and LRPs so we talked about tasks and LRPs they get translated into Diego's Composable Actions things like download a dropler download action run action and again these are imperative actions they're an instruction set and you get this tree and these instructions end up being the components that actually run on the cell sorry, in the container so how do LRPs and tasks translate to these Composable Actions so let's take the scenario where you're staging an app or you're running a staged app you will have three Composable Actions you have a download action to download the droplet from the CC blob store there's another download action to download the app lifecycle binaries and then there's a run action to run all this stuff onto a container the important thing is this is another level of granularity beyond the LRPs but this is still describing the desired activity not the implementation so as I mentioned things like the implementation that you're downloading from the blob store all of that still comes from the cloud controller so the last abstraction I'm going to talk about is closed feedback loops and again this is key to keeping that distributed system in place so the CC bridge has two core components to offer this closed feedback loop it's the ability to handle domain freshness and this is done by NSync and it's also done by the TPS so the cloud controller tells the CC bridge it doesn't speak to Diego directly it tells the CC bridge about the desired app and then NSync tells Diego to go and make that so so Diego has this view of the world that it knows what the cloud controller wants it knows the desired state and it goes through its auction process to make that a reality so at that point everything's good and Diego is running the required level of apps if an app crashes because Diego knows the desired state it can bring another instance of that app back but just stopping there isn't enough because this is a distributed system it's eventually consistent and cloud controllers view of the world may change maybe it wants something different but there are messages to Diego that one of those messages has been lost so there's this NSync bulker that periodically checks cloud controllers required state its desired state and what's actually running over in Diego and then using cloud controller is the authority the NSync bulker can then give Diego a new view of the world a new desired state there's also this TPS listener and watcher as well and this gives the cloud controller the ability Diego and find out what's going on and it also listens for any crash events and it feed back into the cloud controller as well so this closed feedback loop is essential for getting that domain freshness in place and we see this model again and again so the brain to the rep interaction works in a similar way where the auction passes information over and then the conferger is looking for that state to rectify if anything's missing so again that's a whistle stop tour through the abstractions we've talked about subsystems we've talked about workloads and life cycles container abstractions, composable actions and feedback loops so these are Diego's abstractions for dealing with that challenge of complexity and Diego has a layered approach and the layered approach allows for each component to work in isolation and it also affords this pattern of reuse ultimately that gives you better user transparency and with each subsystem it's the Diego subsystem with respect to cloud foundry or it's the subsystems within Diego you get this iterative development this development velocity and better testability and you also get this plug and play capability as well so that's it, thank you very much