which I relied on to get here and know which exit to take. That's Rauta, thanks for your answer. Rauta, I can't remember the name, I thought this would go with the thing. This project is, when Pokemon Go launched around that time, she started looking into all the different Pokemon themed repositories on GitHub and made something to show it off, which is really awesome. So he's basically going to tell us how he did it, what he did. Like, certainly. We could be a leader. So, can anyone hear? Testing, testing. There's no mic, right? There's the mic. That's okay for recording, it's not for anything else. So, yeah, you introduced me anyway. Yeah, so I'm Chee-Yan, I do fire stuff, everything Chris said just now, I do a lot of stuff. Yeah, so you might see me, like, before on TV. Yeah, very weird. So you might have seen me having blonde hair before, back in 2009. So yeah, so it's kind of weird. So now I'm like, I still look the same, right?  Some years ago. So anyway, I'm also speaking for JSConf this November. Anyone knows JSConf Asia? We don't care about JavaScript. We need to strain our soul, there's a bit better, I think. Yeah, it's kind of weird, like a coffee hacker that's like weird. Yeah, it's a long catcher. Okay. So anyway, let me just, okay. So, before this, let me show what's the ad. Got this. I'll just start here anyway. Yeah, so I built an app. Just like you say, it's just a list of GitHub repositories with Pokemon names. So it's kind of like, goes back some time ago. I was like, look at this project called, like, Jorg, you know? So anyone knows this project? Okay. Can you slow down? Okay, yeah. So this project is kind of like an inspiration for me. So I look at this, right? This is really nice. This thing, like, you know. Okay. So I was kind of inspired by this project. Jorg is the name of a Pokemon in which generation? I forgot. The first generation, yeah. So it's a project name. So it's like, GitHub repo project name. You've got a nice little Pokemon image there. Nice image that gives this guy a little bit. So I kind of like, you know, like, when you do projects, there's always this naming thing. Before you start a project, there's always, like, what should I name it? And then some people came up with creative names, like, who knows? Who knows? So do you know what it's called? Anyone knows? Heard about it. So it's actually the name of an anime character in Dragon Ball. Do you know what Dragon Ball is? It's a lady name. Yeah. So it's actually that name. This guy actually, this guy, yeah. So it's, like, JG2. So it's a really interesting naming structure. You might be wondering where does it come from? So a lot of, like, this, like, cache. Cache means cross-platform bash. Yeah. And then there's lathe. Lathe means eval in reverse. So just fit the name. So quite creative in some way. And then, and some projects, they might not have, like, cool names. But it has, like, cool images. Like, whoa, this is really cool, right? You want to use it right away, right now. So it's actually, like, a test-runner thing. And then there are some projects, like, not so cool names also, but it has, like, very cool graphics. Yeah. So you'd be wondering who'd draw all these things, right? Like, some developers out there, they can draw all these cool stuff. Yeah. And then this is one of the cool ones. Even have, like, animated use, like, like, serverless framework, like, paper execution, like, they even can make, like, animated use, like, even less cool ones. So it's kind of like inspiration, right? You know, like, you need cool names and cool graphics for a project, right? So... So I was looking at a list of, like, Pokemon names.  So I realized that there's, like, 720 of them. Actually, more. Oh, a lot, okay. So, yeah, 721 plus a few more unknowns. Yeah. So you can get, kind of, like, you can get all the games here, you know, like, if you're, like, not so creative. So... What I did is, like, I go to this website and I realized that I can kind of, like, track the images here, which is not supposed to be... Yeah, I'm not supposed to track the images. So... No one here works for Nintendo or anything like that. Please leave. That's him leaving the room now. So here's the thing, right? Should not publish the... So it turns out there are less... Sort of copyright. Yeah. You cannot say that this page is from my... So I was looking at this page, and I was looking at this AJAX call. So they actually have an AJAX call that has all the Pokemons. So it's, like, Pokemons, like, that has colors. Yeah. So they have everything here. So I'm like, look at this. The world is really cool. So I just, like, script the JSON. Yes. So I use the githubsearch.ai and if you use it before, it's kind of like... You have a rate limit of, like, 30 requests per minute and a 10 requests per minute if you are not authenticated. So what I do is I write a script to, like, like, search for the repository name. So it's kind of, like, just search and then... So I search, like, 720 calls of them. So it's kind of crazy, right? And since I'm, like, doing it and 30 requests per minute, so it takes about around, like, two minutes, two, three minutes, for what? Like, five minutes. So this is the repo. And this is how it looks like. Yeah. Yeah. So there's a power song. Like... Yeah. Like, even this are the stars, like, zero stars. Like Charmander, like, 46 stars. Yeah. So you can see the description of the repository. You don't really know what it does, but yeah, it's just a theme of a repository. You can see it links to, to the heat up repo, like, if it just... It's a gift, and then it just goes to the... the repository. And then you can find out that, oh, it's called Charmander. Yeah. Which is quite nice. So it turns out that there's quite a lot of them. A lot. Like, yeah. And then it goes through, like, first generation, second generation gets, like, less and less and more, right? Until the very last generation, usually the cool ones, like, are taken. The not-so-cool ones are not taken. Well, some are quite cool, but not taken as well. So it gets less and less, until, like, the very last... Oh, this is quite cool, actually. Yeah, it's quite funny, when you get this, yeah. So, uh, this...  Some of them, like, Legendary Pokemon or something. Final buy. So, until the very last round, it was really, like, upa. Upa. So, this is, like, you know, like, kind of, like, very easy. And, uh... So, when I was, like, posting, when I was like, Chinese, right, I was super excited. And I was like, oh, this is so cool, right? And then I tried to post, like, sneak peeks about it.  And all of them, actually, repository is with, like, at least one people's house. Zero's house. Anyway. So, it was, like, seriously, like, so many. And then, uh, a lot more, like, like, Fever, Shawarma, or the H&M. Yeah.  Yeah. So, anyway, so, in the beginning, right, if you look at this page, you go back to this page again. So, I kind of, like, like, cheat a bit, like, so every single image goes to... Yeah, you can see here, right? It goes to Pokemon.com. So, there's 720 images, so all calls to the server. And it's really, actually, it's quite fast. Actually, yeah. Surprisingly, quite fast. Yeah, so I'm like, I don't care anyway, so I'm like, yeah. But somehow, it's still kind of slow, it takes up a lot of memory and a lot of, like, it blocks. So the browser kind of, like, requests, like, eight images at a time. But since the server is so fast, you probably don't notice it. You probably notice it on mobiles, mobile phones, and then you use up, like, 20 megabytes of your, of your bandwidth. So you probably pay, like, if you pay $1 per megabyte, it's like $20 gone. That's my whole new feature. So I'm like, it's not good, right? And I feel bad, like, thinking to their website, so it's like, yeah. But do you write the reference like the pictures come from? Oh, yeah. It's at the bottom of the page, which you can just scroll all the way down. No, it's not good. It goes, like, you don't write that if somebody else discovered it, they would kick you. Yeah. So, yeah, that's it. So I actually remember to write that from the very beginning. So I kind of, like, tried to solve this problem. Actually, on Firefox, again, it always have that missing images for some reason, and it's quite laggy actually, yeah. It's just not missing yet. Yeah, it's just lag. So when you scroll down, it's, like, very slow, like, I don't know why. Yeah, so I wanted to fix that somehow. So I kind of, like, my first solution, okay, is to speed up, like, well, the description is a bit weird. So what I did is a lot of weird stuff. Okay, so I create a sprite, an image of all the images. So I read, so I write a script to download all 730 images, and then create a sprite image out of it. And then the sprite is, like, super huge. Like, is it huge? What size is it? Like 5,000 pixels or something? Yeah. So, and then, after I generate the sprite, I have to generate the CSS. So the CSS is kind of like this, you know, like, background position all the way. All the way down, man. So much. So I don't even know if I'm saying badly. Did you type it out yourself? Or use a script? I'll use a script, obviously. So you can see my script over here. So I pause the JSON and then download every image you can see, this thing, yeah. And then I actually run through this thing, sprite.js or something. So, yeah, let me just, so it's kind of like, through a lot of calculations that like, generate the image, find out the coordinates. So much analytics there. And then I put the classes, .img, plus id, those things. So it's like, quite nice. I use something called sprite-smith. It's like an npm module thing. Yeah. So, language you're using. Language? Yes. Java script. Okay. Just an npm script kind of thing. Yeah. So it's like, sprite-smith is kind of cool, and then, what do you do?  Okay. Sprite-smith. Okay. This is, yeah. Pretty cool. They have quite some cool stuff. They can also generate, yeah. They have like some plug-ins that wrap over sprite-smith to generate CSS, but I kind of like don't like it, so I generate myself. Yeah. Because some of them they are like, generate really ugly CSS for some reason. Yeah. They are like, try to be nice. Okay. So I like, generate my own custom ones. I also like to say, recommend this website. So, I use a lot of, like this compression things, like a lot of like, whatever you know out there, there's a lot of them. Somehow this guy, this, this panda, is doing a pretty good job. So what I did is I create a png file, convert it to jpeg, and then compress it here. And that's it. And the compression is quite good, actually. Like, you go down to like, you know bytes only, like, so from 20 megabytes down to like, actually not too, too few bytes, just two megabytes. So it's like, 10x the savings. And bytes, engineering. Sorry? 10 times engineering. Oh, 10 times. 10 times engineering. Yeah. And somehow this guy can do it really well. So I try a lot of tools, try to try that, I find it and try this, and I'm like, oh, this is insane, like insane savings. I don't know how they do it, but it's just pretty cool. Anyway, but somehow there's a bug. So someone reported me a bug. It's like, someone doesn't work on like, Chrome, on Android, or the default browser in Android. Yeah. So it's like, shows this, but it works on every single other browser. But it works on Chrome desktop. It works on Safari, it works on everywhere, I've said, like any browsers on Android. So I'm like, this is like weird, right? So something's wrong with my calculations, probably, or like the Repina images, whatever, right? So, I'm kind of like, this is so weird. And then, and then I debug it for a while, I realize that it only happens for JPEG. If I switch it to my my PNG original file, it works. So I'm like, probably something weird there, right? So what I did is, like a super quick fix. So I'll write some, like paragraphs of text for myself to read. So I actually convert the image to WebP. Somehow it works, yeah. If I, so as long as it's not JPEG, it's fine. So I kind of like, make use of this bar to learn something new. So I learned how to like convert the JPEG into WebP for our image using like the command line tools and stuff like that. So you can actually like, research it yourself or like, convert to WebP, which is kind of nice. And then, it kind of like, fix it. I even do like, detection. So it supports WebP class, kind of like that. So detection is actually pretty easy. Kind of weird, but they are pretty easy. You can like, find this on like, Google this somewhere. So it can even like, detect like, or if the WebP you need like, alpha transparency or not, or if you need, like, useless compression or lucy compression. So it's like, it can detect any of it. So it's kind of cool. So this one only detects lucy compression. Yeah. And that's the only image to detect if it supports lucy compression WebP. Yeah. So I'm like, wow, this is really cool. So I learned something new from just using WebP. And then, but okay, before I go here, so it kind of like, solved the problem, my first solution, but it's still slow. So it turns out that the image was too large. So it's like, 5,000 pixels. So even though the file size is small, but once it's encoded by the browser, right, it takes up a lot of memory and then like, but it's scrolled down, it's still the browser do a lot of work, basically. So this is not really relevant to Firefox or anything. So I just read this, I'm like, oh, Safari iOS actually got some limitations. So they do something like, oh, the maximum size what they coded with GIF, PNG and GIF is like, three megapixels. So if it's larger than that, it will try to do some magic somehow. And then JPEG got some special special way of like, decoding, yeah. So it's at 32 megapixels. So I kind of like, this kind of makes sense. Probably it doesn't just apply to Safari. If I apply to all browsers, which you don't really know. So I kind of like, perhaps I should do something different. So what I did is, I make everything into smaller sprites. So from that huge 5,000 pixel sprites, I make it into smaller, smaller sprites. So to kind of like, be nice to the browser, so that probably the browser will be smarter in like, managing the memory and stuff. Like if you scroll and see this image, it will probably try to like, re-use it back somehow. And it's kind of like, cool. What do I do actually? What do I do? So a few changes. But since I make it into multiple sprites, it starts to become complicated to do the CSS. So here's the tricky part. So anyone understands what this do? So it's kind of like, selecting all classes with i0 in front or back, whatever that is. And then i100. So if I don't do this right, I'll have to do something like this. 001, 002, 003, you're all the way down. So I'm lazy, so I just select all of it. Select all of it. So there's the S3 step, which is like a magic thing. We would just find for the value of the attribute, if it has i0. The basic class contains that. So the class attribute contains i0. It's probably not that performant in some way, but... CSS performance is never a problem, really. It's a little bit dangerous when you're using the star selector like that, unless you have control over what you're doing. Because you might have i0 to 1 somewhere and i0 to 2. Something is breaks. So you might say if you start with a way it's with or something, if you can. Or is it? Yeah, it's the start. Yeah, it's the start. It's not the start selector. It's not the start selector. No, no, no. But it's... That's a particular thing, yeah. But you can still have string starts and string ends with it. Oh, yeah. Which is the... I tried the start thing. Somehow it doesn't work. I don't know why. I was like too late. Considering you're controlling the generation of the class until it's safe because it's all controlled otherwise you can be careful. Yeah, kind of like that. It's a super easy way. So since all of them share the same properties, so it's kind of... The only thing that you don't share is the background position. Just like, yeah. But some of it actually share. Kind of like, yeah. Some actually share and like I did a workaround for that. Do you know what they are doing? Sprite scene. Yeah, you can see how I chunk the sprites to like maximum 100 pokemons per sprite. So a lot of mathematics again on the half. So much mathematics. Yeah. So and then I generate one at a time somehow. Generate again. Yeah. Until like it's done. So like basically it's like this. I want to see how it works. So... Okay. Yeah. So that's it. So my second solution somehow worked well for now. It's smoother on most browsers now. So no longer lag on Firefox, on Chrome. And somehow even though I use JPEG the bug doesn't happen anymore. I don't know why. Yeah. So it's like probably the image is so large and then yeah. So large that Chrome on Android they freak out and then like start to do weird things. So when I have a smaller JPEG image it's like, oh I'm okay. Yeah. Yeah. So somehow that bugs anyway. So anyway I would like to what is this? So anyway it's one of those I wanted to highlight this like why side projects should be stupid?  I mean like who want to build this?    Yeah. And that's that's the fun part right? And that's that's it. Yeah. That's striking because writing is reasonably common these days but it's not very well understood. The basic concept is as you see where you have one image that contains many and you only show a small part on the screen and you can find that the renders is very different from that and that's why Chiara's seeing problems. When you show a sprite so say you've got four different parts of the one image shown in different locations the browser renders that entire image four times. So that means in memory that image is repeated four separate times. Then the effect is even though you get less of a download you get much higher memory you see you see the browser and that's why you get problems like you saw with the JPEG in in Android like browser and Android Chrome. So that's what's going on. The solution is pretty much what you did what you worked out by accident which is you reduce the size of the sprite images so you have fewer things than sprites. The reason sprites came around in the first place was back in the 1980s trying to do video games where there was a limited amount of storage that you could actually write any code or anything on to having everything you still want to image paint a whole lot more sense and then you could just render that one thing on screen and once it still bumped up the memory like that but at least it was you basically grew sprites into a certain animation or something so if you've got an animation of Mario running along you'd have every lead movement in the same sprite so you'll have a Mario sprite you wouldn't have a Mario and Luigi on each you just have Mario that way you can contain your memory you should not go crazy that way so you've got there in the end but that's why that happens if you do the maths you can work it out so you do width by height by good depth so the color depth which is usually 24 or 32 bit multiplied by but you can basically work out how much memory is taken out so even though it's still going to take up a ton of memory these days because spritings reasonably common that browsers are semi aware of it but when you're in a constrained environment like Android it's going to be a problem you may not even notice that on your page maybe the render's okay but you're probably bumping the person's history out of memory at the same time so it may look alright when you're testing it but you build you're really careful with sprites that's when you get that stuff questions between you and I? you can consider archiving and so your immediate archiving or archiving is it the or why or is it one of the high-end operations it doesn't work when you've got an image of any description because compression already applies some of the sprites may turn out high-end or some things before compression again with an image of any compression before that of a tone if you zip a JDP you'll probably increase the size of it because it's already optimized same thing with a pink it's already compressed as much as it possibly can if you've got a bitmap which is un-compressed you'll see a change but otherwise you won't see a thing so the difference between audio and an MP3 or something when you're in MP3 you reduce I mean it's lossy you're reducing the file size significantly because you're applying compression if you're in MP3 you shouldn't do anything you can't compress so you won't improve the graphics performance by zipping it if your service config correctly you don't jizz images because it's just unnecessary it's already compressed on a JPEG you don't have to really compress the compressed image anymore you have a lot of images you compress it you can there are a lot of that kind of yeah but that's like JPEG works exactly by doing that it basically brings all those similar pixels in together and reduces it that way PMUs is a very similar compression I think both of these same say but they basically do some sort of compression thing on it so if you can use Zip on a Zip it's much better do anything same thing Zip on any other compressing that's already done you can optimize images if you export something from Photoshop there are like that tiny JPEG there are tons of MPM tools and things to reduce the size of images what they do is they get rid of all the metadata so you have a JPEG it has also it has programming created it has the face of the moon and then the general vibe programmer was in when they were doing something so that should be done anyway but you still got the actual image itself which is impressive so the most you'll say that's whatever the tiny magic art you might have to let you see that but then the overhead actually Zip in the first place is going to counter it then you have to see can you speed and this is not worth it how about the images that are off the screen if you put them I was thinking visibility display none display none which acts as a space so you should be able to see people I think what if that might bring out memory and be combined into this display none would get them out of memory but then you're in kind of screen visibility get it to keep in memory just want to show you see that with so well okay so you could do display none and make the container the same size of the image but it would repaint what? what people who have done image galleries and things like that the massive ones have worked out is you paint or you basically render what's on the screen plus the next reasonable amount of scroll minus the last reasonable amount of scroll and what usually happens is they'll use so you'll see what's on your screen and they'll render maybe the next half so when you're going through normal speed you can see the scroll will basically get rid of the ones on the top and the ones down the bottom so you need to tunnel and dust that for their infinite scroll although they retain the memory on the top which is not very efficient other ones get rid of that so they go because you keep jacking up the memory usage which will crush the browser pretty quickly he managed the non-crushed browser which is a miracle but close yeah that's basically how you get around that you just hold the door and really up the other thing I was going to mention on that is the way because you're only rendering a bit off the screen if you scroll quickly then your skeleton screen sort of shows to the square or something like that to show where any of you just go to load so show you something over the white screen so you know to get something back to notice what's going on anything yeah so and quite a bit last time I did a mobile game and you mentioned about the G&G conference that's what I did so what I did was you joined the G&G the first G&G the band the G&G and so on and so that it's very profound that you just from zero bytes to you know how many bytes in this file so you split up the file in one course then I did mobile game so you append the first 0.01 append 0.02 behind the first file and so on and you get a very big file do you want a big image? a big file not an image it's a file it's kind of like a split so it's a five-byte ring file so you're going to create a very big file what do you do what does that do with something? so you get to pass that you get to join up for the files you just mention all that then you can straighten the files very quickly and you get to like a sprite you get to render it very fast so everyone when you want to get an image out put it on put it into the dome then you turn it into a data stream you have to process you have to have a table what must file size to find out you guys have to use the exact table how do you get that same file size from the big very big file and how big you can grab it and grab it by the big file and then get it and you get to that's not something you can do on a web browser it's clickable for native mobile gaming or something like that it's a way of handling assets on any native environment so very often it's going to have on a web browser it's a really common technique with most native applications and then I would try this with my 1000k or something like that you join them up in memory leave it in some description pull out the assets you need and you're alright but that's that's file management and a little bit of management but the browser does all that stuff for you try to implement that and the browser would be correct with 2070 try it once try it any more more questions what level are you on to at the moment, yeah not very 18 18 I should have checked that before then should have checked what level you were on before before you could come up with a lot I've got a great browser here some of the some of them some of them have their bright pink it's a brief brief browser what's the engine on that is that running I mean is that working yes yes it's a great common browser it's a it's a it's created by the browser created by the idea it's not a dream it's definitely a yeah it's a very interesting way to use brief it's a yeah it's a it's a it's a it's a it's a it's a it's a it's a a it's a you and you and you and you and and it's it's It's still, probably when you get a scratch, you know, you run along and you basically need to reduce your stress on my sprays. Yeah. Alright. Okay. Thank you, Chiang. Thank you.


As an Indian badminton player, winning the Thomas Cup and being called by our Honourable Prime Minister when we were in Bangkok and also being called over to his home, I think it couldn't have gotten any better. I think he's called not just us who won medals but also the uber cup team who couldn't get a medal back for the country but I think so he doesn't really differentiate when it comes to medal winners and non-medal winners. I have been a player myself also and personally I also won a little bit but I've never been called up by my Prime Minister to congratulate so of course it's it's something that they really cherish and will forever remember. 100% all the time he's following the player, following the sport and he does what he wants to do, what connects the player directly. I take care of small things so he knew that Almoda's hair is very famous so he asked me to come and I brought it for him so he knew that my grandfather used to play and my father used to play so I think these are small things that I take care of. Our athletes will always be proud to say that we have the backing of our Prime Minister so that is something that is I think unmatchable and I'm very happy with the way that he spoke to all of us and you know he took so much time out of his busy schedule. He had called us back in back when we were in Thailand and he congratulated but then to meet him in person, to meet him in real I think it was just something else and I think this is one day which I will never forget in my life. I say what's up soon ke bahut motivate hotein players aur aage a chakarna chatein. Just as Sir ne bola tha, hamara Indians mein abhi aage aaye that we all can do it so wo jazba abhi sab mein aare hain we I'm glad that we could do it for India. It's been a really amazing day for us. I think the experience with CM Sir has been incredible. I have butterflies inside seeing him it's like shaking hand with him and talking to me spending almost two hours with us it's been like amazing. Most important thing is the Prime Minister talking to us and motivating us and you know that is very very special so this will be a very very special day for me.


Good morning, John. For Christmas from her mom, Catherine received a logbook that once contained records from her grandmother's hair salon. After she had kids, though, she turned it into a recipe book, pasting in recipes from newspapers and magazines from the 40s and 50s. They are interesting, so Catherine and I decided to try a few of them out. The results were mixed. Here is our journey. Recipe one, golden rabbit. You make it with two Campbell's soups. Stir? Well, one can Campbell's cheddar cheese soup, gradually blend in one can Campbell's tomato soup, add one quarter cup milk, heat, stirring, pour over crackers or toast. Good. Bland. Yeah, it's not a lot of flavor. I have injured myself. I got too excited about my golden rabbit. Doesn't it look like a golden rabbit? This is not on the list of things that I want to eat again. The predominant flavor is crackers. Well, I do like crackers. It grows on you. You're still hungry? Just add more crackers. Serve it with half a raw potato. Recipe number two, popovers. In a bowl, dump one cup unsifted flour, one half teaspoon salt, two eggs.  One cup milk. Beat this with an egg beater till it's just mixed. Now fill the cups half full and set the cookie sheet in the 450 degree oven. You will have six splendid popovers, so golden brown and exuberantly puffed up that it's almost embarrassing. I kind of burned them. Yeah. You know, tastes like bread. Northern biscuit. I'm down with this. Yeah, I'd like to try them again. A hole in the middle. A hole in the middle. I don't know why it's got a hole in the middle. Ha! Recipe number three, beer dressing. Mix mayonnaise and mustard together at horseradish and gradually stir in beer. I mean, it's not good. I don't really like mustard that much though. Recipe number four, Mexican chop suey. One half pound ground beef, fried in butter, when possible. One cup sliced fresh tomatoes, one medium sweet pepper, one half cup chopped onion, one half cup chopped celery, one cup milk, one tablespoon flour. Fry the meat slowly over low heat, then remove the meat, leaving the juices to which should be added. The vegetable ingredients simmer until tender. Add tomatoes after which add milk mixed with flour and simmer again until mixture begins to thicken like gravy. Then combine with meat and serve immediately. You seem really unexcited about this one. It's said to serve immediately, so we can't dilly dally. This is probably the best it's ever going to be. I have a suggestion for improving it. Huge amounts of salt. Is it going to eat more? It's so much better. Right now you're wasting salt. I may be over salted mine. Yeah, you really went to town on it. If you were going to name this dish, what would you call it? Obviously it is inspired both by Chinese food and Mexican food. What? No, it's like a Cleveland chowder. Recipe number five, orange cranberry relish mold. Dissolve gelatin and boiling water, add orange juice and chill until consistency of unbeaten egg white. Cut orange in quarters, remove seeds and put through a food chopper. Add ground orange, cranberry sauce, celery and nuts to gel it and mixture and turn into a one quart mold. Chill until firm, unmold to serve. Oh no! It's like...  Do not serve your friends or family this. Why does Jell-O even still exist? This is what you get for your crazy high school or the hospital. Like, where else? 50s food, Catherine. Would you rather live today? Oh yeah. But I think that there may be some wisdom. Sorry. Is it not in there? Helen, thanks. Thank you Helen. That's my grandmother's name. Not Helen Hunt. Thanks to Helen Hunt too, though. Thanks Catherine for being in the video with me. Thank you for watching and John, I'll see you on Tuesday. You got a lot of cleaning up to do, babe. Have fun! I'm gonna go and set it out.


Okay, we're ready. Thank you everybody for coming. We're going to do the first presentation today. We're going to talk about what Word is doing on OpenStack and a little bit of our journey of how we're going to deploy OpenStack in production, some of the challenges that we went through. I would like to talk a little bit of Word Day, but first, let me introduce myself and my colleague. My name is Edgar Magana. I'm a cloud operations architect at Word Day. I'm also a core developer for Neutron. I've been doing OpenStack since 2011 at Santa Clara Summit, so I have some record on this. I want to let my friend, Imtias, introduce himself. Hi, I'm Imtias Choudhury. I'm a senior software engineer at Word Day. Right now I'm involved with Edgar in deploying our private cloud. So, Word Day is a SaaS company over as a service company. We build everything in our cloud. We provide ham and resources app finance applications for all our customers. We have a number of customers like running every day, their payroll, their recruiting, all number of activities and applications that we provide for them in our cloud. And we are taking this journey into providing OpenStack to have more elastic system for deploying all these applications in our premises. So, we actually have a lot of operations challenge. So, we're going to talk about them. We're going to show you our architecture. We're going to talk about specifically the CI pipeline, the environments that we actually have to create for our development team and deployment team. And we're going to talk a little bit about the key techs await and we're going to have time for questions. We are going to also show you the life of the CI system all back in our data centers in Portland if the connectivity is good. So, let's talk about the operational challenge. Any OpenStack deployment requires some customization. In the case of Word 8, it's a little bit even harder because we wanted to deploy multiple clouds because we have multiple data centers because we want to actually provide it to different teams, different areas inside of Word 8. So, it was very important for us to have an automation system to do that deployment and I think everybody will have that. Identity, because all these number of clouds, we want to have the identical configuration across all these clouds. The hardware configuration went the same, the network configuration went the same, so we wanted that the software that it was running on those data centers were exactly the same in terms of configuration. Security, that was kind of like the nightmare working in the OpenStack deployment. So, you may have already security on your endpoints. That's actually we have certificates for all the communication on all the projects, right? Talking to Keystone, talking to NOAA, Neutron, et cetera. So, that was the first step that we have, but our security teams and compliance teams, they also wanted to have SSL enabled on the combination to the RabbitNQ message queue and also on the MySQL communication. On the top of that, they want us to have IP tables configuration for all the bare metal deployments. For the virtual machine, so the overlay network that we're going to talk a little bit about more later, we have started using Neutron that we switched to at the end. Obviously, they wanted to have a stable production for a company, like what was very, very important, the stability. We have year after year over 95% of customer satisfaction. We don't want to go lower on that. We actually wanted to keep the numbers higher before we wanted to even increase them. Production readiness, our operations team, the team who's going to be the first line of defense if something was broken or something was not working, they want us to have good login information. So, we actually enabled syslog and we send it to syslog server to collect all the logs and use some elastic search to identify some potential bugs. Monitoring, they want to have part of the NOx system. So, they wanted to have good dashboard set on alarms to detect when something was not going well. Any person that actually runs on the real world in the data center, they know that we cannot enable three or four networks on the open stack. You see any kind of documentation that will ask you to have like a management network, an API network, a data network, how to band network, that is not possible in the real data center because that will require a bunch of top of the rack switches that you won't be able to maintain and it's going to increase the cost of your data center. So, we have bond interfaces. We have all the communication through one interface and the second one is just for the out-of-the-band and all that it has HA. Obviously, it has to have a multi-tenant event that is a private cloud and we have a very, very good enforcement of policy groups. I was telling you before our security team asked us to have very specific configuration of communication set up between the VMs and we were able to get that through SDN. So, with that just to give you a little bit of the overview of the high level architecture of what we're deploying, as I say, we have two top of the racks switches in each one of the each one of the racks, each one of the servers they have a bond interface that actually have two communications going to each one of these switches that actually will enable HA. We started using Neutron and then we switched to OpenControl because of the security requirements that we wanted on the Overlight Network and then we also decided to split some of the components of OpenControl in two different servers. We wanted to have all the analytics information which is very, very extensive in one individual server and all the configuration and the control in a different box. The reason behind it is because all the analytics is information that we don't want to impact your control plane at all. And having everything in the same server could actually impact it. So that actually improved the performance and the reliability. Then we have a set of compute nodes in the production system that we have. We don't have storage nodes yet as they are. We have our team working very hardly in actually getting the SEF implemented. So I'm expecting to have like four or five weeks where we are going to start adding a bunch of storage servers into this cloud. So with that, I want to pass over to Imtiaz to start explaining what was the journey of the CI-CD system and why we have to use it. Thanks, Edgar. So how it all started Our goal was to deploy OpenStack with community cookbooks and community resources. We wanted to deploy OpenStack and we wanted to make sure we can leverage from community what's already available from the community. At Workday we use SEF as our configuration management tool. There are other options but Workday already is invested so we wanted to use that. The first deployment we started with the community cookbooks. It's an OpenStack Chef project in GitHub. We started with that and we soon realized the project had a very simple implementation. We could bring up an OpenStack cluster with one controller and compute node with two vagrant boxes and we were evaluating at this point so we could do this on our laptop using virtual vagrant and we soon realized that this model has its limitation. For example, it didn't have a separate Chef server. It could provision using vagrant Chef but there are a lot of features, for example, search that doesn't work in the vagrant option so we needed to have a Chef server that replicates what the same Chef server that we have in our production. So we need to realize that we need a separate Chef server and we soon realized once we use a Chef server and let's say we have this simple deployment with a single OpenStack controller and compute, we don't know after it's set up whether the provisioning is right, whether all the functionalities of Keystone, Neutron, Glance, they're working or not and how do we do that? The best way is to use Tempest so we realized Tempest would be nice to have. Then we realized it'll be, we wanted to build a continuous integration pipeline and one way to achieve that is to add Jenkins so we needed another server and then as Edgar already mentioned we instead of using just Neutron with ML2 plugin we wanted to use an SDN solution and that required yet another server and finally like to benchmark performance we need a Drally so there are all these servers that we needed and we are at this point still evaluating on our laptop and building all these things on separate virtual machines it's very computationally intensive and it doesn't quite work and also to complicate things further we also wanted to be able to share some of the components that we don't need to build over and over again for example Chef server, Rally, Tempest, Jenkins we don't need to bring up a fresh virtual box and configure and install it, it takes a lot of time so we wanted a solution that will let us reuse these components that led us to containers like what are the drivers for using containers, first of all it's lightweight instead of creating six, seven virtual machines on our laptop, we could bring up all these things in a single container on a single VM second thing, it's reusable, the components that I showed, Chef server, Jenkins Rally, Tempest people don't have to spend lots of time building those and bringing it up we can just create them once, put it in a Docker image repository and reuse it and it can be shared with the entire community that's what we did, this is shareable and that led us to our Chef development framework this is our first take at it and we are again still evaluating all the community technical sources at this point and for our virtual machine we used a Fedora 20 Vagrant box in production we used CentOS for using Fedora, we need a newer version of kernel for getting many of the Docker features to work and on that virtual machine we bring up Docker engine and on top of Docker we bring up all these containers, so this was our development environment where we proved that the continuous integration model can work, that we can bring up a Chef server, Jenkins and bring up an entire cloud on our laptop and all the developers who were involved in this project initially we could share the same dev environment and we were getting the same results, everyone got the exact same number of Tempest failures so it was a very reproducible environment and it was very useful for us to get to the next phase of our development and to make things even better we added a few other containers DNS we added, we realized that in Docker as you delete a container and recreate it you get a new IP address and sometimes with Chef it gets a little difficult changing your IP address over and over again, so instead of doing that we used a DNS service and addressed all the containers by their hostname instead of IP address, we also did some LDAP to keystone integration and again, doing that thing, it's a little challenging but having an LDAP container that mimics our production LDAP was very easy and we were able to do that in development as well So if you just let me add something in the previous slide, so I really want you to realize the potential of this environment, so you have a system running in our 4GB VM in your laptop that is actually exactly why you're going to run in production and minimal scale at the beginning a lot of the people in the company were accepted about this because they didn't realize the potential of this framework when we started running Tempest tests across all the developers in our organization one thing was very common as Imtia said the number of Tempest tests that we started failing were exactly the same in every single laptop doing the system and you would say like why you're so proud to have Tempest tests failing in your system it's not about if they were failing or not, we didn't focus on that part we were focused on having an automated system repeatable that it was exactly a mirror from each one of us running in different environments, different laptops, etc this is an amazing environment because A, you are mimicking what you're going to be running on your data center in production and B, think about it for a worthy the journey that started with a bunch of developers that they didn't even know what was open stack they didn't even know what was open control they didn't know a lot of things that involved a lot of the learning and that was the perfect sandbox for them to do it, to play with it, to experiment to kill it, to destroy it, rebuild it again and do it in a very safe, quickly manner everywhere, they don't need to have connected to the network they don't need to do that, when the system is open running they can also contain itself isolated and they can do whatever they want that was the perfect mechanism to drive our team to actually learn on open stack as well yeah thank you so our first iteration, now that we've heard the developers they would first start a vagrant VM as I said, bring up Docker then we bring up a Jenkins container Jenkins was configured to bring up Chef server and we bring up open stack controller well, initially it's just a blank container and Chef provisions is as a container sorry, open stack controller, next we bring up a compute an entire open stack and finally we get bring up Tempest, run Tempest and get the results and all the results you can see from Jenkins and the good thing is also that you have entire log, if in case Chef fails you just go to Jenkins, you have all the logs from the previous runs in the next iteration, we started with Neutron with ML2 plugin which worked fine and we wanted to use OpenContrail plugin and to do that, there were some changes that we had to do and we created new virtual machines and one reason why they could not be containers was we were actually testing IP tables for our SDN controller and it's not really feasible to do it in containers Docker has its own IP tables, we didn't want to make it too complicated and that was one of the reason why we had to split it and this is the development environment so now that we talked about development environment the laptop environment and we proved that community cookbooks are good enough, we can tweak it to use it in production we went to the next phase, we built our continuous integration and development using OpenStack and OpenContrail on virtual machines and this is so our continuous integration on virtual machines we took some concepts from TripleO OpenStack on OpenStack so we have our OpenStack running on bare metal created a bunch of virtual machines that pretty much mimics our development setup that I already showed and the goal was we wanted to have a very disposable test environment developers will check in something, you can bring up an entire cloud do the testing, throw it out and people are doing it in parallel so we wanted many of these small cloud instances and we run OpenStack ISOs so our under cloud, our bare metal environment was OpenStack which was also you set up the same manner with Chef and through the automated fashion and then for we used RubyFog library to interact from Jenkins to the OpenStack controller I mean there are other options, it's just worked as pretty heavy on Ruby so we continued using that and the way it works, our development pipeline is developers, they work on Git, they will check in their code when patches, which triggers and create a garyt review gary triggers Jenkins build Jenkins then talks to our OpenStack controller and brings up a number of virtual machines one of which will be a Chef server and we bring up a few other machines which then Chef provisions them as OpenStack controller SDN controller, compute and Tempest and then again running Tempest result and Tempest test and get the results so that's the same idea but this time everything is done on virtual machines this is the workflow and as I was saying from Jenkins when a developer submits a new patch Jenkins tells OpenStack controller to launch a Chef server in this case we actually created a Chef server image which makes like so that we don't have to provision Chef server to begin with and once the Chef server comes up it goes to our Git repository it fetches the patch set that the developer submitted and it uploads it on the Chef server and then it creates other virtual machines OpenStack controller, SDN controller, compute, Tempest all these things can be done in parallel there's a little bit of orchestration required OpenStack controller needs to come up first before the SDN controller and compute comes up even between SDN controller and compute there is a slight orchestration required like one has to come up before the other so those are taken care of on the Jenkins side the script that we use to bring up the entire cloud it takes care of that part of the orchestration and we run Chef Client and then all of these things are provisioned and then finally we run Tempest and get the results so now I'm going to ask you again to look at this picture and remember the previous one so in the past we have a very great environment for developers to test everything they wanted to test in their laptops then because of the requirements on the IP tables and the SDN testing the laptop environment wasn't good enough and actually was slowing us in productivity so we created this first cloud with that first version that we have in OpenStack and then we create N number of tenants each one of these tenants will be one of our developers and then we have N number of VMs that actually they will deploy through the Jenkins servers that we have to actually deploy all these components another Chef server, an OpenStack controller SDN controller, Compute, Tempest, Rally all the tests and actually dispose all that environment over and over so we were moving from that simple environment to a little bit more robust and then on the top of that we were testing our production system even before going to production as is was Public Cloud so that was actually getting a lot of good feedback inside of the company and also it was giving us a lot of the good information that we wanted for the benchmark in running the Rally test so if in your company you are struggling to actually how to convince the management team going to OpenStack this is a great environment Thanks Edgar and one other thing that this model let us do is also to test how the bare metal are cloud performance because once we started using this thing developers were checking in and we made this a parallel Jenkins build so as people were creating new patches we had times where there would be creating like 20 of these jobs at the same time so we see all this VM virtual machine clouds are getting spawned at the same time so we were actually exercising our under cloud a little bit and see how OpenStack performs and so far I mean we've seen some issues and the philosophy we follow is eat your own dog food and this let us resolve some of this minor issues at the beginning and so that brings us to our road to production so first as I mentioned we started with our development setup which was a simple vagrant and Docker container with Docker containers we were building and testing so people developers are still using it but they're gradually moving into more into the virtual machine thing as because as Edgar was saying as we added more and more components it was getting a little difficult to work in that environment then we moved everything to virtual machines and finally we also have a bare metal continuous integration system so what happens is once a code is submitted and it passes our virtual continuous integration test we merge that code and then it gets promoted after doing a full systems test so we run everything with all the cookbooks not only just the community cookbooks we also have work the cookbooks on top and once we test everything the community cookbooks and the work the cookbooks together and we don't find any issues we promote those set of cookbooks we call those our bill of material they get promoted and we know like these are ready to go into production and then they go into like one Chef server and that Chef server is used for the promotion and like for maintenance we use that Chef server to deploy our bare metal open stack and on bare metal we are also running continuous integration it's not triggered by Garrett as opposed to the virtual machine continuous integration it's a scheduled and the difference between virtual machine and bare metal integration is we don't dispose our bare metal installation like we have a working cloud which is up in operation and production and as we create new patches we just like run Chef client again to add the new patches without throwing it down we cannot do it like that will require throwing out everything so that's not possible in our bare metal it's an upgrade mode as opposed to virtual machines where we do from fresh install every time so that's one of the differences but bare metal continuous integration also runs all the time so some of the key takeaways so it took us a lot of iteration to get where we are it wasn't done in one day as I said like we started with the development environment and even there we first started with just neutron with OVS then we added more and more components and then we moved on to virtual machines and eventually to bare metal the Docker and Vagrant it proved to be a very powerful Chef development environment and it helped us doing rapid prototyping without it I don't think we could convince management to adopt OpenStack with this open community cookbooks the containers were also a lifesaver initially when we're prototyping we didn't have any like machines to even try it on so we needed something and the Docker and virtual machine was the way to go and sharing with images was another thing container images made things much easier for us and by building a continuous integration framework we improved our developer's agility quite a bit developers could submit their patches and not everyone now needs to test their code to create a patch it passes Jenkins they know it's good enough then it goes through review if everyone likes their code it gets approved then it's merged and then we have another build that makes sure the system build passes and everything and this gave us very predictable outcome so everything that we see in our virtual CI was very predictable of what outcome will be in our bare metal if it didn't pass we would see the same results on bare metal so we knew exactly what to predict and so far we haven't really run into any issues where things work in virtual machine doesn't work in bare metal there are maybe few minor things that we found but most of the cases virtual machines were a very good indicator of what we would see in bare metal so we're doing very good in time which is great so I wasn't sure about the time so we're going to be able to show a little bit of the demo what we're talking about here so we're going to show exactly what is our overcloud you are so as you can see here in this dashboard I'm actually accessing my overcloud server my open stack controller and as you can see I'm actually signing with my own username and password right this is connected to LDAP so actually I'm just using my corporate credentials to get into this system and obviously this deployment will take over I don't know one hour 45 minutes almost two hours because you have to deploy all the VMs then you have to create a chip server then you have to blow up all the code books and once all the code books are actually uploaded you actually pull all the packages to create another VM which is probably the open stack controller VM actually get all the packages into that VM start running Chef Clients when we start all the packages and you do that for the SDN controller for the UI part and the analytics part of the SDN controller etc so that will take a lot of time so we actually just simplify here a little bit what we want to show you so as you can see this is the overcloud the 2138 we have a bunch of VMs that we already automatically created through a simple CLI command so we have our chip server we have our SDN controller the open stack controller the open controller controller the analytics part of the controller obviously you want to have a less one compute node for testing we can actually add more VMs and add more compute nodes if we wanted we have our Tempest node and we have also the Elk node which is actually where we're sending all the syslog soup if I show here so this is our Kavana dashboard we're actually collecting all the logs here I'm intentionally not putting all the logs in a readable fashion for security concerns but actually you can get a set of alarms and a set of systems here configured to actually know what's going on in your cloud you don't need to SSH to any other compute nodes everything goes here this is what it needs in a production system you don't SSH to the computes to know what's going on or the SDN controller or something you actually could provoke more damage that actually what you want to fix so we actually get everything here we actually have an also monitoring system and coming back which is a Nagio server coming back to this over cloud so what I'm going to do now is I'm going to run the IP address 109668.3 for my OpenStack controller so I already have another tab here so you check out this, this is the 1096683 so this is my dashboard this is my OpenStack controller on my over cloud so I'm running an OpenStack and OpenStack so think about how powerful is this you can actually test any hot patch in your system be sure that you run an OpenStack test that everything is working properly we're not talking about here DevStack and I'm happy with that but this is for reality this is for production systems and in this one you can actually create any crazy patch that you want to explore as a developer as a system integrator and see what is going to happen actually for this one I'm going to join as an admin to show you a little bit more and obviously the network is down what happened with my network what happened with my network yeah, the Wi-Fi is down it went back so in the meantime that I'm actually looking in if you have some questions so you can actually start going to the micro what I'm doing right now I'm actually creating a tunnel all the way to my data center where my over cloud is connected, it's created there you are, I'm connected so I'm going back here so I'm actually some of these network connectivity issues that we are actually going all the way to the data center at work day so here we have this is my over cloud I just have one compute node so it's going to show up here and I've been playing with this server the whole morning so I'm actually I'm actually destroying things creating things so as you can see we have a CI as the end compute node and exactly the same that I was actually assigned on my over cloud here this is the compute node which is actually what I'm seeing here in this over cloud and actually I could have a controller server actually if you want to get familiar with the control policies this is a good way to actually test it so you can do whatever you want in this over cloud you can actually has a lot of potential right now so Imtiaz and Ivy didn't mention this part we're using the Fedora RDO packages for open stack deployment so we don't want to mess right now with the Python code so if we identify an issue we try to backport it directly on with some of the Fedora or the real packages but the data tomorrow has a lot of potential because we want all our developers to do Python changes in order to do that we can build our own packages and we can actually retest in this over cloud and we can validate if we are actually breaking something not just from the Tempen site also from the rally side so we can actually compare the benchmarking values from functional cloud to the next patch to the next patch and so on we are scaling patches every two weeks right now and to be honest the first time that we scaled the first patch through the chip server we were a little bit scary everything went fine we did not find any issues we did not lose connectivity at all in any other VMs so that's actually we scaled like two hours and maintenance windows we did like in ten minutes or less so that was actually very cool with that so we are done so we are going to ask if you have some feedbacks or some questions so we have a couple of minutes for that thank you you managed the life cycle of your artifacts within these pipelines so for example you said you used the RTO packages and if your test take one hour during the pipeline how do you ensure that a new release or on a chef cookbook is not coming in production that was not really tested in development how can you freeze the versions he's the master of that I'll take that in production we are not taking anything from the internet in our data center has no internet connectivity for security and everything the cookbooks that we picked up from upstream they are copied over locally and they only get promoted and go to our production chef server after they go through this phase so first they have to pass our continuous integration test when they go through that end system test once that's passed their versions are bumped so let's say we start with cookbook 1.0 everything passes and then we dump it to 1.01 and then we label that version of cookbook as promoted and then we have a list of artifacts Jenkins keeps all the list of artifacts these are the set of cookbooks that are already tested with our systems test and we know and we can go to Jenkins and say this is the list of tests and I want to use this policy group which is a JSON file with all the cookbook versions listed and it just takes that one, uploads on the chef server and we use that to do our maintenance and the same for all the libraries that we have we have our internal mirrors for all the RPM packages if we want to explore a new version of the library we put first in a quarantine or testing repo we create this overlap pointing to that repo we verify that everything works and we promote that specific library into our production Thanks guys that was just fantastic I think that's really valuable I just wanted to ask how consumable is this outside of the workday environment is this something you can give back because I see it as something very valuable for us as contributors to be able to get this complete workflow from your laptop on the train on the way to work to be able to get patches in and upstream so I just wanted to see how is this consumable outside the workday environment So it is not yet so we recently as a consequence of the Ops Meetups that we've been having so word has been very active on those operations Meetups so we've basically created new repos onto the kit account that is called Ops Tooling Ops Monitoring and there's another one that I don't remember the name so we are talking to the and JJ Asgard to actually push all these codes once we clean a little bit of the URLs you can imagine today all the URLs for the jump configuration is pointed to our internal data center so we want to put it back into the environment file so the policy groups that we are using for this deployment and then it's going to be consumable for everybody but that's going to be the repo where we're going to push these codes so we're going to go with OpenStack contribution upstream from workday so we really want to get back to the community what we're deploying here Thank you guys There was a lady waiting in the micro so you want to take the micro I have a question about if the code is related to the hardware for example it's a driver of some data store so can this code can be tested in the virtual machine on the docker or you have to left the test case to the mental side it can run in both in the virtual machine when you're running all the tempests and when you're running all the rally things obviously rally doesn't really make sense to run it on VMs so we actually create a new cluster that we call the perf cluster we have our compute nodes in that one exactly the same configuration that we're running in this over cloud for the benchmarking part so actually to have realistic data it will work you will get results actually you have a very low numbers and performance I don't know if you want to add something on that part No Did that answer your question or did you miss something I don't know if the machine in the docker can connect outside to to the storage device that is the different kind of storage device that outside of the of the open stick maybe a storage device like you're talking about SAF or something Yes, this is a physical storage device You can mount storage as docker permits and then you can add storage or some volumes to your host first and then mount it on docker and then we expanded our model which I didn't show we tested SAF on docker as well and that also works it's not easy we have to do things at the kernel level some hacks in there to get it to work but it works and you can actually mount volumes but again on your laptop you're limited to do certain things but it is possible We just have time for one last question so could you just take the micro Okay, first SAF we proved that we can bring up SAF with it gets provisioned the same way with a Chef server so in docker container we don't want to bring up an entire Yeah, the second one actually we create three new containers and we actually deploy the Chef client with the role of a SAF server there is a little bit of an extra orchestration needed for SAF you need to have the first SAF server to be running before running the other ones we actually realize that yes, an orchestration level is needed for all these components to run at the same time on the other hand having Chef client automatically running on the back end of the system will fix it sooner or later so let me give you an example if you run at the same time Chef client for an open stack server and the compute node the compute node will finish faster than the open stack controller because there is less configuration and packages to be deployed so what is going to happen, that compute node will look for the open stack controller server and that server will not be ready on time so the compute will fail will not be able to actually subscribe to the Arabian queue on the controller however once the open stack controller is completed the Chef client in the compute node will keep running we actually have a configuration to be running the next 50 minutes it will run again and it will try to subscribe back again and then it will find the controller so if you deploy everything at the same time things will not work but Chef client will actually let you fix things automatically at the back end to be honest we really don't want to do it that way it seems a little bit messy we want to have some kind of orchestration level on the top of that so we are investigating things like M-collective for these Chef environments I think we are on time so I really want to thank you everybody for coming here this is the first session of the day after a very good party yesterday night so that was awesome to see you all here thank you so much again thank you everybody in case you have any questions you can come to us and I'll be happy to answer any questions that you have