Oh, sorry. I was on mute. Hey, Jimmy. Yep. No worries. Find the setting at all. I don't see a more button or anything of that. So. Yeah.  So like now you should probably see it down there. Oh. Oh, so was it because I was trying to look for it before the time slot? Um, so it's. If you log in using the password, then. Okay. Host option. But if you log in without the password, you can use the host key and claim the host. I see. Okay. Sorry. We use, um, Google meet. I've worked. No, no worries. I'm sorry that it's been confusing. No, um, but, um, That session that you just rescued for me. 47 people. Oh, awesome. Yeah. Yeah. It was bursting with life. So, um, I was pretty thrilled. I think this next one will be smaller, but, um, that was so worth it. So one question we had. Are these being recorded or just streamed? They are being recorded. Um, it's just, we have to. Do a little bit of extra to pull the, the recording from the stream. Um, or yeah, the recording at the streaming service and then convert it back to MP4s and then we'll post them back. Got it. People asked it. And I wasn't sure if they were. It was because they wanted to be recorded or whether they were concerned that it's recording. So I said, assume that it is. I'm glad to hear that that's the case. So yeah. Yeah. How are you doing personally? Like, oh, pretty well. Just, you know, just living this weird, uh, existence, trying to get my kid home schooled and. It's a while. My daughter's 17. So I think we have it easy. You know, she. She does her school stuff or anything. Self-regulates. If your kids are younger, imagine it's much harder. So. Yeah, it's, it's been an adjustment for sure. But I don't know. Yeah. I don't want to be happy when the summit is behind us. I saw the, the whole thing kind of go sideways on the first day. I was kind of, uh, I was thinking with my operators hat on thinking, you know, someone's having a super bad day and I've had many of those recently. So, you know, from systems crashing and stuff.  And. It's been perfectly good since the fix, right? Yeah. Yeah. Yeah. Knock, knock on wood. All right, sir.  I have another session to. Thanks again. Yeah. All right. We'll see you soon. Bye. Hello everyone. Welcome. Thanks for joining. The interesting thing about these sessions is I have no idea how many people are coming. So I'm going to give it a minute or two because I think people are coming off other sessions, which may not end exactly on time. But how is everyone this morning?  I'm shy. So, um, I'll probably repeat this, but, uh, this is a interactive session where, you know, it only works if people speak. This is not a presentation. Um, this is part of the forum part of the summit. Um, and the topic of this one is war stories, which is very specifically, you know, um, we get together and share experience. Um, so I've put the link in the chat to the etherpad for this session. Please, um, throw something up there. You don't need to disclose the, uh, the meat of your story if you have one. Um, but you can put a title or, uh, you know, something in your name and then we'll, um, we'll go through them. I've thrown mine in at the bottom of the list just to make it clear. I'm not intending to make this a presentation. I'm hoping that people will volunteer, uh, to share something that they, they dealt with, whether it open stack or just in this kind of role. Um, we have had people talking about not specifically open stack things, but data center infrastructure network type, uh, catastrophes up to nine participants. Um, the previous session, um, some of you were at, um, I think by about five minutes then we had 45, uh, participants. I don't know how many this will be, but I think giving you a couple more minutes is probably worthwhile. Everyone is, um, neutered and off video. Interesting. This is my first time using zoom, but we use Google Meet. Um, I'm still learning how it works. Okay. There's 11 of us here now. Um, still no war stories on the lightning talk etherpad. Um, it's in the chat here. If you're not looking at the chat, please, um, you know, open that window and click on the link, bring up the etherpad. Um, again, this is, this is not a presentation.  It's meant to be a working session. We talk to each other and share. So, um, no one shares. It's going to be all me and you don't want that. So, um, please do, um, think of something to share. Um, Hey, Terry, I'm just trying to, um, remind everyone this is a working session where we people need to actually share or it's not going to work. It's not, um, not a presentation from, from Chris Morgan here.  So think back. It's, it's entirely up to you. It's very open ended. Um, the purpose is really to kind of, uh, you know, share experience, of course, on open stack or loosely open stack focus, but, um, you know, they have been fun in the past. In January, we had, um, John Garbert volunteered to be the, um, um, master ceremonies. And he actually invented a scoring scheme. And, uh, I wanted private scores. So no one knew to the end. And then, um, we announced a winner and two runners up. And, but the prizes were just, um, swag, you know, like a Bloomberg branded umbrellas and things like that. So I would have to think about one story. I haven't prepared anything, but if I come up with something, okay, so I'm willing to lead if that will get things going. Maybe if not everyone is just not sure how these things work. Um, but it's, it sort of goes against the idea that I'm the moderator, not the presenter. Um, so it's five minutes in. I do respect the time of all people who have turned up. Um, so let's, let's try and get things going. Um, whilst I'm talking, please think of, now it's not mandatory by the way, I can't put you on the spot and say, you must talk now. That's, that would be very hostile, but I am asking people to be involved. You know, so the etherpad is, um, is linked in the chat. I'm going to stop giving me my story. Um, and I'm after that. If I don't have any more war stories to talk about, it'll be a very short meeting. So my war story. So we launched, um, our new open site product, um, last year and, uh, it's all layer three networking. It's all super modern. No layer two networking at all. So it's all a routed layer three network with BGP. Um, and it's also in a new day's center. New hardware vendors for switch gear. Um, and because the hardware is more powerful with more bandwidth, um, people started piling into this new version of open stack very quickly. And some of the people who are, um, very automated, you know, hit us up overnight, you know, like Kubernetes clusters just sprung up and started going, right? Really hard. Um, so it was really kind of disappointing that, uh, in one of our day centers, the entire control plane was starting to collapse and, um, having super, super bad problems. Uh, by the way, so I wanted to do one quick note about, um, um, what we're doing here. This is recorded.  So we'll make that clear because previous open stack operators events have not been recorded and we've been extremely frank. Anyway, back to my story. So we, we now narrowed it down to two problems simultaneously, um, possibly three. Um, so our software stack was incompatible with one of our, uh, switch vendor BGP implementations. Um, so it would fail, um, heartbeat or keep a lives all the routing would get torn down. The networking would disappear for all the VMs. Uh, and, and obviously for control plane, then it's extremely damaging to have, uh, machines go dark. Um, in addition, we found some of the machines were failing to keep accurate time. They were drifting. It turns out to be, um, a platform issue. Uh, Skylake X machines get the wrong TSC frequency or something.  So we put forward a plan of, um, Well, first of all, we, we identified the fixes and unfortunately it needed, um, a new bird, which is the routing demon that exchanges routes, which is an incompatible upgrade. You have to actually remove bird one and install bird two, but that's on machines that are networked. So you have to kind of chair down networking on a machine that you can only get to by the network. We did a new kernel for time, um, accuracy. So, um, you know, it was a tough call, tough mandate from management. And they then said, Oh yeah, and you need to do all the firmware patching on every host. Immediately. You need to do it. And by the way, the firmware patching automation is not here yet. So it's a long story short. Um, we put together a plan where we, um, we did a rolling evac of every single machine, um, did a software upgrade on the empty hypervisors and then had it jump into a K exec. I'm not a developer of the music crazy here. So I might be closer to the details. And the K exacted all of, downloaded all of the vendor binary blobs and, um, patched the, the firmware, updated the host and then rebooted. And how do we do this with this huge fleet of VMs? Well, we, we basically, we had a bit of good fortune.  So what we did was we did a one to one, each in each rack, every compute node, we evacuated to its peer in a new rack, which was at least the same capacity. So the same Ram or bigger. So like if you had compute nodes, ABC in a rack that needed patching, we did it, you know, A went to A, B went to B, and C went to C. And the thing that saved us is we just turned on. A feature that I just mentioned in the previous swap, which is Nova order converge, which is actually a libvert feature. And this will get VMs to move. I don't want to go on about too long, because it's meant to be a lightning talk, but to cut a long story short, if a VM is too busy, it slows the V CPUs down. And then they will merge, move. So we're able to reach rack that needed patching, 100% evacuated with like perfect success because yet another whole rack that was even bigger, most of the time, ripped through with, you know, tearing out bird, replacing it in place, patching the firmware and then rebooting. And then in some cases, that fixed rack would then became the target from another rack that needed fixing. And we, we ripped through about 1500 hypervisors and about 15,000 VMs in, I think about six to 10 maintenance events because we actually, by the way, all of these open-stack clouds remained online the entire time. AZs went dark, but apps are supposed to be distributed. So that's a little bit of war that we did, which it was quite a challenge, but also the feature that I came here to tell you about was the order conversion has just been magic. Of course, being on shared storage is also the thing that enables live migrate to work smoothly. So we're all set back, but it could be some other shared storage for you. So that's my, that's my rack swing mania. We never swung racks before we swung hosts. Put the VMs from here to that host, but we, on one Saturday, we eventually did six racks simultaneously and each rack was done with eight-way slicing. So we did something like 48 simultaneous host to host live EVACs. So we fairly lit the network up, but it's just all worked. It's fantastic. So if anyone has questions about what this platform is, it's canonical open-stack. We're using Calico Layer 3 networking and our project song GitHub. Ask me if you have questions. That was the worst. I don't know how much war you guys perceived there. I'm like, you know, our data center didn't switch off, but I will say that when someone said, well, you have to do it without outage and you have to perhaps patch your kernel and you have to update your firmware. I was like, this is going to take till Christmas. But fortunately, we didn't, we didn't just start, you know, one host at a time and take as long as we need, like we worked on a better plan. Okay. So we have one more war story on the list. We really encourage you guys to share anything that, you know, would kind of share information or maybe entertain. So the person who posted this has not posted their name. So I do ask for the names. It might be, is it Ivan Romenko? Is that right? May have pronounced it badly. Is that you, Ivan? Looks like it. Okay. Are you ready to share your war story with us? Hi. Do you hear me? I can hear you. Cool. We host the public cloud for a couple of years. To be honest, it's almost five years. And during all those time, we've encountered the same error. A young ops does. Every couple of years. So the first time we encountered that it was, it was a problem in the filtering output in a way that. You get a wider result than you might expect. When we, when we first did this, it was about listing listing. Listing snapshots. Oh, sorry. Not snapshots volumes, volumes of a server. We were about to. Introduce a mystery workflow to delete. Instance with all its volumes. If customer wants it. And when we were testing it, testing it in a test project with only one instance with a couple of volumes, you might expect that no matter how, no matter what common line options you provide, you will have one instance with all its volumes. And the, the scoping looks, looks well. So when we start workflow or taking an instance as an input, it was successfully filing all its volumes and deleting it. It looks great. But when we deploy this workflow in our staging environment, it turned out that when you, when you make an error in filter in the filter and. Asks API with admin privileges, you might accidentally get all volumes in that tenant, not only those attached to particular instance. So when you, and when we, when we try to test that workflow, we accidentally deleted all volumes in that. And that project, it was surprisingly bad. And in a couple of years, we faced almost the same issue once again with our new developers. It worked almost the same, but in other, in another part of OpenStack, it was Neutron. You might know that when you delete a project, it gets deleted only on Keystone level. So if you have some resources spread through Cinder, Glance, Nova and other projects, they will remain working for some time. And each operator in a great tooling to, to clean, to clean these things up. And we do that as well. And when we were refactoring these procedures this year, we accidentally provided Neutron with ROM filter parameters to get ports, yeah, ports to, to do, ports that were attached to some instances, instances that should be deleted. And it turned out that Neutron does not respond with error when you provide him with some strange filter string. It just gets you all ports it has.  You will just delete all ports, no matter what, what ports they are connected to instance, not connected, router ports, all sorts of ports. It was nightmare to fix this. Well, so that's it. Okay, very good. So one of the things that we innovated, I think the last couple of these things was after you've kind of shared your story, then it's, you know, nice to share some of the details in the etherpad, especially if, you know, in your case for instance, I think you found that there's, you said that there's still things that, you know, brand new admins keep on making the same mistake and delete volumes. Sharing that kind of stuff is useful.   We don't have any other war stories to, for me to select from lists. So someone needs to, you know, put their hand up and say that they can share something or, you know, we're kind of out of luck here. Terry said you would have to think of something.  The scope can be wider than, you know, my open stack cluster did this, you know, I'm not supposed to put people on the spot, but, you know, we, this is again, it's supposed to be a working session. You know, you have to, you have to participate or it doesn't work. So I can share a fun story. Go ahead. Thank you. So there was like, but when we started to add more automation to the open stack development infrastructure, we were tracking all of the, all of the weird errors that would, they would like to create false negatives on the tests because those would be very costly because they would reset the whole, the whole gate queue. And so there was this error that basically happened every day. You would see jobs failing at whatever, you would have jobs failing every single day. And so there was, that was weird because there wasn't so many occurrences, you know, it's just like a few jobs every day. But at one point, I just like decided to investigate that one. And the, the jobs were failing because they failed to install packages every day at 2am. And the reason, happened to be because there was a cron job on those test servers that would update the app database every single day at 2am. And during that timeframe of like maybe 30 seconds, if you tried to install packages, you would, you would get into a TPKG lock. And so the, the error was basically, we run so many tests. And at every single second that you could still hit that very, very specific case where you, where the cron job would be updating the, the app database. And the lesson I learned on that one was that, like sometimes it's just, you don't think that, you know, a cron job will, will make tests fail every day, but they can.  And in the same way, we, we, every time there is an outage, we're somewhere on the internet, we're affected and we had to put on proxies to make sure that we are isolated from those outages and, and, and like maintenance windows everywhere that, that con job was, was one, one other thing. So it's not really an open stack story. It's not really like an operational nightmare, but it's a, it's a good lesson that at scale, you will, you will see new issues that you weren't really paying, paying any attention before. Like nobody, that con job comes by default on, on, on, with two server. Maybe they removed it since then, but it comes by default and, and yet it's not seen as an operational problem anywhere. And, but for us, it was at the scale at which we were running test. It started becoming one. And the same thing with like, you know, clock, clock adjustments. If you're not, not using, like a time synchronization, then, then you can run into very weird issues when you jump one second. And, and that, that is also something that once you reach a certain scale, you will encounter like daily, daily corner cases all the time. So it's a, yeah, that's all I could think of in the five minutes you gave me. No, sure. So I think that's, that's an eternal thing that we encounter, which is problems you see at scale cannot be anticipated.  I think it's almost like maybe someone could do a, a proof in economics or even in physics, you know. So in some sense, there's always things that prod can do that, that surprise us that we cannot make our test clusters do, you know, like we have one of our test clusters is an entire rack of brand new equipment. We never break rabbit there, but on the, you know, 20, 20 rack production clusters, we certainly can. So this is actually something I don't have the answer to because prod can always break in ways that we can never rule out beforehand. I think the only way that we protect against that is we actually have one open set cluster per data center. We move software to one at a time. So I don't see any more war stories here. I hate to be the person who just keeps blabbering on, but I will, I could share another story that most of you won't have heard. That's okay. But I brought much prefer if someone else had another story to share with us. And I think Terry showed you that the scope can be almost anything that we do in this, you know, movement considered very grandly, right? So this one is for structure. So let's develop some of his lead development teams. I spent many years developing or leading the development team of a part of the Bloomberg terminal software C++ windows, completely different. I'm not going to get into war stories there, but I want to encourage you to think, you know, what have you been through that you could share any, any lesson or really just entertain your, your fellow attendees here. No one's putting anything. Maybe war stories. I can take a short one. Eric from city network. Perfect. Take it away. We had a case earlier, I think it was this year or late last year and during maintenance window when we plan to replace the hardware of all our network nodes in one region. So basically we are free agents and so on running their virtual routers. And you went to 1604. The full reinstallation of the new nodes did not cover like specifics when it comes to kernel version and so on and managed to land on a, on a, on a kernel that had a very peculiar bug when it came to a lot of fragmented packets being sent through the namespaces on the nodes, cause the whole node to freeze. So we ended up in a scenario where we migrated over basically all virtual routers and load balancers from the old nodes to the new ones and then got hit by the problem that one node at a time, they started to freeze and effectively bring down like all three, all free traffic for the whole region temporarily. So that was a, that was an interesting case. When you, What lessons did you learn that you possibly share? For example, he decided to have maybe a distro diversity or maybe not change all your nodes in one operation or share with us how you do better next time if you would. I mean, in this particular case, it boils down to like trying to, to simulate on a package level as well, including kernels and so on in a pre-prod environment and doing the tests there, which is always kind of tricky when you start to talk about like on package level, you should have an identical environment in tests or dev and production depending on how you build your images. So I want to blow your nodes of course, but yeah. I think it's very true. I think the difficulty is that they can still surprise you as cherry points out at scale. You see extremely unlike the events or almost guaranteed to happen. So I think these intersect very well. Sorry.  Carry on. No, it's fine. I mean, and in this particular case, we were kind of unlucky in a sense as well because the problem didn't occur until we basically moved everything over. So it was just a couple of namespaces that I think we conclude at least that it was a couple of routers that caused the whole thing. And we are running all the routers in HMO. So basically what happened was that one node died, the failover occurs to keep alive. They will kick in and just move the VIP to the other node and traffic will continue and it will effectively bring that node as well and so on and so forth. So it was an interesting case where you kind of had a ping towards the different nodes and you lost one by one. So yeah. I feel that as I said, we had a vendor that's great. I'm not going to name names, but cheaper features, everything's great. And then we only found that it didn't actually play nice with our production and open stack product after we had all the VMs already loaded and control plane already running but struggling. And no one had ever seen that in extensive testing before this went live. So I'm just trying to see if we have another story here. I feel like there's so there's 18 of us now. I know that some of you have been through many things that would make good stories here and I know that's not easy to speak up if you're not used to doing these things, but we don't judge here. And in fact, maybe by sharing rookie mistakes that we make, we demystified what's going on here because in the previous session, it's not really the lightning talk, but the previous session I mentioned that another thing that we did on our new open stack cluster is we somehow managed to run the MySQL nodes with the out of the box defaults, which are terrible for months, tiny buffer sizes and, you know, flushing all the stuff to disk all the time. And we're just thrashing our root disks to death on all of those nodes fixed now, but it was all tuned in the old version. And somehow it just didn't come across to the new version of the software. And it just threw it all away. So if I don't get another war story, I'm going to inflict you guys with another one from me, which I'm even reusing stories from previous events. One of the things I wanted to mention, I did link the war stories from the past to in-person events. So you guys can maybe go and drill into those if you're interested. And if you have any more volunteers for another story, I'm told I'm not allowed to put people on the spot. I did kind of mention Thierry because I think, I kind of figured he had something that, okay, I'm going to do another story. So this one is about, I'll fill in, later, this is about recording who owns things in your open stack installation. It's important for things like billing and fair allocation of resources, but it's also very important for things like billing and fair allocation. I'm not sure how many people here have actually decommed whole cloud. We've done that, but we also had this particular instance where we decommed a flavor. So our flavors now are all durable storage. It's all Seth backed, it's all shared storage. You can live migrate these things. We never lose anyone's files. It's good, right? But for a while we had what we called ephemeral. It's not the same as open, you know, storage on a local stripe set. So I'm sure you're all familiar with RAID. A stripe set is a high performance, merging of disks together with no data protection. So if a disk fails, you're dead. So it was in the documentation. It was called ephemeral in the flavor name. But we found that a lot of users just picked it because it sounded cool, or because it was the first item in the dropdown. So they're trying to do their, my first VM. What kind of storage? Oh, ephemeral, that sounds great. I'll just pick the default. So we ended up with a lot of users on the ephemeral flavor type, which had better performance than Seth did at the time. But there was inappropriate usage. They actually couldn't tolerate data loss. So my story is the day that we tried to retire that flavor. So we stopped it. We prevented it from being used for creating new VMs. So it stopped getting worse. And then we just did automated mailers to the users. And how do we find the users? Because we had recorded from, you know, the instance ID to an identifier for the team. We could then look up and send them email to the owners. So we're actually able to get most of those instances rebuilt on durable storage. Everything could. So my boss and his boss said, OK, just delete the rest. They've been warned. It's far too late. They should know better. And I was very reluctant. I, my first job involved loading tapes for backup at the end of the day after doing data entry. I don't like losing anyone's data. So I actually disobeyed. They said, just delete it. Just tell them they're too late to delete the VMs. Because we had ownership information, I tracked down a person for every single VM. I'm not talking like a huge amount. We're talking like 20 or something. Actually it was less than 20 people because quite a few were owned by just a small number of teams. So the very, very last ephemeral backed VM that we got rid of. And I was weeks past the deadline where I was ordered to delete it. I found a guy and I gave him a phone call. Hey, you passed the deadline. I'm going to, I'm under instructions to delete this VM. What is this VM doing? And can I delete it? Oh, so the person says, no, don't do that. It's the CI CD master for, and then some name that is relevant within Bloomer. But basically imagine the most lucrative single application within the Bloomer terminal. The CI CD master controller was a VM single instance on the inappropriate storage that at any point in time, if any disk failed on that particular hypervisor would be dead. So did I delete it? No. I said, please rebuild it as soon as you possibly can on any other flavor and, you know, we'll help you with whatever you need to get off this. So I avoided breaking the most lucrative app at Bloomberg is the, is the headline thing, but the real lesson learned is when you build any new cloud, no matter what your purpose, you need a way to go from any VM and any IP address to the person whose job it is to do something if it's, if it's quote unquote bad. So that's what that's my other war story. It's kind of a, it's kind of a crisis averted story rather than actually having a crisis, but I think you can use your imagination that if, if the cloud team deleted the CICD process for the app that makes most of the money, there would be some tough conversations. So part of this was actually inspired by David Medbury, who did a great talk about retiring the cloud at Charter cable, I think it was, because they switched platform. And again, you know, that thing about being able to, it's already well to, you know, put opensack on some machines and let people make the ends, but can you find the team three years later when maybe no one in the team even built it? Can you find the person whose job it is to fix it? That is actually the, I think where you earn the big bucks, mythically, I'm not saying we earn big bucks, but if we do a good job, that's part of it. It's a bit like the old saying, you know, backups are easy. It's for stores that are where the, you know, the skill is. Okay, so that's two stories for me, which is not how I wanted to run this. I'm going to make a final plea. Does anyone else have something they'd like to share? We've had good stories from Roman, Terry and Eric. Surely someone else can share something. The scope can be extremely wide. You know, you didn't have opensack in the title. I guess, so I can use the time then. So I think that's it for the war stories, but I will say that the session we did earlier this morning was extremely well attended. We had 47 people. And in addition to a great discussion about, you know, blockers about upgrading and scaling problems and testing and tools. I actually asked the people assembled whether the, the ops meetup series that we used to do would be better in a different form given that we're all at home in front of the computer. We're not traveling. We're not getting together. And the suggestion we have is instead of trying to arrange large events focused around open stack operators, which we used to do twice a year kind of midway between the summits. How about we do something quite regularly, but it's only like an hour. So I was thinking of like these talk radio people who just have a regular slot and then you can have a topic and you can have a guest. So I called it ops radio. And you can see the etherpad. I put up two options, you know, traditional events or ops radio. And we got a very, very clear mandate from all the plus ones to try and do ops radio. So that's something we're going to put together so that the thing I want to leave you with is if you're interested in ops related events where we get together and share information about not always war stories, but more often tools, upgrades, long-term maintenance, scaling, those kinds of things from an operator's point of view where you're actually running stuff and you have the real problems of, you know, upgrading clusters that are elderly that are on old versus open stack. So the longest short of it is we are going to give a go at doing an ops radio thing, possibly on the open stack zoom platform. I haven't talked to them yet about it, but definitely in this kind of thing, if you have any zoom or it could be the open in for a Jitsie influence instance, which is called meat pad, I think. Anyway, what I want to, the one thing I want to share with you is if you have any interest now or potentially in future, please, you know, follow this Twitter feed because we get more engagement from this feed than every other method put together, you know, individual emails, mailing list emails, blog postings, added together all of them. Don't get the engagement that this Twitter feed does. So probably only when we launch this, it'll be on the Twitter feed.  And then next week, it might be, oh, an hour about OBS now is not for me. I don't use it. So, you know, we are trying to adapt our thinking to the current circumstances. So that's really all I have for this talk. I see some good discussion finally on the, no offense, but finally on the etherpad. I will fill out a bit more about my story and Roman and Terry and Terry and Eric. If you could also fill it out a bit. You know, we bring these things forward to future events, as you see, and it's very useful to have some examples of these things for the future. The other thing I want to mention is when the pandemic eases and we can get together, of course, the OBS Meetups team is going to attempt to get things back into the more normal thing. But I will say that the OBS Meetups team right now is not me, but the other people and the other volunteers are kind of having personal issues right now. So we are a little bit thin in manpower. So the ops radio thing might be actually more achievable than, you know, the two-day events which have, you know, catering and a venue and all that stuff. That would be a bit hard to do right now, even without the pandemic. So that's all I have. We have five minutes left officially. I want to throw it open for any comments, feedback. I'd love to hear, you know, whatever you have to say, really. Again, I emphasize that these are forum sessions. They're meant to be working sessions. You're meant to participate.   That's not what we're doing here. So I see some great detail here about exactly what was the cron job that surprisingly could trip things up. Erica, I wonder if he's still here. I wonder if you could be persuaded to share a little bit about the, I think it was a router upgrade. You said that eventually, or a Linux sister upgrade where the virtual routers eventually locked up. There'll be kind of a good share. That's all I have. Any final comments?  I don't know if you've ever moderated a room where no one speaks. It makes you kind of, makes your eyeballs swivel. Terry, do you have any final comments?  It's much more difficult to get people to participate in a virtual session where you can't like, you know, wave at them or anything. So you're, you're right to call random people out so that, you know, they, they might be in just to, to participate, but. Well, I'm, I'm done for the day. So I'm, you know, I'm good, but, you know, for future sessions that you go to where it might have a brand new moderator, have mercy and at least speak up and say, you know, something, right? Otherwise I have some of the times I'm even like, wait, is this on a Zoom working? Like, can anyone hear me? So anyway, I think I will thank you all for, for coming. Please do provide feedback in whatever form you want to. I mentioned Twitter. You don't have to use Twitter. My email is on the sheet. You can just email me. I'm still an active participant in the ops meetups. We want to do events that serve the community. We want to mold them to make it work. In particular, in addition to being short, but frequent, we want to do varying time zones. So that's one, the good for North America, then one good for Europe, one good for a pack like that, you know, so that we maximise the utility of it. Love to hear your feedback. So please take that phone away. I'm like a private thanks, thanks for here. Thank you. You're welcome. And I, I think we're done. I will now finish the, the chat, but you know, reach out if you have any, you know, feedback. Thanks all. Goodbye. Thank you, Chris. Thanks.


What's going on everyone? Sam here with late in sports cards, and I am ripping five 2022 panini national silver packs for Mary H. Good luck here, Mary. See what we got in here for you Rated rookie Evan Mobley.  There you go, Mary Got Russell Wilson as a Bronco. So strange looking man. So strange We've got orange tom brady buccaneers 40 out of 199 Congrats there, Mary See some more one-on-ones come out of this man Got cleaton kershaw bass And Is that red jabari smith Auburn 17 out of 99 Nice, uh, jabari now a houston rocket Got kd as a net And orange julio rodriguez rookie 191 out of 199 And we've got nice josh allen bass and purple tom brady 21 out of 25 Congrats, Mary Sweet hit there That'll do it for your packs there Mary congrats on all the hits we'll get those right out to you appreciate it again


The motion is called on to First Amendment 2012 and 1.1.1 in the name of Shona Robison, as a financial resolution for the agriculture and rural community Scotland bill, be agreed, and members should cast their votes now. The vote is closed. The result of the vote on motion <IP_ADDRESS> in the name of Shona Robison is, yes, 89, no, No zero, that with 31 abstentions the motion is therefore agreed. And the final question is that motion 12671, in the name of George Adam, on behalf of the Parliamentary Bureau, on committee meeting times be agreed. Are we all agreed? The motion is therefore agreed and that concludes decision time and will now move on to members business in the name of Audrey Nicholl.


Flowing through the mountains near the Kentucky Virginia State Line, Grassy Creek flows along the base of the Brakes Interstate Park before it flows into the Russell Fork River. The Center Creek Trail, located at the Kentucky Virginia State Line on Route 80, takes you along Center Creek until it runs into Grassy Creek about a quarter mile down the trail. Filled with thick forests and unique rock formations, the trail is a half a mile long, making it a full mile hike from the trailhead to the Russell Fork River and back. While the trail is relatively short, it is by no means an easy trail. Hikers can expect to climb up and down slippery rocks and make creek crossings along the way. Sturdy hiking boots are highly recommended for this trail. The trail begins by following Center Creek through the forest and descending down a narrow draw. You will soon come to some amazing overhangs and rock formations, which is where you'll have to decide where it is best to cross the creek. Be especially careful here, the rocks are wet and extremely slippery. Soon you will reach the confluence of Center and Grassy Creek. From here you will begin following Grassy Creek downstream as it twists and turns its way through a narrow gorge. The remainder of the trail is filled with amazing rock formations, cliffs and small waterfalls. Soon the narrow little gorge along Grassy Creek opens up and puts you right in the middle of the Russell Fork Gorge, giving you a river level view of what is known as the Grand Canyon of the South. It is here where you can see how the Russell Fork River for over half a billion years has carved its way through Pine Mountain creating one of the most spectacular gorges in the eastern United States. For EKB Evening News at 6, I'm Ronnie Hilton.


Hello everybody. I'm John Waddell. Okay. I'm speaking too loud, huh? Welcome everybody. I'm John Waddell, and I'm here with Ryan Brown today. And we're talking about scaling your application on App Engine with Ruby and Doobie. So, we're going to talk about App Engine without any feedback, hopefully. First of all, what is App Engine? It is a cloud computing platform, and that means you can run your web app on Google's infrastructure. And Google spends a lot of time and energy figuring out how to build efficient hardware and data centers and also build applications that are available. And so, App Engine is a way for Google to provide some of that to you to build your application. And we also provide platform as a service.  And Ryan's going to talk about architecture. Right. So, when you have a request come into App Engine, the first thing that happens is it goes to like close school data center, travels through our network to get to the App Engine front-end. We've got load balancing here, so you don't have to worry about that. Your application is running on at least three app servers, so you've got automatic redundancy there. We automatically bring up more instances of your application as you're getting more traffic. Another feature of App Engine, you just pay for what you're using. You pay CPU as you're using. If you have your app sitting there idle, nobody's accessing it, you don't pay at all. Then each app server provides you access to a set of App Engine APIs. We don't allow you to open sockets or anything like that, so we provide you APIs for accessing the outside world. This is an example of one of our higher-profile applications we've had. In the beginning of the year, the president used this for a national on-TV White House event. So, you can see traffic is going along. Right here, right at the deadline where you could ask the president a question online, bikes up like twice the normal traffic. If you're using a normal provider, you would have to provision your servers to handle the 700 TPS the entire time you were at this ramp for three days. You'd have to have the 700 TPS provision the entire time and pay for those servers the whole time. Whereas with App Engine, you just pay for it for those 15 minutes while you were actually running the 700 TPS. So this is what we charge. It's free to get started. You could buy a bandwidth, you could buy a data, six and a half, a few hours for free. And then as you use beyond that, you pay, you set a daily budget if you will never charge you more than that.  But we'll never charge you more than you ask for. And you also want to be careful about building an application that's not transferring, you know, you might have another paragraph somewhere else. If you transfer data in between our data center and your data center, that's, you're going to pay for that. But you might, you know, build an application that has data elsewhere and the client's bringing data into it so you can avoid that. So let's talk about JRuby on App Engine now. Or Ruby on App Engine. So we're using JRuby. As Charles just told us, Ruby on JRuby is fast. It is just Ruby. It is faster than the MRI 1.8 and maybe about as fast as 1.9.  Also you have a wealth of integration options. And by that I mean there's all these Java libraries that you can just basically use from JRuby that are very nice. And also you have all of the supported App Engine APIs for Java. So Java is an officially supported platform. They're engineers Google working on it. All of those APIs are available to you. And there's not necessarily anything else you need in order to take advantage of them.  You can build a nice little rack application. I'll show you how to do that. You can, right now today, very easily build a Sinatra application because you want to load as little code as possible to respond to each request. And Sinatra does that nicely. If you're feeling kind of experimental, you could try using Merb or Rails. Right now, Rails 3 is not really released. So you have to experiment a little bit with pulling something off GitHub and try it out. And I'll show you a demo of that later. And also you have a doobie as an option and we'll show you that later. That's something that's not technically Ruby but is going to be very familiar to you. We also like data macro for persistence. Unlike Active Record where you create a model that is in some cases empty and there's some migration that describes your schema database and then the system synchronizes that. Your schema is actually in the model and there are no migrations. If you decide at some point, I want to add a new attribute to the model. You just store it in there and now all of a sudden you have records that are created with this new attribute. Hopefully your older records that don't have the attribute don't create an issue for you. But you can just put whatever attributes in there you want. Datastore is just going to persist those objects. There's no translation to SQL. There is no SQL. And you also have access to extensions like validations, Active Record finders like find all by first name, etc. You can use those. All the extensions should work fine even though some of the operators that we have are limited because Datastore is a very simple persistence layer. And also it works with the existing DM adapter for Datamapper that Ryan wrote. So you can start using it today. Okay, so it's very easy to install and these are our tools. You just sudo gem install Google App Engine and it brings down the whole development environment that we created and you should do it today and I'm going to have Ryan talk about some of the things that are there.  We have App Engine Rack that's just our tool for integrating with Rack. That's how you configure App Engine. The tool that the Ruby SDK itself lets you run your app locally in the same environment as production. So you can test out all the APIs and everything and App Config is what you use for actually uploading your app. The App Engine SDK is actually the Java SDK, so we install that for you. You don't have to download it separately. And the App Engine JRuby jars, that's our package version of JRuby. So you run this all in MRI, we automatically start up JRuby for you at the right time so you don't have to worry about any of that. And then inside your application, we use Bumbler as you could have mentioned, we use Bumbler to package up all of your gems so that you're running your app locally with the exact environment that you will when you deploy it. And so you need to install into your bundled app the App Engine API and App Engine Rack again. Okay, we can talk about the API as you can. So as I said earlier, there's these Java APIs for App Engine and in some cases there are restrictions in App Engine like you're not allowed to talk to sockets or start threads or write files, things like that. So if you want a timer to time your URL and open a socket, well, there's a lot of things you can't do there. But we give you URL fetch API for that. We also have at the top this user API, you can just integrate with a Google account for authenticating users. You just say it requires login or it requires an administrator. And administrators are the people who are developers on your app. Require login is just anyone who has a Google account. So you don't create a database of users, you just use a Google account for that. The Datastore adapter, of course, is how we communicate with Datastore. We have an memcache adapter that is a lot like the memcache you're familiar with. It's modified a little bit more than App Engine. But you shouldn't really need to worry about how it's different. We have a mail API and I'll talk about XMPP API, Ryan's initial demo of that later. Those are incoming and outgoing APIs. The URL fetch API, as I said, is really designed so that you can just say I want this URL and it comes down to whether it's SSL or get or post. You don't worry about all the gory details of an HTTP like you would normally see. You just use our API and the URL comes down. We also have an images API that is, you know, the low level Java API. And we had a contributor that created a wrapper from Ruby to use that. And I'll show you a demo of that later. The logging API, of course, is used because you can't write files. So you use our logging API and then the logs show up in our logs. And this is a task for scheduling tasks. And I'll have Ryan talk about this demo here. So this was just a quick demo I wrote to, for example, using XMPP. So incoming messages, whether email or XMPP, get sent as a normal HTTP request to your application at a special URL. So you just need to define a handler for that URL and mark your application that you want to receive XMPP.  You don't put this here, you won't ever get charged for XMPP messages. And so then in your handler we provide stuff for parsing the messages. So this is just a simple echo. It takes the message, sends it right back to you. And this is a figure U file that has a RAC application inside it with some RAC middleware. So you generally aren't going to build your whole application inside of your RAC file, but you can certainly do that. And I think that's a great way to get started as you want to know how the API works. You build up a little simple RAC file like this, try it locally, deploy it to the server, see that it works, and then you can start planning how you're going to actually build it into your application. Okay, so we're going to do a demo. We have a little bit of a connectivity issue here, but I'm going to try this anyway. So I've got this very simple hello RAC command, and this is what you would see in the RAC documentation. Okay, so I'm going to see where I'm here. Where's my N? There it is. Okay, so as you saw in that last screenshot, we have a lot of configuration to publish it. So where was I? I was typing a little application here. Okay, so we're going to say the response is okay. We don't need to worry about the headers because App Engine is going to figure out the content links and all that stuff for us. So we're just going to say hello. Hello RubyConf as a matter of fact. Now, so I have this one configure you file. If I run the dev app server and I tell it, please run on this current directory. It's going to see the configure you and say, ah, perfect, that's all I need. It's going to generate a gem file because I didn't have one and generally you're not going to do anything with that one. So it just created an empty gem jar for me. It also installed our version of JRuby that doesn't have Ruby gems in it because that's going to have a penalty associated with it and we don't need it. We also installed JRerac that allows our RAC application to communicate with the infrastructure on App Engine. And it generated the configuration files that are actually just XML files that you never need to look at or touch. In fact, if you modify them, we'll just blow away your changes. You just do all your changes and then configure you. I will show you that in a second. So if I say new here, hello RubyConf. So that is a very simple RAC application and deploying it to production is just telling it to update and I would say that in production. I can show that in a second. Okay, so can you hold that for me for a second? All right, so let's take a look at what we have here. I told it to run. I created a public folder. It probably created robots text and the icon for efficiency. I have it configure you. It created an empty gem file that looks like that.  It also created a web directory and that has a couple XML files in it. If you really want to look at the XML files, you can. We allow you to insert additional things into the XML files by putting middleware in the configure you. But for most part, you should need to worry about that it gets started. So let's go back out here to a little more interesting example. I have a guest book. I'm going to look at this other app here that we have. If you want to run this yourself, you really just want to go to our blog or the code site. I have the browse source code link here or the blog. There's a demo. There's a Sinatra demo. The code I'm looking at is this one file here. It is a little guest book app. Just running through real quickly. I'm going to use DMCore. We have Datamapper. I'm using Sinatra. I'm going to connect the TAP engine. I also have one model here which has an ID and a message in it. That's really all there is to it. There's really no configuration to the database. There's no passwords or ports or hosts. It's just your app has a data store associated with it. You just don't worry about it. If you want to get data in and out of that app, you might build some code to give the data to you. But you really just don't worry about connectivity. We also are saying here that I want to map hosts and get to slash and either list all of the shouts or create a new one. What happens when I run that? Well, I'll do that right now. Again, I'm just going to run dev app server. In a second, it will be ready. We'll reload it. Here it comes. There it is. I wrote nothing. That is the simplest app that is putting data in the desk or pulling back out displaying it. Really not that exciting, but that's really all a web app is in this case. You want to have a specific player. You want to make it simple to create a new book. The other demo that I'm going to show you guys real quick is my Rails demo. As I said, it's a little early to be depending on that, but there's certainly something you can look at. There are a lot of conventions and shortcuts in Rails that people want to take advantage of. At this point, I guess I can talk about some of these concerns we've had and why we need Rails 3. Rails 2 really had tight integration with RubyGems. It was using RubyGems for doing a lot of things and managing your local gems and system gems. We really don't need any of that because it's a bit that we take on spinning up a new app that we really don't want to have. We're just using Bundler to figure out what the gems we need before we deploy.  Rails 3 will actually function that way. If I reload here, I should get this app. I'll just basically bring up this little screen to show you what it's doing. Can you guys see that? Is that too small? I have a modified version of that properties info screen. I see that I'm running a new version of JRuby here because I've got 187. I told RubyGems that it was version 0000 because I thought maybe someone would ask for it and I'd put something there. But I don't even have RubyGems loaded. It's basically stubbed out. We also have Rack. I have a bunch of gems that I brought up at GitHub. Then down below that, I have some environment variables that I thought would be interesting from App Engine. I am in development mode. I've got a version of JRuby, JRack, the APIs, et cetera. It also says that my app is called Rails Primer and it's version 3. Let's take a look at some of those files real quick. My gem file looks like that. These three things at the top here you really don't want to mess with those but I've basically set my sources and I'm pulling something up GitHub. It's a little bit of an experimental situation to be pulling code right in and expect to work but you can certainly do it. We don't hinder that and if you want to work on free release code, your own code especially, you really want to have an repository because it's fun when you get it in some way. The configure you looks like this. I'm saying that it's version 3. I could also make this version 3.0.3 very easily. It doesn't matter if it could be version test. I can make that version of string or an integer. It will be converted to string. I also tell it that I want to exclude any of the files that are the Rails files that I don't need in production. This Rails exclude is just an array of things like don't give me all the directories that I need for testing but I don't actually need in production because every file you put in production should take time. I also tell it that I want to set my environment based on App Engine Rack environment and App Engine Rack is going to figure out whether I'm in production or development mode for me and that's really the only way you want to do that. Then you can do some things in production with testing and such properly. Also, of course, I require this environment and run Rails. Is there anything else interesting in here? That's basically it. As soon as Rails 3.0.3 gems are officially released, I imagine we'll flip a guide showing how to use it and you should be able to load a request with a lot less code than Rails 2.0 without using RubyGems. Then you can build parts of your apps in Rails and maybe other parts of your apps you want to optimize a little bit. We'll have Ryan talk a little bit about ways that you might optimize your app. We'll do that now. Let's talk about current issues. I've touched on these a little bit. These are really the main issues right now. The time to spin up a new instance of your application takes for that Sinatra guestbook just under 10 seconds. We're looking at ways to let JRuby work smarter and let App Engine work smarter with it so that we can reduce that. But basically that is an issue right now. Once your app is running and has traffic, the traffic is going to be responded to in milliseconds, but every so often as we add a new host, there might be a hit that someone takes. Also, some gem extensions aren't ported to Java yet. A lot have been. I can't see that Nick Seeger is taking care of Hprocat at this point. I imagine that's going to help if you want to use Hprocat. But basically you need to make sure that gems have extensions that are in Java as well as C and then they'll work. Also, this is not an officially supported platform by Google, but Google isn't hindering you in any way. Everything you need is part of the Java platform and it's just a matter of you finding a way to work around some of the issues, trying to load as little code as possible for efficiency. And there are some issues, of course, with using data storage a little bit of something new for most people, but hopefully when you get in there and use it, you'll find it's pretty cool. All right. Okay, is this one working? All right. Okay, so before I started working on App Engine, I worked on making Google Web Search faster. So for me, every couple of milliseconds matters. I really don't like the idea that, you know, one out of N requests, however many it is, takes 10 seconds. So I was talking to Woody one day. I was like, wouldn't it be awesome if I could write something that looked like Ruby and it would just spit out this super fast Java code that we could use. And it turns out that's basically what Doobie does. So Doobie looks a lot like Ruby, except that it's statically typed using type inference. It has no runtime other than the Java runtime. And it uses Java's type system instead of Ruby's. Actually, you can configure it to use any type system, but since App Engine has Java, that's what I'm going to talk about. So here's a sample Doobie program. Looks a lot like Ruby. The only difference is right here. You have to declare the types of your function arguments. From that, it infers everything else. It's going to infer the return type. If you had other variables declared within your method, it figures that out for you. So then when you run the Doobie compiler, it's going to create Java for you. This is what it outputs. You don't have to write this, but... You don't have to look at it, actually.  Right. But, I mean, the thing you can see, there's no... It's not starting up JRuby. It's not doing anything complicated. It's just calling system.out.print line and doing recursive function calls. So here's the more complicated app. This is actually an App Engine app that I wrote as a quick sample. It's meant to look a lot like the Sinatra one you saw before. We've got this super simple datastore adapter here. It's kind of based on Datamapper. So you do the same type of thing.  So one of the things you'll notice, I'm not declaring any types anywhere here. Well, except for this. But in the serverlet, these are all inferred from the fact that we say we're subclassing HTTP serverlets. There's also these things here. These are plugins which we use for metaprogramming. So I think this is one of the other really awesome things about Doobie is this idea that you're doing metaprogramming at compile time and also adding metaprogramming to Java. Like, there's all these great APIs. Java has, like, Google Web Toolkit, all kinds of data structures and things. But I hate writing actual Java code. So here's as an example of what a plugin looks like. The Doobie compiler is written in Ruby.  But what's going on here, the Doobie compiler, when you run it and it sees this def-edb thing which we've got here, it reads the file name, it goes, opens that file, and calls erb because it's running in Ruby. And so it just has erb, you convert it all to Ruby code and then evaluates that at Doobie at compile time so that when you run it later on, it's just doing string concatenation. So this is what your template looks like. Again, it's using type inference, getting it all from your class, so you don't have to declare any types in your template. So... And I guess it's important to add that you don't... If you have something that works really great in Ruby but it's not especially efficient, it's not an issue because it's a compile time step that your plugin does and then the output is just a Java class. So the step of parsing through your Hamel or erb or whatever is just happening without users involved. So here's the same app actually running in production on Doobie. Same kind of thing. Another thing Doobie can do is output Java source code. Normally it just outputs class files for you. But I added the ability to generate source code so that you can use Google Web Toolkit. Now, I mean, if you went to the talk on JavaScript yesterday, you might have heard people making fun of it, but he was really making fun of Java, not GWT. GWT is an awesome API. It does so much more than just JavaScript. It does do all these tools for making a super fast website, which, like I said, that's what I want. So I've got a quick sample of a really simple Doobie application in GWT. So hello, RubyConf. This is all written in Doobie and the Doobie compiles to Java. It compiles the Java to JavaScript, and so this is just an HTML file. It's all running client-side in the browser. I'm not sure how much you guys have looked at GWT, but GWT compiles, compresses, in an obvious case, the code for each browser, and every browser that connects gets its copy. And if the browser is unknown, it has some generic one that should hopefully work, but it's just not the kind of thing that any person would do by hand. Right. When I first talked to the GWT people, I was like, why would I use GWT? I'm sure I can write better JavaScript than this thing can, but it is an actual compiler. It's doing optimizations for you. Like you said, it generates a different version for each browser. You can set it up to, like, generate a different version for each language or things like that, so it's really awesome. Yeah. How much have you looked at Groovy, though, really? Okay. So the question is, why not Groovy? Well, if the Groovy guys were trying to make something that uses the Groovy syntax, then I would understand that it has a different... I think they're trying to make something that's palatable for Java people. And... I mean, also, I think one of the big things is metaprogramming. I don't know that there's any way to do metaprogramming in Groovy, and I also don't think that Groovy is really statically typed. I think it's got a big runtime that you still have to wait for, and so it still runs much slower than just Java. So at this point, we're just ready for any questions or any other demos, yes? I don't actually have it on this laptop. Yeah. Yes. You mentioned some other libraries from the Groovy side that haven't yet been ported to Java, and you mentioned Hvercott. What ones are key ones that are necessary? Well, so NoCogiri and Hvercott are kind of an interesting example. I love to use Mechanize to do all kinds of scraping and building cookie jars and things like that, and NoCogiri uses some native XML libraries that will not function on App Engine, and there's some work done to fix that. It's not complete yet, and Hvercott had had some issues working with Mechanize, and so now the current version of Mechanize uses NoCogiri. So there's some work there, but it's not insurmountable. It's not that if you really want to make it work and you know a little bit about how to do the Java, then I imagine it could be done. We just need to find someone. Towards the beginning, you talked about pricing and everything, and I noticed that you had 5,000 emails after all the interviews per day, and 50,000 through admin. Who in the world would have 50,000 in one admin account? Can you explain? Because our company sends a lot of emails to our users, of course. We charge for emails basically to keep people from using it for spam, so that's why there's a much lower limit for actual sending to emails other than yourself. You can send as many emails to yourself as you want. So 50,000, it could have been like $1 million. Maybe it's just keeping you from some bug that's going to cripple you having a limit. Yes? The 10-second startup time? Are there cheats that we can use? Well, it's not like there is a 10-second startup time. You can say hello, CGI in 4 seconds or something, but that's not very useful. In between now and when things are sub-second, can we cheat and say respond just immediately That's the whole point is that you can... Actually, can you hold this? I'm going to show you guys an example, I think, of this. I have a app that illustrates this, and I haven't visited it recently or all day today, but let's bring it up. It's called Fiddlesticks. Hopefully I have connectivity. Did I need to hit return? It's going. That seems like that's not right. Well, okay, I think that's kind of hotel data. Okay, so Fiddlesticks is... This main page you see is on a caching server, always available instantly. So assume that the first time you saw this page was the first time we actually connected. So these first two iframes I populated are servlets. One prints some nice HTML, and the other one sends the JSON, and the other one says hello, rack. And I guess you guys are watching, it took about six seconds for rack to actually respond because I haven't been touching the server, it's been sitting idle. But all I have to do here in my config.ru is say some of my URLs go to this servlet or doobie is actually going to be compiled down to a servlet. You don't really have to worry about all those details, but you can write pieces of your app in doobie and they compile the servlets. So yeah, that's another thing I love about GWT. If you use GWT, it's going to compile you a static HTML page for your main page, and then you load the rest with XML, HTTP or RPCs or whatever. You could build your app in Rails 3 and then find well, there's this one piece of my traffic, I can refactor that with doobie, and then it's always a second. And maybe Ruby's getting faster and faster trying to catch that one second, but you can do the doobie today. I'm going to take this question back here. Yeah, we don't have it on this notebook, but maybe you come talk to us afterwards. I had heard that early on that if the app wasn't spooled up, that it would possibly 500. Is that fixed now? Well, you could take the existing code for Java App Engine and just put Rails 2.3 on there yourself and put RubyGems and just put something up there that takes 40 seconds to spin up, but is going to get cut off at 30 seconds and see it not work. So that is absolutely plausible, but you don't want to do that. Can you talk a little bit more about Datastore because I looked at it a while ago and it looked kind of funky, kind of hard to do compared to MySQL or MongoDB or whatever. Right, well, that's it. It's not really comparable to MySQL. Basically, the way I like to think about it is just it's like an array of hashes. Each entity is a hash. You have a string name for all of your properties and then you can store one of, you know, various different types in there and then you can create indexes on those. You can't do things like sums and counters and stuff like that. You can't do joins because, like, the data is on different physical machines. So it's a lot simpler. There are ways to work around it. There are lots of talks online. If you go to, like, the Google IOS side, there's all kinds of videos about how to use the Datastore efficiently. Personally, I would much rather use the Datastore than MySQL. I don't like SQL. I mean, that's one of the reasons, like, I... I think this is my answer. So I have some, you know, code that I've been working on and there are certain things that just work great. Like, you don't have all of the operators.  All the extensions work. You can do associations, certain associations, by just using the extensions for the associations and they work as you expect. Which you're going to do has many belongs to many where there's a join table for you. You don't want to do that on App Engine. So what you end up doing is you say, I have a property called My, like, whatever, subscribers and it's a list type and you actually can shove an array of entities in there and search for them and say, give me all the records back that have Bob and then you'll get back a list that Bob is in that array and you can't do that with relational database queries but you can on App Engine so you can bolt your join table kind of into one side or the other and right now you have to do the association stuff by hand but, you know, it's really, it's not that big of a deal. Right. And so, I mean, if somebody wanted to contribute this there are ways that you could efficiently implement joins on App Engine and, you know, somebody could add that to the data map or adapter and it would just be there to you. And that is what we need. We need people to come and say, I built this app and I need this one feature and we can certainly help you or maybe you'll build it and then the next person will add on to that. No, let's take this one. On the same line, what kind of times do you have to content distribution networks where you want to push data out to hundreds or thousands of clients using a web app that's hosted on this server? So, I mean, we don't have a specific caching CDN for you. Like we said, if you put up static files on App Engine they are served from our front ends before it will never even hit your app. So, those will always serve fast. They're served like you're... people are routed to the nearest Google data center and then going over our network to wherever it's actually served from. So, it is going to be faster than just hosting it at some, you know, one single location somewhere. Or some other competitive content distribution network there's nothing blocking you from making the calls that you need to make from there to push your data out. Oh, if you want to do that, no. Yeah. How long those same lines if you have a more efficient way of doing it or a cheaper way of doing it I want to hear about it but I also have to have a feature because without that my app doesn't work. So, I... Yeah, I don't understand exactly what it is you need. Embedded multimedia display units. Right. I update content over the network. I could be on a daily basis cooking on an hourly basis except for these screen stickers. I need to be able to have some kind of controls there. Change those directions, change those things and say, okay, on Monday push this out to these thousand displays so that the ads that come up in the hospital can all say it. Yeah, I mean, depending on how you do it, you can definitely do something like that. You can use URL fetch if you need to have the data stored on S3 or something like that. Right. It's also... It's pushing static content is very easy and it's possible that on S3 you might have a lot more work you have to do to build the app out. In our case, you just push your app code and we handle everything else and it looks like we're out of time. So thank you very much everyone.