OK. Let's get rolling. So today we're going to be talking about continuous integration, specifically rolling up and bringing on board a Zoolci cloud. So a little bit about me. In the US, I volunteer as an OpenStack ambassador. For those of you that aren't familiar, that means I help run the user groups and put together demonstrations and basically get donations into the OpenStack community. So this is one of the projects. I have a consulting company, started about in 99, provide various cloud consulting services as well. OK. So today we're going to talk about Zool. We're going to talk about continuous integration using Zool. A little bit about Zool, for those of you that aren't familiar with it, we'll go through why Zool exists and why it needs CI clouds, why more so than the other types of continuous integration, specifically why we decided to build and donate a continuous integration cloud and why the OpenStack community needs CI clouds, roadblocks that we encountered going through and building this cloud and things that were specific to CI. And then how we use those Zool tools that there are various software components, software projects within Zool and how we use them to improve operations. And then since then, how we've expanded Zool to add in a new functionality. OK. So first off, what is Zool? For those of you that are familiar, it's something from Ghostbusters. So there's the Keymaster and Zool is the gatekeeper. So in OpenStack terminology, Zool is what keeps track of what's coming in to OpenStack and what source code comes in. So Zool itself is a software project that's originated within OpenStack about three or four years ago. Since then, it's grown beyond OpenStack. It's now its own standalone project that's used by a number of other software projects as their CI system. So it's a combination of Zool and whatever type of repo you're using to keep track of your source code work together for the whole CI experience. I'm not going to get into too much detail about Zool specifically. I'm going to give it a high level. There are plenty of discussions the next couple of days about what Zool is. But as you go through, you'll find Zool is actually used out there quite a bit. So Wikipedia is using it. This is their cover page and their Zool status page. So if their changes go through their instance of Zool, and it's public facing, you can go to their website and get all the status of all the gating jobs within their Zool environment. OpenLabs, it's a Huawei project. They actually, they run Zool as well, so they have their own instance, and they're doing various checks related to their projects. And Software Project, the Software Factory. This is a Red Hat initiative. They're using Zool as well. So here are just three, and as well as the OpenStack installation. And then here in the next couple of days, some other private installations of Zool that are being used. So for those of you who are familiar, this is the code review, check and review process within OpenStack. And here there are actually some pointers where you'll see where Zool has run. So any time a change gets checked into OpenStack, Zool will actually see that change and go and run through a number of gating checks. The second arrow on the right there, you can see it says ZoolGate, and then those are the CI checks that Zool went and then it ran it. Down at the bottom, you can see the whole timeline of when Zool went through. So in this case, this was a configuration file change. I went through Zool, went to ran two CI gate checks, and successful completion of both of them. So within, as Zool goes through and runs, it does keep track of log file. And here's, I just circled, you can see some information about a node. A node is a virtual machine instance. It doesn't have to be a virtual machine instance, but with an OpenStack, it is a virtual machine instance. And later on in the presentation, we'll talk about bare metal instances of Zool as well. But a node is a virtual machine that is used to run those CI tests. So people are always asking, well, why are we using virtual machines? Why aren't we reusing virtual machines? Why aren't we just using static hosts and running the jobs in there? The idea behind running a virtual machine is you get a pristine environment. You know the configuration. You know that there hasn't been any changes to it. And so if you want to go through and rerun a CI test, hey, I just need a pristine VM. I'll rerun the CI test. I'll get the same results, or I should get the same results that I ran previously. So in the circle there, you can get some information about what cloud provider the virtual machine came from, the IP addresses, and the operating system disk image. And then also in the log file, you can see some stats of the various checks that were done. So in this case, it was a configuration file change. So a linter was gone through and run through. Really inside of these nodes, DevStack installation is run, and DevStack is run to go through whatever software tests need to be done. And for those of you who haven't seen, this is the typical Zool Status page for OpenStack. It'll go through its listing, all of the checks that are happening, the chain of checks. And you can go through and drill down and see. So if you submit a job and it fails, you can go through a drill down and see specifically why it failed. And actually end up at the log file and see, okay, why did my job fail? For those of you on Jenkins environments, maybe use Jenkins. You're sort of familiar with sequential continuous integration. So in this typical environment, this is a change gets committed. The CI job runs. Second change gets committed. It gets queued up until the first job finishes. So let's say it takes 10 virtual machines to go through and run all your gate checks that you want to run. And it takes two hours. So in this case, you've got to wait two hours until the next check can run two hours. But you're using a fixed amount of resources using 10 virtual machines at a time. If you have an eight-hour, 10-hour day, well, great. That means you can do five, six check-ins if they're all take two hours. So this is great if you have a small community, but OpenStack is a continually changing project. You have a lot of people submitting changes in. You don't want to do sequential continuous integration because you're never going to be able to get all the changes done. You want to get feedback back to the developers as soon as possible as to whether the job succeeded, whether the job failed, where it did fail. You don't want to have developers waiting around. So this model doesn't quite work in the OpenStack world. So, okay, well, how about the next model? We run things in parallel.  We go through and we validate the changes. Well, this works great. Developer checks in some code. We immediately spin up 10 virtual machines. The checks get run. We send them a result back. In the middle of that, a second developer checks in some code. So we go through the 10 additional virtual machines get spun up. His code changes, which I'm calling change B. They got run. Same thing. We're going to have concurrently running change C. And that's another 10 virtual machines. So in this case, we have 30 virtual machines running. We're immediately getting resources back to the developer. We're immediately getting results back to the developers as to what passed, what failed. The results, we're using three times the amount of resources. We've got from running 10 virtual machines concurrently to 30 virtual machines. But at the end of this, do we really know that the combination of change A, change B, and change C all put together work? And we've run the CI changes across all the A, B, and C. Unfortunately, no. Because what we've done is we've run code change A independently from code change B from code change C. So we don't have a run through of all three of those. So while this model does give the feedback to the developers quickly, it doesn't give us that final reassurance that all three of those code changes. But we're getting there. So how about if we do speculative continuous integration, where we go through a change gets committed, so change A. We immediately spin up 10 virtual machines and we start verifying change A. Then we have change B gets committed. But instead of validating just change B by itself, we merge code changes A and B together and we run the CI tests for A and B. And at the same time, change C comes along. So then we merge the codes for A, B, and C. And we run checks for that. Change D comes along. We run the code merges of A, B, C, and D. So now what we've done is we've used quite a bit more resources. Now we're up to 40 virtual machines running concurrently. And we're checking all four of them. And we're working on the assumption that everything is going to pass. Because before we've actually gone through and done this testing, we've gone through and done a one-off parallel testing of the individual code by itself. So A individually, B individually, C and D, each of them individually. And now we're combining them, code merged all together. So the end result is when C of the merge code A, B, C, and D pass, that we know that all that code works. The downfall, however, is if any one of these fails, then we have to throw away all the work that we're doing further down the line. So for example, if the code merge of A and B, if that test fails for some reason, then we have to kill the VMs and stop the CI work that we're doing on the merge of A, B, and C, and A, B, C, and D. So this is great. I mean, this gets feedback back to the developers. This gets code merged back in. We're running in parallel. But we're using quite a bit more virtual machines and more computing resources. And so that's where the CI clouds that Zulu uses becomes important. We need a big pool of computing resources to go through and be available to process all of these jobs that are coming through. So in the open-stack world, we depend on the donations of cloud resources from public cloud providers. If you're a public cloud provider and you want to donate some computing resources, great. You provide the credentials to the infrastructure team that runs Zulu, and they will start submitting jobs to your cloud through Zulu. And specifically through a node pool, which is the process, a software process within the Zulu project that handles communications and spending up on VMs. As my, I run a couple of user groups within the United States and PAC-IPS, it's a data center provider. They had been providing me computing resources to start up open-stack clouds to run clouds at these meetup sessions. So people could actually go through and use open-stack and try it all out. So they had an interest in donating more resources into the open-stack community. And we thought, hey, running a CI cloud would be a great way to do it. Now, in order to do this, you contact the open-stack infrastructure team. And they have a web page that goes through and details out all the requirements to donate computing resources. So at a minimum, each virtual machine needs to have, I just listed out here, eight gigs of RAM, eight virtual CPUs, a public IP address, and 80 gigabytes of storage. And you need to be able to commit to at least 100 virtual machines concurrently running. So if you're a private cloud provider or a public cloud provider and you want to donate, that's the minimum requirements. So PAC-IPS was kind enough to donate the computer resources to make that a reality. So they donated, we'll get to that in a second. But here's a status page that just shows all of the open-stack public cloud providers. So you see a lot of big names there, CityCloud, Vexhost, Rackspace. They're all providing multiple instances across all of their different regions that they're running. You can see right now, I'm going to say there's 16 independent cloud providers that provide all of the computing resources. The way that Zool and NodePool, NodePool doesn't have its own logo, so I'm using the infrastructure open stack infrastructure logo, the little ant there to symbolize. So NodePool is the software project that's responsible for going out and talking to all the public cloud providers. It has the cloud YAML file, so it knows all the credentials. And it talks to all of the clouds. It keeps track of how many virtual machines are running on each, how many it's allowed to, and it always makes sure that there's a pool of virtual machines spun up and ready to go, because you have a little bit of a spin-up time required, so that when Zool needs an instance, Zool can, they're ready and available for them. So NodePool keeps them up and running, puts them in an Apache Zookeeper instance that Zool then goes through and gets the IP address, connects into, and then injects the CI job. So in this instance here, I'm just showing that there are three public cloud providers in real life. There's over 16 now. This is the Grafana page. It shows status across all of the cloud providers. On the left there, we're going to see building. So 65 virtual machines across all 16 cloud providers are currently in a building state. That means NodePool has reached out to the cloud and requested virtual machines to be built in there, currently being built depending on the cloud. That's a two, three-minute operation. Once the virtual machine is built and up and running, it is put into a ready state, which means it is waiting for a CI job to be allocated to it. So it's spun up. It's ready to go. It's just waiting for Zool to SSH into it and submit the job. When a job is submitted, it gets handed over and it's put into the in-use pool. So oftentimes you'll see a ready count up high and then a number of CI jobs are submitted. The ready count will drop. The in-use account will go up. So at this instance time, there are 812 virtual machines being used across all of the clouds. And then once those machines are at the conclusion of the CI job, which typically is less than two hours, the machine is deleted and returned back into the public... returned back to the public cloud provider or private cloud provider for just general use. And the process repeats itself. Node pool sees that the quota has gone down, that it has resources, and it'll go through and start building up brand new virtual machines again. So, like I said, Packet had wanted to provide more than just what they were providing to the user groups. So they decided, hey, let's donate some bare metal. Unfortunately, they are a bare metal provider. They're not an open-stack provider. So they said, hey, John, can you get involved and can you provide the support for running the cloud on top of that? And so that's where my role as one of the open-stack ambassadors got involved. So we spun up about, I want to say 12, yeah, 12 compute hosts as well as some other support ones. We had a couple that are running some Intel Optane systems. Each of them, 24 physical cores, 256 gigs of memory, 2.8 terabytes of SSD storage. Brought them up. Now, the interesting thing is, I figure we can run about 360 virtual machines. Our limit, however, is IP addresses, IPv4 addresses, because Zool does need to connect in and Node Pool does need to connect in to these virtual machines, and they each do need a floating IP address across the public internet. So we're planning on moving to IPv4 so we can up that limit and move away from limitations of IPv4. So the cloud that we built, we ran all the typical open-stack services, so you're going to see Nova and Neutron. However, we didn't run any persistent storage. The idea behind that is this is a cloud that's dedicated for continuous integration. We don't need to go through, spin up any of the VMs, do snapshots of them. We don't need to store anything long-term. The VMs are going to get spun up. They're going to run for the couple of hours to run the test. And at the end, they're going to get deleted, thrown away. We don't need to go through and have any block storage. It was all just ephemeral storage. We also don't need to install things like heats. It's all VM-based, so we don't need any Kubernetes support, anything like that. This allowed us to use the computing resources to run as many CI instances as possible, as opposed to spending them on controller services. Interesting enough, Zool itself uses Zool to manage its configuration and CI. So I'm just going to walk through the updating of the Zool configuration files. So in here, we see there's a YAML file that goes through and lists out all of the data that's required. And we'll go into the configuration of this in a little bit. But basically, the first check-in was just adding in the new cloud, so the glance images could be brought online. Then they increased the max servers from zero to ten. So go through, make the change, Zool runs a linter on this, a CI test to validate that, yes, the configuration file is valid, that there's no syntax errors on it. It's approved by the core team. And then the change goes in place. The node pool sees the change to the configuration file and then it sees that the max servers is now up to ten, and it goes through and it starts spinning up VMs on the cloud. Then they increase it up to 95 and then eventually up to 100 machines. So then you can see that the packet US West cloud is online and it's listed on the dashboard. But that's day one, right? So from day one, it takes us into day two and continues operations after that and what sort of things do we run into? So it's been an interesting couple of months that we've been running this. One of the first failures we ran out is just mysteriously virtual machines wouldn't spin up. It turns out that the infrastructure team switched formats from compressed disk images to raw images. Every night they run disk image builder to rebuild all the latest images and then those images are pushed out to the mirror servers. There's one mirror server located on each CI cloud. So on packet we have a mirror server and the disk images are copied over there and then they're pulled into glance. So they decided to switch raw images. The way that I had the cloud configured didn't have enough disk space within that partition to hold those raw images. So interesting thing I think what it comes down to is make sure to keep in touch and communicate with your users so you know what sort of changes. Changes they're up to so you can anticipate these types of failures. The other next issue found is something that CERN found as well where we are spinning up and spinning down virtual machines faster than it's expected. So I have a stress test that I run through. I run a node pool instance where each virtual machine is only living for two or three minutes just basically to stress out the control plane and the quota system isn't able to keep up to date at least the default settings. So what ends up happening is the quota gets exceeded and things grind to a halt so you end up having to increase the quota even though you know that that number of virtual machines isn't actually running. So it turns out there is a setting in OpenStack where you can have quota go through and update its database more often and like I said CERN ran into this issue as well. Just when you're running a CI cloud the number of instances that is running and the number of instances that are dying and coming back up is much higher than you're going to see in a traditional cloud. Like I said each one of these instances typically lives for two hours and a typical normal cloud environment you're going to see virtual machines live for weeks, months, maybe forever. And a couple of other issues that we saw mainly having to do with configuration. Things weren't set correctly on individual machines which sort of took me down to a path where I need to make sure that every single physical machine that gets added into this cloud is identical and can't go through and make changes by hand. You have to have an automated procedure to go through. So what I started doing is adding and removing machines using Terraform. So for those of you who are familiar with Terraform, you can write out a configuration file of what you want your environment to look like. Terraform's got a bunch of providers that allow it to speak the different technologies so AWS OpenStack. They have a provider that talks the packet APIs. So I had a Terraform configuration file that goes through and will allow me to grow the underlying cloud by adding additional hardware and also I can shrink the cloud so I can remove hardware. So if I have a new configuration, for example, we had an MTU issue where the MTU configurations, the transmit units on the IP networking weren't set correctly and they need to be changed on each of the network devices. So rather than going through and changing the network configurations by hand, which is kind of dangerous, you change the network configuration, you change it incorrectly, you're going to disconnect yourself from a box and the box is halfway across the country. You're not going to be able to get back in it again. So what I do is I write up what the change is, put it into the configuration file. I add a new machine into the cloud. So I run Terraform, Terraform sees that I want to add a new machine or two on. It adds the machine on. It goes through, runs my script to set up the underlying bare metal machine. Then what I do is I add the machine into OpenStack, into the OpenStack cloud so I have the additional physical machine added on. And then I do the reverse. So then I go through and I take the physical machine and I put it in maintenance mode, basically tell OpenStack not to add additional VMs onto that machine. And since this is a CI cloud and each virtual machine lives for about two hours, I slowly see the number of virtual machines on that physical node reduced down to zero. Once it gets to zero, then I run Terraform again and I take that machine out of... The physical machine gets deallocated from my packet account and it's reallocated to packet who then goes through and they reimagine the machine and make it available to another customer. So it's sort of like treating physical machines like virtual machines in that rather than fixing the virtual machine, you go through, you spin up a replacement one and then you go through and you get rid of the old ones. So I did this for new RAID configurations, changes to network configurations, changes to disk partitions, or even if I just needed to grow and shrink the cloud environment. And one of the things I'm looking at doing is automating this so that if I see that there is a heavy load on the cloud and there are physical machines available, depend on different types, that the cloud will be intelligent enough to go through, update the Terraform configs, go through, add the physical machines on, grow the cloud itself. The workloads, you should start to see a very distinct pattern where Sunday evening, US time workloads peak and then basically Friday night they drop down back to zero and the cloud sits idle on Saturday and Sunday. So maybe it makes sense to deallocate hardware Friday night and then Sunday afternoon add the hardware back into the cloud to bring it back up to full capacity. So KPI, so key performance indicators. Node pool provides a great graph here. For me, from an operations perspective, it's hard for me to know when things are running successfully. The last thing I want to do is have someone from the OpenStack infrastructure team that's running Zool contact me and say, hey, John, you know, the cloud isn't working. Some things, you know, there's a problem. From here I can go through and I can get some key values out. And I found these to be very, very valuable. Time to ready. So time to ready is how long node pool is recording it takes for the cloud to go and spin up a virtual machine. So you want to keep this number as low as possible. So when you start seeing it get to 7, 12, 11 minutes, obviously you know something's wrong. You want to keep it down below three minutes for the time it takes a virtual machine to start up. So that's a red flag. Okay, something's wrong. The next one over, error node launch attempts. That's node pool is connecting in, trying to launch a virtual machine. For some reason the virtual machine is failing. It's keeping track of the number of failures that happen. So obviously you can see something happened here on October 2nd, 3rd, where the time to ready increased and the error node launch count increased as well. So some type of error. So then that kicks off. Okay, let's go dig in and find out specifically what's happening. The other metrics I like to keep track of is how long each API operation takes. So this is create server. This is each individual specific API call. So that will tell you if your controllers are responding quickly to API requests. Okay. So how do you go through catch operational issues of running this type of cloud and you don't want to rely on the Zool team that's running. They're busy running their own CI stuff. I'm just responsible for making sure my cloud runs. Well, how about if I run my own node pool instance? Basically running just the component within Zool that is responsible for spinning up and tearing down virtual machines. So went through and did it. And node pool has a pretty easy configuration file. You can go through and set it up. So what I did is I wrote up a configuration file that said max hold age. Max hold age is how long a virtual machine can stick around for. I said, okay, five minutes. I don't want these. This is just a stress test. Max servers. Okay, I want to make sure I can hit 100 virtual machines. So I put max servers to 100. Minimum ready. I always want 100 virtual machines ready. 100. And then the rate. How quickly it should submit API requests to. So install node pool, install zookeeper, write a configuration file. So that is connecting into the cloud and just let it run. Let it run, go through, run it as a stress test. Can your cloud actually run 100 virtual machines at the same time? How fast a rate until it, you know, keels over? Where does it keel over? Is it, you know, are you going to run into a quota issue where a quota is not updating fast enough? Are you going to run into, you know, or are you going to identify one particular compute node that's acting up? And when you take a look at your open stack dashboard, you see that that compute node has got one tenth of the number of virtual machines that the other ones have. But this is a great way to go through and stress test the cloud without having to rely on the node pool team. I had run rally previously. And for those of you who aren't familiar, rally is an open stack tool to test various performance. But this way I'm using exactly the same tools by running node pool that the zoo team is using. So I get the same experience that they're experiencing as well. The one word of warning is you don't want to run this at the same time that they're running their production workloads. For example, this requires floating IPs. I have a limited number of floating IPs. So while I have plenty of compute resources to run all the virtual machines, I don't have enough floating IPs. So keep that in mind. Or if possible, you can configure your node pool instance just to use private IP addresses if you can do it all within a private IP address space. So, okay, so the next thing I had is had some users come up and say, well, we want to do continuous integration, but we have very specific hardware or we need to do performance testing. And we don't want to run it in a virtual machine. So software for open stack, great, run it in a virtual machine. You can run DevStack in the virtual machine. You can test it all you want. But we did run into some specific CI tests where we did need to have some sort of bare metal continuous integration, but we still want to use Zool. So node pool has an abstraction layer within it. It's called the packet, oh, sorry. It's called the node pool driver. And what it allows you to do is write your own Python code to talk to whatever technology you want to. So you could have a talk to AWS. You know, there's an open stack driver for it, obviously. There's a static driver which basically keeps on connecting into the same machine and just reusing the same machine. So what I did is I wrote a node pool driver and included in that allows it to talk the packet APIs. So what we can do now is Zool will go request resources. It gets the credentials back, but it happens to be an actual physical machine and it can go through and it can run its CI tests that are once there on the physical machine. People get their performance testing that they want on a bare metal machine and they can have access to whatever specific hardware specialties are on there. Okay, so in conclusion, you know, donating to open source is rewarding. The open stack is rewarding.  Most people go through, they write source code, documentation, but here's another way. You know, get involved by running a cloud or donating the computing resources to do it. If you're a cloud operator, KPIs, I think node pool is a great way to get data back out of your cloud. Time to ready, failure rates, API responses. If you're not already monitoring those, I found that was a great way for me to stay ahead of things. Automates. Like I said, initially we ran into quite a few issues where we just didn't have all the physical hardware configured in the same way by going through and automating some sort of bare metal deployment of the physical systems that helps reduce those errors. And then Zule is pretty flexible. Like I said, you can use it for stress testing a cloud, you can use it for operational monitoring, and then you can extend it through some of the abstractions they have to things like bare metal CI. Okay, well, that sums it up. Thanks, everyone, for coming out. I hope that you're having a great time here at the summit. So if there are any questions, we do have microphones, and I'll be sticking around here afterwards as well. So I'm happy to post the slides.


Yo daddy! Noise or mandatory? Oh, look at this walking steak. It's amazing. How does that steak move like this? Wait, what? It didn't even swing. I hate this new combat so much.   I think Thank you so much for purchasing that toothbrush from quip That helps me man quite a bit Enjoy that brush Jarrah blaze certain valin bunch of the people got there is Initial reports that they were generally Generally satisfied. I don't know if anybody had any problems with theirs, but they were generally satisfied You maybe had to run Minecraft on a core to duo, huh? neat neat All right, we're gonna make this cobble. I thought I heard something else around me. Is he done with his slimes? I mean the other half Chad is too delicious. Oh Yeah, oh, yeah, look at it go feel it in your bones Oh Yeah, I think it's just the right mixture of honest the oil and vinegar and the and the jalapeno mix It's making my mouth sing It's like spicy and it's like that Pungent play. I don't know how to describe it. I'm not a food network regular. I don't know how to describe taste. I Don't know how to describe taste It's just GOOD How you say good, yeah, oh Yeah That's exactly what I want it right there. That's it. I read it. All right. This is the mine I'm gonna make some signs Here's your sign. I Need to stick a whole bunch of this I don't have a whole bunch of this yet.   I'm sorry, sir. We don't have song requests Yeah Don't know are they got to mr. Roboto You're wondering who I am Secret secret. I've got a secret. I've got a secret. I got a potato.  We're just gonna bake potatoes. Did they nerf my potatoes chat? I love me some taters boy What?  I like it lemon's getting tired. What happened? Well, it's like way past his bedtime. Don't worry lemon. It'll go up on YouTube. I'll put this whole thing on YouTube. All right Just for you Just for you. I'll call it one of returns to Minecraft for lemon episode 1 through 700 Yeah Yeah Yeah Yeah Just for you It'll be our little thing No, I gotta admit I gotta admit the last two emotes he made shocked me how good they were This is triggering you there Think it's this That's better Okay, I think it's time to dig a hole chat And whatever we dig the hole that it's gonna make this house Because we started with wood. We're going slowly to cobble and this is going to be made of gold Something like that But I got finished the sandwich.  I'm being nice Face of hyper clap exactly Oh man I don't have to add that to my personal emotes. I'm very This is my hole man. This is where my hole is going to be man I got a real good bite right there on some uh, jalapenos Woo a really good bite chat Let me tell you Oh, man That one's gonna stick with me a little bit. Oh Woo boy. Howdy. Don't you know Oh, it's night. All right. I need to take a bed with me too cool Do I have enough wool to make another bed? Let me see Uh You can't mix them so I need one more wool Or can you mix them can I have two white one black? Ebony Then ivory Live together in perfect Harmony side by side on my piano Keyboard oh lord. Why don't we You can't mix them come forth here then some such my good man I will make a mixed bed Yeah Oh, you will have to be why? I can't mix them Why would you do that to me? Why would you lie to me no mel?  I didn't get to work What is this? I need another sheep and and I'm going to be racist.  Welcome, man. I don't see no sheeps. All right. There's gotta be some in here Looks like og's gonna lose Okay, why is everybody rooting for og because I'm oh I've always been rooting for eg And all of a sudden I guess they want them to win because that is there people on the team that never wanted ti That people are rooting for That underdog-ness Because eg hasn't lost has they have they I don't think eg is lost Oh, they almost disbanded. Oh, it's to come back kids. Okay And it's their first time in the finals Okay We're talking about doda to the ti they're playing for millions of dollars the international Something I usually watch but I'm not watching this year because I'm streaming Yeah Yeah I'm a full-time streamer chat. I can't be entertained while i'm entertaining. That's not allowed What's club penguin? What is that? Come on. I have some sheep Don't knock club penguin.   That's my face. He drew. That's it. That's my face I can like scrunch my face and turn like all my two I don't see the sheep man. And this last bite tawned me. Let me go ahead and eat this last bite That was pretty much all bread you have wool for me. Is it white?  I got one Thank you. No, no I got one It was one on the way You pointed me in the right direction, sir specter. What's up? Did I time travel? No, you didn't I I promised I stream I do a full minecraft stream if I hit my sub goal Try your elevator. All right.  Am I gonna die? Is this it here?  How you get back up? Well, if you're gonna die, you might as well do it in full stomach But Okay Okay, how'd you do the bubbles? Wait, how'd you do the bubbles? Is it bad? I never played minecraft. No, it's not bad. I use the bubbles How was this? How is this the thing? What is that? That's some kind of Nether ore or something Carbonated elevator. I know right How's the word not spit out the door? I know, right? How does this even work? I just feel like I'm gonna die. Whoa Well, a little bit to this minecraft stream. Specter. Thanks for the love Oh, he's got something here He's got some kind of block there. Thank you so much speckle for the 3 000 bits. Everybody shows some love in the chat That's a lot of bits chat Thank you so much, man Looks like a prison Hey, he's doing a half block roof. I used to do that now to think about it But I used to combine it with the other carved blocks The minecraft The half block life, but it was glitchy Something I didn't like about half blocks. They were really glitchy It was like a piston not a prison. What this? You just give me all your wool. Well, thank you I could build beds for the entire server now for the entire server Buona's bed and prorium come by today 50% off labor they sell Free financing through april 2019 Buona's bed and prorium Come wait a minute. They don't stack Or didn't spin them out No marks uses chat.  No marks uses Buona bed and prorium You can't go to sleep at night come by Buona's bed and prorium today Yeah, there are going to be some good bills bed sales coming up and I'm not going to be able to afford them Like the sleep number beds are already on sale and I'm just like oh gosh.  I got a lot of a lot of junk in here. Uh, let's cook these That apples Apples some extra doors Some raw mutton chops Some dirt You really only need one stack of dirt Hey Tiger strike. Thanks for the host man. Everybody's showing some love in the chat. Thank you tiger strike for the host Appreciate it, dude Hurry up and cook Sleep numbers snake oil, but you know what I guess I'm subscribed to it because it's better than my crappy bed I know there's cheaper ones out there too.  All right. We got to dig our hole chat. We haven't started digging our hole yet This is making me sad inside All right extra wool glass seeds Um, I don't have a lot of arrows. Do I? No Hope you're doing well. My man. Love the relax stream as always. Thank you recon pastry. I'm doing okay, man I'm nursing a headache. I just ate some food and this made me feel a little better. So That was I was hungry Nursing a little bit of a headache, but I'm okay other than that Whoa, whoa Wow Spectal dropping five gifted subs To sink Xavier to christmas 16 to rt gears to neon nineties and nunya getting that sub re-up Thank you so much. Mr. Spectal on top of the 3 000 bits dude. Just drop Five gifted subs everybody's showing some love in the chat do it. I ban you You get banned. You don't show love There would be love love D Thank you, man. That's that's a massive love, man I'll tell you man. Maybe I should stream minecraft more often right chat, right? Maybe I should stream minecraft more often because Hey rusted minecraft I've been sabotaged chat. Thank you so much any rap Holy rusted minecraft That's just evil I'm always watching and he puts a head on my windowsill chat Thank you any rap for the 500 bits. Everybody show him some love in the chat Can we get some uh, can we get some boona boys for any rap and chat just a single boona boy Boy boona boy But who did this? It's just heads Some boona boys boys I'm always watching That's just that's that's like creepy At the same time. Okay. Um, so Yeah, I got everything. Well, I got gray wool sudden. I don't even know I guess Let's go dig a hole chat. I've got no tools. That's cool Let's build a whole bunch of picks Hopefully we'll get enough iron down there And by a whole bunch.  I don't have enough wood for the shovels. Do I two shovels We're using stone tools because I'm I got to get more iron. I only got these guys two shovels And a patch heads in a patch Extreme elements. Okay. Why what's happening? Should I gamble all of my 400 billion points? Of course was three elements? Oh, that's a feature That's a that's a feature when you have an emote combo. I was scaring me. I'm thinking something's wrong That's the first time 60 few have typed the same emote at once And it only took me to ask you all to do it. Yeah Did he win again? Dude He's in the lead by a wide margin chat 800 billion points What day is it what time and what noon what the deuce is happening right now All right, we're gonna do a I don't know I guess a three by three


Hey everyone, welcome back to theCUBE's continuing coverage of AWS re-invent 2021 live in Las Vegas. Lisa Martin here with David Nicholson. We are running one of the industry's largest, most important hybrid tech events this year with AWS and it's enormous ecosystem of partners. We've been talking yesterday and today about the next decade in cloud innovation. We're pleased to welcome back one of our CUBE alumni, Alan Henson, the vice president of Parvada Solutions. Welcome back to the program. Thank you, glad to be here. So talk to us about what's going on at Parvada. What are some of the things that you're focused on in the energy space? So for us, specifically, we're looking at energy as it's redefining itself, right? It's no longer just oil and gas, it's no longer just coal. It's renewable energies, it's carbon capture, it's retail energy, it's power generation. We're trying to understand as this energy industry redefines itself, how companies are starting to participate in there and how technologies are really helping them as they redefine their business models and go through their digital transformations. I was doing some reading that Amazon is already the largest purchaser of renewable energy in the world and will be 100% renewable energy by 2025. Talk to me about the partnership with AWS. So our partnership goes back quite a while. We were one of their early partners. We saw the early potential of AWS jumped in, both feet in the deep end, and have had a wonderful partnership ever since. More recently, over the last couple of three years, as they've really matured their definition of energy, we've been right there with them, partnering them on a number of different projects across multiple industries, and really getting excited about how they define their journey going forward in the energy industry. We're in such a state of flux right now, we have been for quite a while, but talk about the energy transition. What is that? Why now? Are we all ready? Oh, great question. So the energy transition is really about us as a society looking to new sources of energy over fossil fuels. 83% of the world's energy today comes from fossil fuels, about 30% from coal, a little over 50% from oil and gas itself. We as a world are looking at this as energy that can't continue to be our primary source. It has been a pivotal source of energy getting us here. It is cheap, it is reliable, it is abundant, it's very good at what it does today. However, it's causing harm, so we need to address that. So what we're doing looking forward is helping companies understand what does it mean to be net zero? What does it mean to monitor and capture your carbon? What does it mean to transition your business model from a pure oil and gas play to one that's looking at new energies like carbon capture, like wind, like solar? So we're helping all these companies go through that journey as they figure out those next steps. So where does Paraveda come into the picture? What, take this down to the level of an engagement with an energy company. Who are the people that you parachute in? Sure, to engage with a customer. Well, let's talk about oil and gas first. The most near term best solution that we're seeing is really about operational excellence. How can I better run my business so I reduce my footprint? Am I having better maintenance to reduce accidental spills? Am I controlling my unexpected emissions, like flaring? How can I use technology to help me run my business better? Then we have companies in the retail energy who are looking at their customer base who are purchasing large amounts of energy who they themselves are striving to be net zero by 2050, 2060, and creating solutions for them so they can begin to capture their carbon solutions. So we typically parachute in a couple of different types of groups. Paraveda is both an advisory services firm and a technology services firm. So based upon where a company is in their journey, we may start more in the strategy space, working with the C-suite. What does it mean to build and structure an organization to strategically go after this new energy space? Or perhaps they're focusing more on operational excellence. We're trying to figure out how to leverage technology and process to run their business better, more cost effectively, more importantly, more ESG friendly. ESG is a big buzzword these days. Are you talking about conversations at the C-suite level? Has it always been there or is that something that with climate change and all the things going on, you think of Hurricane Ida, the drought in California, the fires, the flooding in Washington, is that something that is now more of a focus at the C-suite level? I think it has to be. Global warming first became mainstream back in the 70s. When we first started realizing there was going to be an impact from our use of fossil fuels. Originally, it wasn't quite sure the scientific data wasn't there necessarily to support it, but now we're starting to see that there is a climate change. Whether or not it's caused by fossil fuels is a debate for another day, but we are starting to see that. As a result, we're also starting to see a lot of pressures coming from very different areas of our world. The financial industry wanted to become a lot more transparent about their investment holdings, which means they're looking at their portfolio and choosing whether or not to put investment into capital projects, which is the heart of oil and gas companies. We're seeing social pressures, not just with campaigns and protests, but where people are choosing to work. The products they're choosing to buy, the brand that they're using to associate their identity, and that's also creating pressures for the C-suite to start to pay attention because that impacts their whole pipeline of talent. So I think we're starting to see that because of those impacts becoming much more far-reaching than just some scientific publications. Well, the focus on people is critical. You know, we talk about often people process technology, but the people focus is critical, especially since this has such, from an optics perspective, global visibility. Yes, people is critically important, not just from a talent acquisition, but also from a talent development perspective. These organizations are going to be going through some pretty dramatic changes. They're going to be leveraging technology they're already familiar with, perhaps some processes they're already familiar with, but they're ultimately moving into new industries, new competitive markets. So you need to not only be able to recruit the top talent by promoting an image that people can align to, but also be able to build that talent internally so that you can make them effective as you go through these business model transformations. You know, there's been a theme that we've seen just in the first full day today here at AWS re-invent, and that theme is that the AWS ecosystem is thriving and critical to the success of what AWS is developing. You've given a great example of an organization, your organization that connects or bridges the divide between technology and the value that technology can bring. You said it at the outset, you're not going in and talking technology first, typically, it's let's talk about strategy, let's talk about people and culture, and then let's find the tools that are best suited to leverage to achieve the goals. Because you start out talking about energy, so which compute instance do I need for that exactly? Somebody has to connect those dots. Absolutely, that's what I love about AWS. They offer a broad suite of services in their overall cloud infrastructure that allow you to start in multiple areas. Let's start with one of the very first projects that we got into that really focused on operational excellence and improving how an organization ran its business. This is a public reference, so I'm going to go ahead and say it, we did work with AWS, partnering with them, at ExxonMobil to help them redefine how they collaborate with their engineering, procurement and construction companies. So this was a cloud platform that allowed them to change the way they interface with those vendors as they took on these large capital projects. We were able to take time out of the system, help that organization run more profitably, which allows them to invest in that new energy technology. Then we moved to retail energy where we've been looking at a large adoption in IoT technologies. That IoT capability of AWS allows organizations to monitor their infrastructure, understand how their equipment is performing, where carbon emissions might be occurring or other greenhouse gashins might be occurring, and bringing that in. Then you bring in the AL and ML stack capabilities. One of the leading ways of detecting emissions right now is image machine learning around emissions, looking at satellite photos to see if there are changes in the atmosphere to where there might be a methane leak. So all these technologies work together to help us derive better answers for how to be a better energy company and how to be a more environmentally friendly corporation. What's the customer flywheel like? Often we talk with AWS, they talk about really, we start backwards, we start from the customer, we work forward, the customer obsession, we saw the NASCAR slides this morning of all the logos, I'm sure many more that didn't fit, but talk to me about the alignment between Paraveda, your focus on the customers, how they help you guys innovate and create new solutions. I imagine similar culturally to AWS. That's one of my favorite aspects about being a premier partner with AWS is the cultural alignment. We have a process that we call right to left thinking. It's beginning with business outcomes before we ever look at the technology, ever start to design the solution, ever start to build that solution. Are we meeting the end user's needs? And where that cultural alliance with AWS is it aligns with their working backwards process where they sit with their primary end users and pick some point in the future and say if I were to do a press release today, announcing the solution that we just built, what would it say? And then if I had questions, I wanted people to be able to find answers to, the frequently asked questions, what would be in those frequently asked questions and what would those answers look like? So those two approaches to starting with the right to left, business outcome focus, helps us begin with the most important thing, what we call jobs to be done. So we're not working on a symptom, we're working on the actual problem. And that's where we really align with AWS and our cultures have helped us focus on the most critical issues at hand. So we've talked a lot about energy and your affiliation with AWS in these efforts. But tell us a little more about Paraveda in terms of just in a more broad sense. What's the history? Sure, so it's the pitch. Yeah, absolutely. Paraveda has been around for over 18 years. It started with a vision that our mission should be about developing people to their fullest potential. Start there and everything else will come. Since then we've developed into a both a strategy services firm and a technology services firm where we want to bring together what we consider to be the three primary components of an enterprise architecture. Basically business, product, technology all wrapped around strategy. And so we want to focus on those areas and we use those to help deliver projects. So whether it's technology, whether we're helping travel and hospitality companies that you probably would recognize or we're working with sports leagues to help read brand, we're working in the agricultural industry to change how they capture data from equipment in the field to working in the medical industry, redefining the way that doctors work with their patients by capturing their entire conversation for them. So they're spending less time translating their notes instead of valuing our notes to ensure top quality healthcare to working energy. So we're based in North America. We have 10 offices in the US, one in Toronto. We're about 750 strong and we are really focused on deliberate growth versus just exponential growth. Well that outcomes focus is absolutely critical as you talked about earlier for every organization. And one of the things that we learned during the last 22 months is that real time data access is no longer a nice to have. It's absolutely essential. But also we're also seeing every company is becoming a data company. But they have to learn how, how can we work with technology partners like AWS like Paribetta to be able to capture the value that it unlock it quickly so that we can iterate and be able to deliver, especially in this interesting climate that we're in and pivot and pivot and pivot. So that outcomes focus that Paribetta has is really critical for enabling businesses in every industry to be able to survive and thrive these days. You said it well, we agree with you completely. What we've designed at Paribetta is what we call the modern data enterprise where it looks at the holistic vision of becoming a data company. Everything from governance to technology choices to how I structure my organization to have the right roles, the right leadership, the right executive support and thinking about the full picture of delivering a successful data platform so that we can really focus on something that one of my mentees calls data liquidity. It's the ability of a company to convert their data to value as quickly as possible. In order to do that, you have to come at it from multiple angles. You do and that's a competitive differentiator these days. Yes, absolutely. It's more than just let me bring you a data lake and hook up some data pipelines. Again, if that's where you start, you're probably not starting in the right place. We want to start with the end users. What questions are they answering? What jobs are they trying to get done? And then moving back and say, well, what data do we need and what form do we need and how do we present it to them so they can do their job extremely well and create their own competitive differentiation. Right, that's really critical. Last question is we have just a few seconds left. Here, Alan is so much announced today alone. This is like David said, the first full day of coverage at ReInvent. But from a visionary perspective, what are some of the things that we can expect to see from Power of Beta as we finish the year 2021 and enter 2022? So let me speak a little selfishly to energy. What you're going to see is us as an organization are going to work side by side with AWS Energy to help shape the picture. What does it mean to be an energy company? As these industries start to converge, we're going to build more holistic platforms, more holistic approaches so organizations can figure out, how do I still do the business I need to do today to fund the business I need to be tomorrow? And you're going to start to see us bringing better messaging to that on both strategy, technology, and product approaches. Awesome, Alan, thank you for joining David and me talking about what's going on with Power of Beta. Awesome approach, I love the people focus, great stuff. I love it, Lisa, David, thank you for having me. Oh, our pleasure. For David Nicholson, I'm Lisa Martin. You're watching theCUBE, the global leader in live tech coverage.


Let us return to our question of whether ethnography is a process or a product. If you consider these elements, you may notice that each is both a process and the outcome of that process. Ethnography is a process. In ethnographic practice, we draw on various methods of research such as interviews and observations. We are in direct contact with our participants in the context of their daily social and cultural lives. We observe various phenomena and interactions and ask questions about what we observe. And it is also a product. As a product or an outcome of a process, ethnography is a rich descriptive account of human experience. It respects the particularity of that experience and does not generalize it. It acknowledges the role of theory in shaping our understanding of human experience and recognizes that the representation we create is filtered through the ethnographer's subjectivity. It presents the research participants as part object and part subject. This means that we learn about our participants and we learn from them. It acknowledges their agency in shaping their context and our research. Let us try and grasp this idea by referring to an earlier example. We return to the study of the students and teachers in a school to explore their relationship with their institute. If you were designing this study, what are the different ways that you would use to engage with the participants? What activities, context, interactions will you include in your fieldwork? Pause the video and list down all the ways you can think of. Now let us see if some of your responses match with ours. We could use multiple methods of engagement to learn about our participants. We could observe the formal and informal interactions between teachers and students in the classroom and outside of it. We could spend time with students and teachers in spaces such as the staff room, the library, the canteen and so on. We could also accompany some of them to their homes. We could map a day in the life of the classroom by asking our participants to plot their daily activities across the timeline of a day. In this mapping, we could learn about the activities, artifacts and relationships that make up the classroom. We could interview students, teachers, administrative staff and others. We could also participate with some of them in their daily activities like learning or teaching. By using these methods and maybe some others too, we would engage with our participants in a direct and sustained manner. We would learn about them and their context through different kinds of observations and interactions. And we would ask them and ourselves questions regarding our observations. In this manner, we would be conducting a research that is ethnographic in its process. For our study to be considered ethnographic in its outcome, it would have to reflect a certain side of understanding and knowledge. This would include an understanding of the relationships that the participants have with each other. The study would represent the perceptions of various participants about their classroom. The study would present what the classroom means to different participants. For instance, the classroom could be seen as a space for learning or as a space for meeting friends and mentors. For some, it could be an escape from home. For others, it could be a restrictive dull space. The study would include an understanding of the particular context, experiences and ideas of individual participants, such as the participants' understanding of themselves, their homes and their relationships. These individual perceptions would help us understand the specifics that form the shared experiences and ideas in a classroom. The study would draw connections between the context of individuals and that of the school. So the study may look at the physical location of the school, its fee structure, the academic performances of its students and so on. These various insights would find support in existing theories of learning and teaching, and maybe theories of community bonding and of relationships between people and spaces. In representing all of these observations and analysis, the research would present the agency and knowledge of individual participants. Their reasoning, their perceptions and their opinions would also be presented in the outcome. At the same time, the study would describe the circumstances and events acting upon the participants. It would acknowledge the subjectivity of the researcher, their ways of seeing and their assumptions. In fulfilling all these criteria, our study would be ethnographic in its outcome. You may note that in the listing of these attributes, it is hard to distinguish which attributes refer to the process and which refer to the product. This is an inherent quality of ethnographic research. It is through following an ethnographic process that we may create an ethnographic product. For instance, we need to observe and engage with the home context of individual participants. This is a process related attribute. But it is this process that enables us to present particular experiences of individual participants. And this, as you can see, is a product related attribute. So it is the very attributes that make ethnography a process that also makes it a product. Let us take a break now. We will return to unpack some fundamental concepts that inform the processes in our next section.


This study examined the relationship between nurses' demographic characteristics and the use of pain assessment tools for critically ill patients. The results showed that certain factors, such as hospital type, academic qualifications, years of experience as a critical care nurse, and hospital affiliation, were associated with increased use of self-report pain assessment tools for verbal patients, while other factors, such as hospital type and hospital affiliation, were associated with increased use of observational pain assessment tools for nonverbal patients. These findings suggest that nurses' demographics may influence how they assess and manage pain in critically ill patients. This article was authored by Mohamed Rababa, Shada Al-Sabah, Anwar M. Ayodat, and others.