Hey, this is Brett and this 2022 Chevy Silverado 2500 Crew Cab Short Box LTZ is stock number 13689Z. I am here at Summit Automotive in Fondlack, Wisconsin, your new and used heavy-duty truck headquarters. This 2022 Chevy Silverado 2500 has the 6.6 liter V8 gas engine, puts out 401 horsepower. This truck has been fully-safetyed and inspected by our service shop per the state of Wisconsin inspection process, has a fresh oil and filter change, all the fluids have been checked and topped off, and this truck is 100% ready to go. I'm going to go all the way around in this video, inside, underneath, start it up, and take a look under the hood to give you the most accurate representation of the vehicle possible. Summit White is the color, and I shoot all my videos in 4K, so if you have HD capabilities on your computer, tablet, smartphone, or television, turn them on now, because it is your best way to check out the quality, condition, options, and cleanliness of the vehicle before seeing it in person. As you go down this side of the truck, you can see the boxes in excellent shape. Cab and doors are in fantastic condition as well. I didn't see any dents or dings on there. Paint is very nice and reflective. If you like the video, you can subscribe to my YouTube channel, which is youtube.com slash Summit Auto. Click the bell notifications and you'll get updates on the videos I do each and every day here at Summit Automotive, as well as access to one of the largest catalogs of vehicle walk-arounds on YouTube. Very nice on that side comes with the painted and polished aluminum wheels, and it has Michelin LTX AT tires, these are LT27570R18s, and I would say they have right around half the tread left on them. Frame and underbody is in nice shape, it does have the Rancho shocks with the Z71 off-road suspension package. This one has the LED headlamps, LED running lights, and LED fog lamps, we'll turn all those on at the end of the video, so definitely stay tuned for that. You get the chrome trim grille, no dents or dings on that front bumper, and you get the sport hood, didn't see any dents or dings on the hood, and the passenger side front fender, no major dents or dings on there. Passengers side front wheel, no scuffs or scrapes, really nice condition. And as you go down this side of this 2022 Chevy Silverado 2500, take note of how clean that body is, how reflective and mirror-like that paint is. Take these HD videos, so if you are far away or even if you're close by, just cannot make the trip down, but you're still interested in purchasing the vehicle, you can see the truck, hear the truck, and of confidence in the vehicle that you're looking at before you even get here. So when you do get here, there's absolutely no surprises, and you can make a smart and informed buying decision from wherever you're at. If this video helps you make that buying decision, let your salesman know that you saw the video, that it was helpful, and that Brett sent you. Back wheel is in excellent shape, and the back tires have just about as much tread as the front tires. Even underbody, pretty nice condition, a little bit dusty, but nothing too bad, lower rockers all look great. Has some nice rough country step bars, you get the side steps on the box as well. Coming around to the back, you get the LED tail lamps, the rear bumper steps, the rear bumper parking sensors, no dents or dings on that rear bumper. Full towing package, which includes a receiver hitch, seven pin wiring, and then you have some auxiliary cameras you can hook up for a trailer, and then the seven pin wiring right there. This one has the power drop down tailgate, as well as the power lift tailgate. It's in pretty nice shape, goes down like so. The only thing I noticed is one little ding on the top of the tailgate there. Spray and bed liner has the toolbox LED bed lights, and this one also has 120 volt, 400 watt plug-in. Spray and bed liner goes on to the tailgate, and this tailgate also goes up and shuts nice and solidly. Back wheel is in nice shape as well. Hand down the rest of the side once again, all looks really good. It does have the capitalist feel-fo, which is a very nice feature. You do get the telescopic tow mirrors. They're heated blind spot monitoring, LED side lights, directional signals on there. There's your 360 camera. Inside the LTZ package gives you the black leather bucket seats, no rips or tears on those seats. They are in fantastic condition. Poor driver's seat with lumbar, factory all-weather floor mats throughout. Auto headlamps, it's for your cargo lamps, fog lights, push-button four-wheel drive, tilt telescopic steering wheel, power windows locks and mirrors, memory driver seat, and you do get the Bose sound system. These mirrors do power telescope out. I always like showing both sides so that you know that both sides are working properly, and then they do power fold in as well, and out. Take a quick look at the back seats. They are just as clean as the front seats, no rips or tears back here. Does have the latch child safety system for any child car seats you may have. Power sliding rear window with the built-in rear defrost, headliners in great shape. Also has the rear seat storage bins. These fold up. There's storage underneath there, factory all-weather floor mats back here. You get a USB, USB seat, and a 12-volt power point. There's a little storage area there as well. Child safety locks on the back doors and the bottoms and inside of the doors all look fantastic on this truck. Before we hop in, I'll show you the VIN sticker as well as the tire or the trailering information sticker. GM is nice enough to give you that, and then they have the tire and loading information sticker right there as well. We'll hop inside, check out the miles, the radio, and everything that this truck has to offer on the interior. So you can see that this one has 18,671 miles on it. Instrument cluster is very nice and clean. You get a compass display on there as well. Comes with the heated leather wrap steering wheel, information, and Bluetooth controls on the right. Cruise controls, adaptive cruise controls, and heated steering wheel controls on the left. You get your six-speed automatic transmission with the optional tap shift, and this one has the Chevy infotainment system here. You do get the factory navigation system on there. There's Highway 41, so that is working nicely. You get AM, FM, and Sirius XM radio capabilities as well as Bluetooth capabilities on there.  Let's see if I can get to that. You got this loading cam as well. That's pretty cool. And then you can look behind, down the sides, and then of course your backup camera right there. And then you get the 360 cam as well, right there. So that works nicely. Way a cool system to have.  You can power drop or raise that tailgate. Hazard light stability control, that's for the plug-in in the bed. Downhill assist control, factory brake controller, two USBs, well, one USB, one USB-C, 12-volt power point, and a 120-volt, 150-watt plug-in. You get the wireless cell phone charge pad there, keyless entry with room to start. And dual gloveboxes, passenger side format, and seat are in fantastic condition as well. Once again, the headliner is like new, smells very clean inside this truck. I don't think it's ever been smoked in. And you get your home link buttons for your garage door security systems and lane systems, map lights, power slide and rear window button, and your SOS and on-star buttons right there. We'll start it up and take a look under the hood and check out all those LED lights in the front. Starts right up, no check engine lights or anything like that. Set my hoods open. And just to show you that is the adaptive cruise control there, not just the forward collision warning. I would personally like to thank you for checking out the video today. And hopefully from this HD video, you've been able to verify the quality and condition of this truck all the way around. There are those LED side lights, they are super bright. And like I said, hopefully you've been able to verify the quality and condition and of course, cleanliness of this truck all the way around. Look at all those LED lights there, extremely bright, very nice. And under the hood, we have the 6.6 liter V8 gas engine, engine base, very clean runs, very smooth. Once again, this truck has been fully safe and inspected by our service shop, has a fresh oil and filter change. All the fluids have been checked and topped off and this truck is 100% ready to go. Here is the emissions sticker. And I would highly recommend this truck to make quality and condition standpoint. I would have no problem shipping it to California, Texas, Michigan, New York, Florida, you name it. Whoever is going to get this truck is going to be very happy with it. And to see more pictures of this truck or one of our other 550 new and used cars, trucks, SUVs, minivans, Wranglers, halftones, recoater tons, one tons, you name it. We've got to go to that website right there, summitauto.com. Full pictures and descriptions of every single vehicle, all at summitauto.com. And if you'd like to check out more HD videos, you can go to youtube.com. So at summitauto, click the bell notifications, you'll get updates on the videos I do each and every day. Here at Summit Automotive, as well as access to one of the largest catalogs of vehicle walk-arounds on YouTube. In fact, in a second, you'll see a link to subscribe to my YouTube channel on the upper left. A link to more Chevy 259 truck videos that I've done in the upper right, a link to this vehicle on our website on the lower left, and a link to one of our latest YouTube videos on the lower right. Click those, check it out. We're super excited to help you with this Ultra Clean 2022 Chevy Silverado 2500 Crew Cab Short Box LTZ in Summit White Clear Coat. Thanks again for checking out the video. Remember to like, subscribe, and share on the YouTube channel. I really appreciate it. Thanks again.


As a dedicated VHS collector, I've spent countless hours souring thrift stores, flea markets and online auctions in search of rare gems to add to my collection. From classic movies to obscure releases, my shelves are lined with pieces of cinematic history. But amidst my treasure hunts, there's one elusive item that has haunted my dreams. The Lost Walt Disney Home Video Logo. It all started innocently enough. I stumbled upon a forum thread discussing rare VHS tapes, and one user mentioned an obscure release rumor to feature a never-before-seen version of the Walt Disney Home Video Logo. Maybe this release could be any old Disney movies from the 30s to the 80s? Intrigued I delved deeper into the mystery, determined to uncover the truth behind this elusified. After weeks of relentless searching, I finally struck gold, or so I thought. A listing appeared on an obscure auction site, authoring a VHS tape simply titled, Disney Home Video. The description was vague, but something about it sent shivers down my spine. Without hesitation, I placed my bid and anxiously awaited its arrival. When the package arrived, I tore it open with trembling hands, revealing the nondescript VHS tape inside. Excitement coursed through my veins as I rushed to my VCR, eager to witness the legendary logo that had eluded collectors for years. As the tape began to play, I was greeted by the familiar black screen, but something was off. The quality seemed grainier, the colors muted as if the tape itself had aged beyond its years. Nevertheless I pressed on, anticipation building with each passing second. Then it appeared, a faint image of Mickey Mouse, dressed in his sorcerer or his apprentice outfit, fading in against the black backdrop. There was something unsettling about his expression, a sense of a knees that sent a chill down my spine. The camera zoomed in slowly, too slowly, as if reluctant to reveal what lay beyond.  And then, without warning, a white star burst flashed, accompanied by a discordant jolt of synthesized music that sent shivers down my spine. The magic spark danced above Mickey's hand, but instead of writing out Walt Disney in the familiar corporate font, it seemed to twist and contort, forming grotesque shapes that bore no resemblance to the beloved logo. I watched in horror as the words home video materialized beneath, but they were warped, twisted into jagged edges that seemed to pulsate with malevolent energy. I wanted to look away, to eject the tape and pretend I'd never seen it, but something held me captive, a morbid curiosity that compelled me to keep watching. And as the logo faded out, leaving nothing but darkness in its wake, I couldn't shake the feeling that I had stumbled upon something far more sinister than I could ever imagine. To this day, the memory of that lost Walt Disney home video logo haunts my dreams, a chilling reminder that some mysteries are better left unsolved. But as a VHS collector, I know that curiosity will always get the better of me, driving me to seek out the truth, no matter how terrifying it may be. At least it's not one of those brain rot spin pasta copycats, am I right?


Thank you, Julien. Merci, Merci. So I'm going to jump right into it because since we have all this time in the discussion time, discussion slot, I'm not going to go over a lot of the sort of comparison debates about the different approaches. There are some excellent articles, especially issues of paleoanthropology, among others, that have gone over some of the differences. I thought today the point was really to compare, contrast, combine ideally these approaches and ideally separate these differences and put them together, reconcile them if you want. Reconciling sounds like maybe we had a divorce, but I don't think we ever had a divorce. It was probably a amicable separation. But anyways, we can come back to this. I just took a lot of this from Odu's and also from Gustave Van's article in 2011, and we can come back to it. And these are just maybe some more talking points. But what I want to do today is give you three examples of work that I've done in Eastern North America, Eastern Canada, Quebec, from the three different periods. That I hope will show you that we are able to combine these two approaches. We're very fortunate in Quebec that we, of course, French, so we can read a lot of the French work as well in French that may not be translated. And I'll give you three examples, one from the Thalia Union period, so the oldest period, where it's really dominated by Bipatial technologies, one from the Archaic period, where it's a radical change to Unifacial Micolar technologies, and then one from the later Woodland period, which is, I don't even know what to call it, but there's groundstone and some stuff that's chip stone tools that sometimes I really wonder what the hell they're trying. So the Thalia Union period, of course, some of you may have heard of it. It's the first colonizer in North America and Quebec, the late Thalia Union is a period when people are still hunting caribou in periglacial settings, nine, 10,000 years ago. This is what the environment looked like near the site that I'm going to present. It is the kingdom, the empire, the by-face. People say that North America were obsessive by-faces, I think it's true. I think these people were obsessive by-faces and let me tell you they were incredible Flintnappers. I worked for several years on a series of quarries and lithic workshops in Gatsby of these late Thalia Indian people and essentially what they're producing, 99.99% of the time is by-faces. And what's wonderful about these quarry workshops is that we have everything, the entire sequence of production from the extraction of the rock from the quarry face right to the production of finished by-faces and points projectile points. Now, when I first looked at this, being trained in North America, I took one of our Bibles, which is Eric Callahan's work. Eric Callahan, if you don't know it, it's a fantastic piece of work. You can get it in PDF format. He's an experimental archaeologist, replicated a lot of this stuff himself. The drawings are fantastic. I think they're as good as the European drawings. And it's really an incredible piece of work for understanding bifacial technology, especially for paleo-Indians. So I thought, this is fantastic. I have like my guidebook. I'm just gonna apply it. So what I did is I find it. I played his approach to reduction stages of the by-faces. I used, it's primarily based on sort of ratios of width to thickness of the back lights and pre-forms and finished by-faces. And that was great because it kind of helped me to talk to other specialists in the paleo-Indian and tell them what we were doing, what these people were doing up in Gas Bay 9,000, 10,000 years ago. But of course, at a certain point, all you can really say is that, oh, okay, they produced a lot of pre-forms. And sometimes they finished the by-faces. And sometimes they went down to the seaside and finished the by-faces down there. But of course, at least it's an excellent way of studying these by-faces. And it's based on, like I said, replication, a lot of replication. Now, I wasn't really totally satisfied with that. So I went to another excellent source, DC Waldorf, another plant napper, another guidebook for North American analysts. And DC Waldorf could actually look at this similar kind of by-facial production where you're starting out with a tabular piece. And I thought, okay, maybe I can get a better understanding of what they're doing here. I'll use DC Waldorf and I'll sort of understand how they're going through the sequence from the tabular pieces down. And that helped quite a bit. We were able to even see how some of the by-faces kind of rotate, the cross sections rotate on these pre-forms. And it really, really was very similar to what Waldorf had described. And so that took us one step further. But I still felt like I hadn't really gone to the end of the story. So more recently, one of my graduates was by Nicolette Carr. He was trained by Nicolette Pijot. But he's in Montreal. So I said, my Nic, you know, I think I've only, I've gotten to the end of the story of what I can do with my training. I want you to take a Chenepe d'Agoin approach. So he has done a Chéma-dietritique of hundreds of these by-faces. This is his drawings. He has grouped them into technological groups. And his PhD is almost done. And so what he's done is he's essentially shown us that there is, of course, much more variability in the production at these quarry workshops that we probably didn't see. He's given us a much better insight into scale and the learning that's happening at these quarry workshops. And he's even able to group these into technical technique, which I think are going to give us a much sort of better insight or finer insight into what these people are doing. So I think that's sort of a success story. Of course, I can't take credit. My neck is really the guy who can take credit for that. Now, we jump forward into the archaic period, a very long period. This is, there's a big change in lifestyle here. Forest covers coming in. People are becoming more sedentary. They're starting to fish. They're starting to trap a lot of smaller mammals, for example. It's a really big change in lifestyle. And I looked at a collection from a site near Quebec City. This is a crazy collection. I mean, I am at ease with by-faces. Anybody who analyzed the courts know that the court is hard. Every tool in this assemblage is about this big. There's no by-faces on this assemblage, none, zero. And it's all made out of courts, both vane courts and crystal courts. And so I said, I don't really know what to do with this. And archeologists in Eastern North America were kind of, I don't know, they were kind of stumped. They were at an impasse, I think I was too. We didn't know how to explain this. Obviously it was a big technological change, but we didn't know how to deal with it. And I think that we were kind of falling back always on our typologies. And because the typologies had no place for this technology, we didn't know what to do with it. So I was very lucky because Killian Driscoll came to spend a year with me. Killian, those of you who know, is a court specialist from Ireland, worked primarily in the Neolithic and Mesolithic, he's published a lot, has done revocations, trampling, even here in Barcelona, he was here in Barcelona.  I just want you to look at it as a person coming from Europe. And sort of do a more of a technological analysis, not necessarily the Shemadi of Hicic, but maybe look at how you would classify it from your perspective. And so he identified a lot of small bipolar cores, platform cores, things that he classified as scrapers and replenishments. And that of course was great because it kind of gave us, again, a different view of what these tools were instead of just saying, oh, they're tiny, what could they possibly be doing with these things and are they really tools? And I looked through these collections with him and then after we did this, well, we said to ourselves, okay, well, obviously we were a bit frustrated because we still wanted to know really what they were used for. So this is just the sort of the classification just to show you that there's a lot of flakes that are a lot of sort of tools or retouch flakes and things made on flakes that are produced bipolarly. And then there's cores that are also used as tools. And I wouldn't call them recycled. I think the cores themselves are the blanks and then they're flakes that are also blanks. So there is an intentionality in this production, but I don't have time to get into that. But really what we wanted to do was then maybe do use wear. Now use wear on quartz is really hard, but Malini Shev's Young was trained in Denmark and analyzed quartz in sandwiches from the Arctic. And I asked her if she could look at this for us. And we weren't sure if we would find stuff because this has been 9,000, 10,000 years in the ground. So 8,000 years in the ground and a lot of free stuff. But she had a pretty good success rate in finding use wear. And I guess what was amazing was that we were able to find a really very wide variety of uses of these tiny tools, whether they were used flakes, retouch tools, scrapers or things that Killian had classified as an end scraper or side scraper. In some cases they were just flakes that we randomly chose to make sure that we weren't missing something. So I mean, again, I think this was a great combination of sort of different perspectives, especially the high power use wear approaches that are more typical of Europe than North America right now. Now a third example is again a very, very different technology. Now we're talking about people who are living in large villages, they're horticulturalists, they grow squash and corn and beans, they live in these big long houses and make beautiful pottery. And it seems as if they really didn't need stone tools anymore. We have a great assemblage of ground stone tools, which is great. And I admit that I've had to learn a lot about analyzing ground stone tool forms. And so that's one part of the assemblage. And certainly that's relatively straightforward to deal with. But when it comes to the chip stone tool, again it's very, you know, loud word expedient. It's made on different local raw materials, very poor quality. Sometimes they're just things that we might call a retouched flake or a used flake because you might see a bit of use wear on it. And so what I did is in this case, I first started to see if I could do some kind of a technological analysis, a very sort of careful technological analysis of these pieces. But what I found was that a bit like I had mentioned in the abstract, there's, we don't really have a lot of core technologies in North America and with what the French would call complicated imagination. So there's no, it's hard to see a logic and a lot of this technology.  Pables, a lot of it was bipolarally reduced or worked and then to produce maybe a flake or two. There was a regional material, again, Hornfels Terrain material, you know, Hornfels that they were in some cases flaking. In this case they were managing to rough out forms and I think they were actually then grinding down into axes. So that's interesting. That's a completely different addition to that one. And then the only real formal tools we ever found was that maybe we've won two projectile points and those always showed up on the site in finished form and they come from often quite long distances. So in this case, I wasn't really able to apply the traditional, as you would say, the technique in French of these materials, but I was able to apply it more in terms of what I would call the organization of the raw material economy. In other words, how, when you analyze each raw material, how each one is going through a different Shinneville Atois and how some of these are segmented. In other words, obviously the tools are coming in finished form while you're only having the end of the Shinneville Atois. In fact, they're rarely even reworked. And then in some cases we have local materials that are completely worked and then turned into tools but generally very expedient tools. So that was very fast. We're all into a tour of 12,000 years of human history and have a concert. But I guess my conclusions or just more just like things that we can maybe talk about is, you know, my question was can these two approaches be combined? I think they can, but not uncritically. This is something that Gustav and said in 2011. Are they two approaches compatible? I think they are. But again, maybe not necessarily on a theoretical level in the sense of theoretical as North Americans might refer to as sort of higher level theory or social theory. Again, this is something that Gustav and his mentioned would lose as well. And are there challenges? Well, obviously there's challenges. There's always challenges. I think primarily it's at the training level. I've noticed that, for example, in my university, even though we can read French and it's great to be able to read all these French authors. For example, we don't have a course where we teach people how to draw, the Shimadye Kitsi, to actually do a proper analysis and draw the technical drawings. And so that might be a thing that we could talk about. And then in the future, well, you know, what is the future for this? I think obviously the future is really doing maybe what we're doing here, especially with the younger scholars, that can hopefully move across the Atlantic or it can be trained in different places and have the ability to have this mobility either through Erasmus or other ways of being exposed to different approaches. So, I'm done.


I'm not alive. I'm going off right now. Don't worry about it. Can you grab me? Can you grab me? I just got a four. I got four. Are we spawning? Are we spawning? He's low. He's low Invader Invader Invader mid mid mid mid. No, I play brush. I play brush. He's mid. He's mid. Oh Oh, dude, he's not just blade brush the freaking immune guy. Yeah, I did I did No, I'm dead. I'm dead. I'm dead. You better hold this please Get him get him get him I'll have it. I'll have it. Let's go Yes Let's go, let's go, let's go. Let's go. Let's go. Let's go. Let's go. Let's go. Let's go. Let's go. We're going to Meeple, we're going to the tower, we're going to the tower, we're going to the tower Let's go Yeah, I told you we got this. It's good thing Sunny was before me though. Literally, we have three people at the end Yes, yes It's chest Wait, wait, which one? Which one? Which one? Yes Let me open. Ah, you punk Which chest? Yes Call me.


Hey, guys. So I'm Abhishek. I'm an engineer with cloud scaling. So OK. There you go. If you don't know cloud scaling, we are an enterprise-grade OpenStack company. We do OpenStack for our customers and end-to-end solution based on OpenStack. So starting from planning the data center to training, deploying OpenStack to support everything. So I started with a focus on software-defined networking in my career. And then when I joined cloud scaling, I ended up as a jack of all trades and doing everything from, sorry? Oh. And slower. I ended up doing everything for cloud scaling, starting from deployment to support to sales to marketing, basically everything. I have to say I'm still pretty bad at configuring actual network devices. I can barely manage to write the code. I'm Randy Bias. I'm the CEO and founder of cloud scaling. I am or have been in the past a hands-on router jock in building ISPs and large-scale data centers. So I'm just here to help Abhishek out. He's more the SDN software side of the networking part of the equation. And I'm representing our network architecture team, which is probably about three, four people strong. Sure. So we are going to explore our network architecture in detail today. And we will start with the motivation, the design goals that we have for the architecture. We will describe the physical topology in detail. We will look at the software components that actually make this happen. We will look under the hood how each component works. And we will end with our future directions based on SDN and moving to Neutron. By the way, our networking model is mainly based on Nova Network. We run as a Nova Network plugin. So let's start. But let's first look at the vanilla OpenStack networking modes without the Nova Network. First, let me ask a question before we jump into that. How many people are using Neutron? And how many people are using Nova Network? Oh, OK. More Neutron users. OK. How many of you are finding problems with Neutron? OK. I don't know. How many of you are having problems with Nova Networking? Oh. All right. Oh, but you're probably using a default networking mode, which Abhishek's about to talk about. So yeah, Nova Network supports three basic networking modes. The single host mode in which all the traffic goes through a single x86 box running Nova Network. The multi-host mode distributes the network manager into the compute nodes. And then the flat networking model. And on the right-hand side, we have the two networking modes that we have in OCS. The OCS L3 networking mode that we are discussing today. And we also have an integration with OpenContrail, that is a VPC-type networking mode in OCS. OCS is our product open cloud system that's a cloud operating system based off of OpenStack. Right. So the single host mode. All traffic goes through a single instance running Nova Network, an x86 box. It's pretty easy to use. How many people are running this? All right. That's fine. It's pretty easy to see that at some point of time this will run into scaling constraints. And when that box goes down, the whole network goes down. Not a very good situation to be at. So this also necessitates VLANs to support multi-tenancy. And that comes with all the problems with VLANs. All VLANs run out of possible tags at some point of time. And if the tenant requires multiple mode nodes and there are tags, then you're out of luck. The second mode is multi-host mode in which the network manager runs on the end boxes, the compute nodes, and is distributed throughout the whole network. The problem with this is that the network manager is responsible for providing NAT services. So in this case, NAT has to come down to the compute nodes. That's not cool. You don't want to expose your compute nodes to the internet in any case. And the third mode is the flat networking mode in which, again, no one network is centralized. And that runs into scaling issues on the network plane. So we didn't like these networking models in OpenStack. We had to come up with something of our own and which scales pretty well and meets our customer needs. So we started looking at what others have done or had to solve the same problem. It turns out that a lot of other people have actually come up with their networking models on OpenStack, on top of OpenStack. It's also turned out that a lot of them use Neutron and not Nova. I'm sorry, it's the other way. They use Nova Network and not Neutron. So CERN uses a custom driver for Nova Network that has a flat database of IP to map mappings for all of the networks. And when a VM boots up, it requests an IP address from that database. Nova Network goes and grabs an IP from that database and assigns to the VM. ANL, on the other hand, added infinite bands and VXL and support to Nova Network to increase the throughput. Given that information, we were ready to go and design our own network architecture on top of Nova Network. And here's what we have. Let's look at the architecture in detail. So what are the design goals for this? At Cloud Scaling, we believe in hybrid clouds. We believe that OpenStack's future is in working as a hybrid cloud to EC2 and GC. That makes us support hybrid cloud better. And we have to have EC2 style networking in OpenStack. That was our primary goal. We wanted to use pure L3 as much as possible because pure L3 is fast and it's well understood by people. People can just go and debug it. You don't need any extra expertise to debug a pure L3. We did not want VLANs because we did not want to run into all the scaling issues. And we did not want to run any STP because spanning tree protocol because STP requires a single interface to be on and be the primary to eliminate routing loops in the network. And that kind of kills HA in a way.  Oh, come on. I usually get like almost everybody. We did not want to have any single points of failure in the network. And that rules out single host and flat mode in an over network. We wanted to do distributed DHCP at scale for all the VMs. And we have a very clever way of doing that, which we are going to see in some time. We did not want to have any virtual routers in the whole packet path in the data plane and the control plane. So why L3 and not why anything else? That's because L3 scales. And that's pretty much a known fact. Layer 3 networks are hierarchical. That enables route aggregation. As you go up, you can always aggregate routes and announce aggregate and not individual routes. The best example of that is the internet.  Large cloud operators do not want VLANs. VLANs limit scalability in a number of ways. And a pure L3 model always forms a great underlay for an overlay SDN, which is where the whole networking industry is moving towards. And why not L2? This slide shows a breakdown of all the problems. L2 is where the topology is flat and the addressing is flat. So each router has to maintain a database of all the entries in the network that obviously does not scale for a huge cloud provider. And then we run into the same problems with STP and VLAN, spanning trees and VLANs. We wanted to have small failure domains. So if a zone node fails, it takes down one part of the network and as small a part as possible so that the whole network does not go down. And we have some throughput from the network. So an example there is if you're running single host mode in open stack and you have HA with Linux HA and you have a failover event, then your networking basically stops for 60 seconds for your entire cloud. I mean, that's not really very acceptable. If you look at the larger scale cloud operators, somebody like Google, they're running four, five, nine uptime environments. And they have regular outages all the time, hardware fails all the time. But it only takes down a very small piece of the system. So it just tends to make a lot more sense to have a networking model and overall cloud model that encourages failures that happen in isolation.  And the worst thing that could happen in a production environment is you get a cascade failure where A takes down, B takes down, C.  So I think this is a principle that still gets lost a lot of the time. You see people betting big. We're going to get a double pair of EMC Vmaxes in, and that's everything. That's the entire system. And that works great until it doesn't. And then when it doesn't, you basically have lost the entire system. And so we really felt that that's an anti-pattern really these days. Sure. So our physical network topology is derived from the spine and leaf model that Arista popularized a few years back. The spine and leaf model has a network of routers which forms the spine. And then top of the rack switches on the other end of it which forms the leaves. The most awesome thing about this model is that you can increase scalability by horizontally adding more routers and more top of the rack switches to the whole system. And nothing breaks. Yeah, spine and leaf is what allows us to focus on maximizing the amount of bandwidth that you can get with East to West traffic, right? Because we're getting rid of spanning trade. We don't have any valence across racks. So we get all that capacity back, and we want to use it all. And as you probably know, I mean, if you look at most modern data centers, even if it's not a web scale data center, and the amount of bandwidth coming out of the front end is some fraction of the overall bandwidth in the back end of the system. And so you want most traffic in most networks is East to West. And so a spine and leaf allows us to build a very big network with very cheap gear. You can even use one rack unit, 10 gig switches for that spine initially. And you can scale it out pretty far and you can get tremendous amounts of East-West traffic, which makes all kinds of problems kind of go away. So another awesome feature of this architecture is the ability to use of using OSPF or BGP on both sides. And then we can do ECMP so that we get a type of HA. Basically, a zone node can go round Robin and try each of the links using ECMP. And then whatever works, it uses that. How many people are familiar with ECMP and our network folks equal cost multipathing? Oh, that's fantastic. That's awesome. So just to clarify just a little bit on what Abhishek was talking about, we use ECMP obviously between the spine and leaf so that we can have n number of links between the spine and the top rack switches. But we also do some other clever things with it where we have physical servers or virtual machines participating in the OSPF routing fabric and basically being nodes on that. And that's one of the ways that we get load balancing for like our API endpoints. Rather than running HA, we run load balancing and we actually have each of the API endpoints participate in OSPF. A little bit more complicated than that, but basically participate in OSPF so that the switches are doing the load balancing across the API endpoints. That also removes any overhead of using static routes and leads to a fewer number of problems in the network, debugging problems. So given that background, let's look at the physical topology the network has. So this shows schematic diagram. At the top, we have the data center switches, whatever the data center provides us, which comes down to an edge network which forms the edge of the cloud, the customer cloud. There is an optional NAT service that I'm going to describe in some time that sits in between the edge network and the core network. And then we have the core network which forms the spline network. That comes down to the leaves, basically, which are the zone racks or the resource racks, all of the racks, which actually hosts the zone nodes, the compute nodes, the storage nodes, all of the nodes. So let's look at what are the software components in the whole network architecture and how they work. We obviously have NOVA network because our network plug-in runs as a plug-in in NOVA network. That forms the network manager. And the network manager is distributed and synchronized across all the zone nodes. We use MySQL and Galera for HA on my top of MySQL. That helps us to achieve seamless load balancing across all the multiple databases, multiple zone nodes. And whenever a zone node goes away, we can always have a multi-master failover automatically. We don't have any downtime for failovers. And this also means we can run as many zone nodes as we want to. And when we want to add a new zone node, we can just plug it in and configure MySQL to it. And we are up and running in no time. So just a little background. And we built our system a little bit differently than some of the other folks who built products were on OpenStack. When Obeshack is talking about his own node, he's referring to the control plane. So the control plane for our system is scale out. But it's centralized into a pair of racks, typically, so that we can do security enforcement around it. Because the security policy you want to enforce on the availability zone control system is different than the resource nodes. So we don't run a hyper-converged system where all software is running in all systems. And it's actually different. So it's scale out, but it's on a set of fixed hardware and network infrastructure that's designed for the control plane, which is separate from the data plane. The control plane runs on a set of zone nodes, as many as you want. Sure. So the other component of it is the virtual interface driver that sits on the compute nodes. And it's actually responsible for provisioning the network for the VM. It creates the network interfaces for the virtual network interfaces for the VMs, assign the slash 30 addresses to the VMs. And it also provides DHCP service for the VMs. And Nova Network actually provides the EC2 style metadata service for the VMs. OK, it also configures routing, obviously. So we have an optional NATR component that can sit between the Edge network and the Core network to provide floating IP services for the VMs. And we have a custom utility that we call L3 AdNet that is used to pin slash 30 addresses at slash 30 networks to each of the compute hosts so that the VMs in those compute hosts can use those slash 30 networks. How does the NATR work? The NATR is actually very simple.  Sure. I just want to clarify another thing. So it may not be immediately obvious. When we put down a rack, that rack has a network routed to it statically. And then that network is chopped into subnets, and each of those are statically routed to the hypervisors in the rack. And that allows us to have a relatively stable network that doesn't have a lot of things moving around, keeps OSPF very simple because we can do route aggregation at the top of rack switch, obviously. And it also is what allows us to have a very clean layout. You know which network block is assigned to every hypervisor system. And then out of that, we carve out slash 30 networks for each of the VMs that are running there. And then that allows us to actually know deterministically on a given hypervisor which IP is mapped to which virtual machines. And then to derive the MAC addresses from the IP address. So it's kind of a relatively static hierarchical layout. Like you would expect kind of ISP days with dial-up modems or even broadband DSL type kind of system. So how does the NATR work? The NATR is very simple. The NATR sits between the Edge network and the Core network. And it actually has a config file which has Nova's credentials for accessing the database and so that the NATR can actually access the Nova database. The NATR keeps on polling the Nova database for new floating IP and private IP pairs. And whenever a cloud operator goes and assigns a floating IP to a VM, that goes into the database. The NATR sees that new entry. And the NATR knows that the entry has not been processed yet. NATR then uses DC rules to install one-to-one NAT rules on its outgoing interface. And so that whenever a packet goes out, it can be written. And when it comes back, it can be written again and goes back to the VM it wanted to go to. This actually shows inside of the NATR in three possible cases. The first case is when a traffic from a VM tries to go to the internet. When it goes out, its source destination IP and the port are rewritten. When it comes back, it's rewritten again so that the packet goes back to the VM. The second case is VM to VM NAT. It doesn't really need to go all the way out, out of the Edge network. So it goes to the Edge network and comes back again. The third case is cell-to-cell VM NAT. And in this case, also, it comes back again from the Edge network. So the key with the NATR layer is that we designed it to be between the core and the edge so that any traffic inside the cloud basically doesn't go through it. It's when you're egressing the cloud that you actually transit through the NATR layer. And then a typical NAT system that you would see in a load bouncer or a firewall is sort of stateful. And you have to do synchronization. But we did the same trick where we're basically using ECMP to load bounce across in NAT devices. And those NAT devices basically have the same configuration because they're pulling the NOVA database, pulling it out, and basically setting up the same rules on every system. So they don't need to be synchronized or synchronized through the NOVA database. And so we can run any of those at scale. And when we tested it, we get about five gigs a second out of each Linux box for 50,000 one-to-one NATs. And we haven't really done any performance optimizations on it. So you run 10 of those, and you can do 50 gigs a second out of the front end of the cloud for NAT, which is very good, considering you're using very cheap and expensive 5K Linux boxes. And the NATR piggybacks on the zone nodes for maintaining the database state, which is again HA. So ultimately, the NATR is totally stateless. The other side is the WIFT driver and the NOVA network working with the NOVA network manager to build the VMs and the networks for the VMs. So whenever an administrator launches a VM, it gets scheduled to a particular host. And the WIFT driver in that host gets called, and it builds the Linux bridge for that VM. The network manager gets which host the VM went to. The VM got scheduled to, and lists all the available networks on that host. It finds the first unused IP that can be used for the VM and calls the WIFT driver again. The WIFT driver then goes and creates a virtual interface for the VM to be used by the VM and the bridge, a Linux bridge. It then sets up a UDHCPD process for that VM so that the VM can get DHCP services. The Mac to be used by the VM is calculated based on that IP address, and DHCP uses it. Then the virtual interface driver creates a slash 13 network for the VM and assigns an IP to the bridge and one IP to the VM, and then starts up the DHCP process. It then adds routes so that the VM can talk to the gateway and adds, we actually have two interesting features. The feature, the inability to block a particular set of networks, and then the ability to whitelist particular hosts in those networks. These are enabled by adding IP table rules on the compute nodes. What that's used for is to protect the control plane from the data plane. I mean, you need multi-layered defense, right? You don't want to depend on any one mechanism, but we want to make sure that from the hypervisor layer, we're blocking any packets that try to get out to the actual control plane in the system. And because it's all fully automated, when you set up the control plane, the IP address ranges that are in there are basically saved in a centralized metadata store. And then when we do the automated provisioning of a host, the hypervisor node, those whitelists and block networks are just automatically added. So you don't really have to manually manage that. So the dual is the event where the VM is removed from the compute host. The drive goes and stops all the DHCP processes. It cleans up all the config files. And the network manager then deletes all the IPs associated with the virtual interfaces and cleans up the Linux bridges in the host. So the L3 AdNet utility that is used by cloud operators to pin networks to hosts, it takes in a cider, a host, IP, or host name, and a pair of DNS addresses to be used. It loops over the input cider and breaks it into its slash 30 chunks. And then loops over each of those chunks and uses the lower managed API to create an entry into the database. So that gets used by the scheduler because typically in OpenStack you've got a sort of a global pool of IP addresses when you're doing dynamic allocation. And the assumption is that you just cherry pick out of that pool for all VMs everywhere.  And when the scheduler picks a node that the VM is supposed to be allocated on, it's got the smaller pool of slash 30s that basically pick the VM out of whatever is not used and allocate that IP address to that virtual machine. Let's look at the challenges that we faced while writing the Custom Nova Network Driver. So OpenStack does two releases per year. It used to do one. So that has always been a moving target for us. More often than not, the plugin interface has changed, the library dependency has changed. That brings in the extra overhead of making sure that the thing works and all the regression tests and testing it extensively. Initially when we started out, the database API wasn't rich enough. The SQL Alchemy API wasn't rich enough. So we had to actually write straight SQL queries in the matter. Well, Nova Network is supposed to be deprecated in favor of Neutron long back, but that never actually happened. So we haven't really thought of moving the whole network architecture to Neutron. So let's see where that takes us. The most common question that we always get asked is, why not Neutron? And why not Neutron right now? Primarily because when we started during Diablo, Neutron wasn't around. So there's no question of using Neutron. We don't really think Neutron is stable right now for use in large clouds. It has a few problems around API changes and interfaces not being very stable. The command line interfaces and the procedures are not very intuitive and not very operator friendly. And then not all Neutron plugins are equally usable and equally good around the network architecture. The other thing I'd add there is it, it still feels like Neutron is being used to try to be the network as a service cloud system. But I mean, does anybody really believe that we're gonna like model layer two through layer seven semantics and everything that the network industry's done for the last 30 years? Load balancers, caching appliances, wide area, acceleration products, firewalls. Like, are we gonna build all of that into a single unified API? Like I just, I can't wrap my head around that. It seems like a boondoggle. Like, that's one of the things here that I think has to get sort of a rethought around Neutron. We need to keep the scope down a little bit tighter and then maybe have a set of services that provide us different API models for firewalls and load balancers and sort of a higher order network services above layer two, layer three. So our problems with Neutron brought us to like trying out different plugins and then we ran into OpenContrail. That actually proved to be a very useful thing for us. OpenContrail met all of our requirements. Primarily because OpenContrail uses all commodity technology and nothing custom on top of it. It's just layer three with MPLS, GRE, and I don't know, XMPP, XMPP, some BGP, all of these. So our network underlay, that network architecture forms a perfect overlay for, perfect underlay for OpenContrail to run on since it's pure L3 and it can just work on top of it. So right now we have two network modes in OpenStack or in OCS. One is the default L3 networking mode and another is VPC with OpenContrail. Generally customers who want to have seamless scalability for its tenants who don't want to use VLANs, they will offer the VPC specific deployment with OCS. Yeah, so sometimes I get asked why did you choose OpenContrail and it's kind of a long story but the short answer is that I engaged with people like Nasirah in 2008 while I was at GoGrid around SDN and OpenFlow and it just felt a little experimental and I know there's a lot of network people in here and we always get nervous about putting kind of experimental stuff in the network. At least I certainly do. And you know, Contrail at the time, now OpenContrail was interesting because it was basically well-known protocols, MPLS, BGP and all that good stuff and it wasn't too experimental. I mean it was reconfigured in a different way and we're doing L3 all the way down the host and doing L3 over L3 encapsulation. Those are a little bit new but I felt like that was a lot less experimental and a lot less proprietary because a lot of the OpenFlow vendors were actually putting extensions onto the OpenFlow protocol to kind of get it to where they needed to be. And so that's kind of why we started there. We're not married to OpenContrail but it felt like the least experimental approach with a lot of the right kind of architecture with scale well. Same could be said of Nuage which also takes kind of a similar kind of BGP type based approach for scaling. The second ties down to our belief that the internet scales awesome level so we should just model the internet and use something like BGP that can aggregate routes up. So we also get asked what's the packet path for our L3 model and the VPC model and let's look at that. In our L3 model when a VM tries to communicate with a VM in the same network, it goes up to the spline and then comes back. It does not go up to the edge network at all. But when a VM tries to communicate with the internet, it obviously goes up to the edge router and then goes out to the internet. Right, so here we're getting the advantage of using that ECMP mechanism between the top of rack and the spine for that sort of scale out model. So you can design a network where you've got racks of compute that maybe have 20 gigs out of the top and are designed for sort of web server instances and then you can have other racks which maybe are running block storage which have 160 gigs out of the top of the rack. And so you can sort of actually vary the bandwidth to the spine and then maximize that east-west traffic as you might say, okay, for every block storage rack, that's gonna support somewhere in the order of four to 10 compute racks and you can appropriately set up the network bandwidth to sort of meet that need and get that load bouncing between the layers because you're just doing direct routing straight from the hypervisor node to whatever the other endpoint is inside the cloud. Also notice that this does not use encapsulation in any way, reducing the overhead of encapsulation. In contrast, open control does use encapsulation using GRE, that forms pretty nice L3 over L3. Open control layer three over our underlay network which is again layer three. So every traffic that comes out of the virtual router running on the compute nodes is encapsulated using GRE and so when a node wants to talk to a node in the same network, it goes through the spline router and goes back to the node. When a node wants to talk to the outside internet using a floating IP, it gets decapsulated at the edge router and goes up to the internet. So in this case, the edge router has to be a device which can actually read and write GRE. Future directions. So we know that we have to probably move to neutron at some point of time when over network is totally deprecated. We don't really know when that is and let's see what happens with that. We are looking forward to adding more SDN type capabilities using OpenContrail to the OCS product and we are most interested in the service training thing that they do. You can place arbitrary services in the network using the service training feature. That's pretty awesome and that's very useful for NFV type services for service carriers, for example. And that's pretty much what we have for today. That was Abhishek's very first presentation ever. Excellent. I didn't even have to ask who's got questions. Just write up to him, Mike. I've got a couple questions. Number one, so just trying to dive deeper into the model, it sounds like you're still running your DNS mask IP tables. No DNS masks. OK. So what service are you running for? I'm just scaling out DICP. It's UDICP from Busybox, UDICPD. And are you still using IP tables on the compute node? We are using IP tables on compute node. It scales well enough. Yeah. But you're just not running NAT on those IP tables since you're running it on that NATR, right? And is that basically what the NATR is running? Is IP tables just with NATR? It's using IP route 2, which is in the Linux kernel and it uses TC to configure the IP route 2 in the kernel. So basically, think of it as a very fast way to rewrite packet headers as the packets go in and out of the external network interface. And we just have a one-to-one mapping of what the NAT is. And basically, as it goes out, the packet gets rewritten. Last question I have is you brought up the point around some of the limitations or challenges Neutron has. You helped me understand why this work wasn't, or why didn't you focus your effort on bringing this work into, I mean, because it's really good.  Totally bulletproof. Yeah. So help me understand why you didn't work with the Neutron project and get this stuff in there as opposed to doing it in Nova Network. And the reason I asked that is just I feel like from a user and just a general OpenStack perspective, it causes confusion within the user community of, OK, maybe we shouldn't go to Neutron. You know what I'm saying? Especially when you say the roadmap where it's like, hey, we're going to have to move this over to Neutron. Totally. So first of all, Neutron or Quantum at the time was not even an incubation when we built this stuff. And how it sort of happened is we had a bunch of people who had worked at ISPs and carriers. We were doing stuff, building clouds before OpenStack even launched and existed. And I went to Chris Brown, who was the lead developer for EC2. I said, how'd you guys do it? And he basically told me. And I was like, that is freaking smart. And I said, we're just going to do that. And so we started down the path but weren't complete when we did the big cloud stack deployment for CREA Telecom in 2010. And then when OpenStack came out, we immediately saw that it made sense to build it in there. And that's where we started doing it. There's just no option to go into Quantum or Neutron, right? And then as things have evolved, it's just been challenging to figure out how to get it back into the community for a couple of reasons. One is, I'm a small startup. I'm resource constrained. Two is that this is a network-centric approach, unlike some of the default OpenStack networking models, which kind of were made by systems guys who don't really understand networking.  I mean, that's just a little challenging for people to get. Now, we're open to pushing it back in. If I could get some help, we'd do that. And we're open to getting it in Neutron. We're not trying to hold it back. It's just sort of a resource and sophistication kind of problem. Yeah, I mean, have you filed any blueprints and working with a Neutron group, getting any of their feedback yet? No. No. Give me a couple of engineers full time for about three months, and I'll get that done for you. Well, I mean, even if you just file the blueprints and then socialize it, I mean, that's part of what the summit is for is to get people behind it, right? Because, again, I mean, it's good work. And I think people will get that done. I think that's fair. So you're signaling us for time? Right. Four minutes? All right. This is from Keshav from HP. Is there any data you try to do on L2 with the ISS and running with Trill? Because you're saying you don't like L2 and you only want to use L3. Is there any data available using the same thing with ISS on L2? Using ISSS instead of OSPF on L2 itself? No, but it shouldn't be any reason that ISS wouldn't work. I mean, I assume you're not asking about Trill, but just ISS, right? Right. Yeah, I mean, why not use ISS? It's perfectly legitimate. We just like OSPF. One of the reasons that we really like it is it's got a really good link state detection failure on the interfaces for ECMP. And it converges really fast if you've got a relatively small number of networks and you do aggressive route aggregation throughout the network. It's pretty bulletproof and we know it well. It probably doesn't scale as well as ISS, ultimately. But for any size, style, cloud, and enterprise is going to build, it's going to be fine. You can jump in here. It's a great model, I think. It's good to know a different model of provisioning through network. No one network, but I think, again, going back to the previous point, integrating with Neutron would be awesome as well. So one question I had was, so you use slash 30, you use per VM, actually. So that's consuming more addresses than just one. So is it V4, V6, and? It's V4. This is probably the weakest part of this. And again, it's been a resource issue. What you could do is instead of allocating, instead of subnetting out the network after you route it to the hypervisor, because the hypervisor isn't consuming any of those IP addresses directly, you could just have that entire network on the hypervisor and you could do network enforcement so that none of the VMs on the node would see a slash 26 or whatever was appropriate, but they wouldn't be able to see any of their neighbors. They'd just be able to see their gateway. So you're actually doing the partitioning of the switch fabric, basically, at the switch levels. That makes sense. I mean, we could make it more efficient just having, due to time constraints. And IPv4 is challenging because you're running out of space. Trust me, I got this gigantic enterprise right now and they literally can't give me any RC 1918 space. They're stealing public ranges that aren't routed in order to get around it. So it's definitely a challenge. We need to fix that and address it. And it would be something where we'd get the advantage if we could push it back out to the community of having other people help us fix that. Because it's not that much code. It's just one of those things where it falls off the priority list. Right, and the model of deployment matches pretty much how V6 deployment would happen with each rack having their own subnets and further dividing it to host. I mean, to be just totally honest, we were kind of shocked that people had this level of interest and there were 300 and some odd people there were going to come to this presentation. I didn't actually realize that people would be that interested in how we solved these network problems. It was a little bit of a surprise to me. So I'll take that as feedback. One question my friend had here was overlapping IP addresses. How do you handle that? We don't, not with the layer three stuff. That's what VPC is for. How are we supporting multi-tenancy? If you want multi-tenancy and you want to do basically. Yeah, overlapping is not supported then. Well, so multi-tenancy can be a number of things. They're signaling me to get out but you guys still ask me questions so I'm gonna keep going. And they can cut my mic, I'm loud. But multi-tenancy can mean kind of like how do I enforce security between two tenants? So like Amazon does that with security groups which are supported in OpenStack which we fully support for layer three so you would use that for network partitioning. But if you want to actually do isolation, full isolation of the IP, then you need to use virtual private cloud, our SDN based product uses ConTrail as an option on top of it. And then you get encapsulation in virtual networks and tunneling in classic enterprise networking model. Okay, thank you very much.


Resertification of all public assistance program PAP clients will be conducted during the period March 27th to May 31, 2023. During this time, a team of enumerators and social workers from the welfare division of the Ministry of Equity, Social Justice and Empowerment will be out in the field conducting home visits. After the resertification process is completed, cards will be administered to those who are eligible. Only note that new applications will not be facilitated during the review period.


Hello and welcome to the session. I am Deepika here. Let's discuss a question which says if each element of a second order determinant is either 0 or 1, what is the probability that the value of the determinant is positive? Assume that the individual entries of the determinant are chosen independently, each value being assumed with probability 1 over 2. So, let's start the solution. According to the given question, each element of a second order determinant is either 0 or 1. Now the determinant of order 2 by 2 is a11, a12, a21, a22 and this is equal to a11 into a22 minus a21 into a12. Each element of this determinant is either 0 or 1. Now the number of determinants that can be formed by using the digits 0 or 1 is equal to 2 raised to power 4. This is equal to 16. Now we have to find what is the probability that the value of the determinant is positive? So, the value of the determinant is positive a11 into a22 minus a21 into a12 is greater than 0. That is a11 into a22 is greater than a21 into a12 since each element of the given determinant is either 0 or 1. So, this is possible only if a11 is equal to a22 is equal to 1. One of the elements or both the elements a21, a12 are 0. Hence, the all possible determinants are 1100. That is here a11 is equal to a22 is equal to 1 and both the elements a21 and a12 are 0. Similarly, we have 11 and there in this determinant a21 is 1 and a12 is 0. Again, we can have the determinant of the form 1101. Hence, the probability that the value of the determinant is positive is equal to 3 over 16. So, the answer for the above question is 3 over 16. So, this completes our session. I hope the solution is clear to you. Bye and have a nice day.


My name is Jasmine, it's not working again. I run the community and the developer relations team at Aluxio. I did have my email there, but in case you need it, it's jasmineataluxio.com. And here's my partner Lou, who has been the engineer who's working on this project. She's a machine engineering engineer as well as Aluxio open source PMC member. Her email is louataluxio.com. We did have these slides uploaded, so if you guys need to go visit them later, you can download it. So I did have agenda, so we're gonna walk you through that. We're gonna start first to give you an introduction of what Aluxio open source project is in a rough detail. And then elaborate a little bit on what Aluxio is as accessing layer for analytics and AI. And then talk about Dora. She's gonna go on Dora, which is our next generation architecture for AI and machine learning. And then we will show you sort of a Dora's integration with AI and machine learning, different cases, dive a little bit deeper into the technical side and then provide some sort of our projection on the upcoming and future work. So we're good for now. A big picture of Aluxio open source is about. So we are an open source project that started at UC Berkeley's M-PLAB in 2014. Back then we were called the TAC Young Nexus Project, for some of you who've been in data science and data engineering. Great, it's on. So for some of you who didn't know? Oh, I thought it was on my bed. It's fine, we start from there. So for those of you who are not familiar with the M-PLAB, that was also the lab that gave birth to SMARC, HEMISOS and all in the array. So we're just anti-scaled at the company. So it's been a pretty productive lab. And back then there was a PhD student who started that project called TAC Young Nexus. And then slowly building to this big company and a project, when we changed the company, commercialized company, we did have to change name due to some copyright issues. Now as a project itself, we have more than 1,200 contributors and still growing. On the Slack channel, we have more than 11,000 people active. We are a Java based open source project. We're named one of the top 10 critical Java based open source project by OpenSSF. We're also GitHub's top 100 most available repo. I did put our Slack channel there. So for those of you who are interested in checking out this open source project and discussing with all the other contributors or users, feel free to find us there. I'm there all the time. So get on there, send you a message. Introduce myself again. Don't hesitate to say hi. All right, what is Alexio? To give an idea before she dives in.  It's the Spark, Presto, Trino, SQL engines, PyTorch, TensorFlow. And then on the bottom layer, where you see here are all the storages you may have whether it be local or on cloud, on-prem. And Alexio sits in the middle. So we are a sort of make it a virtualized one layer, a new layer between the compute and then the storage. And our layer provides a virtualization across all of the data sources, regardless of where your data source come from. And we serve the data to those applications on top layer. So by sitting in the middle, this application, our solution is applicable across all environments. Like I said, whether it be bare metal or it be on-prem, on-cloud, hybrid, multi-cloud or as a mix. Where we're best at is analytics and AI. So that's where we highlighted there. So that's what this data platform does, Alexio. All right. I give you a list of sort of the snapshot of a different kind of industry and companies that are currently using Alexio just to give you an idea on internet companies. So those are a mix of both CE and EE version users. What is CE is community addition. Those are free. That's the open source that we offer. Of course, as a company, we offer a EE version enterprise. So we are enterprise software, B2B business. There is feature difference, support difference, SLA and et cetera, between the CE and EE version. So this is a mix of both our community users and our commercial buyers. So on internet companies, so to give you an example, Uber and Meta, they use Alexio local cache that helped them to improve their performance on the SQL engine, especially on Presto. RaptorX is one of the projects we collaborated with Meta a few years ago before it was still in the name as a Facebook. And then A improves their Presto SQL with 10 times faster. And then Uber also deployed Alexio local cache in large scale where we actually see, I don't remember when I put a link of the Uber engineer block there, but if you're interested in looking into that to speed up your SQL engine, you can look into Alexio local cache and all of those engineer block would jump out. Another case will be Expedia. They actually use it in a slightly different way. They deploy implement Alexio tools, federated cross region data lakes in AWS. So that way they don't need to worry about data migration when they're building their machine learning data lake, data pipeline on top of their data lake infrastructure. All right, so we go a little bit on sort of the architecture diagram of Alexio as an access layer. So I'm gonna draw your attention to the left side where it says, if you imagine it's only on this side the offline data platform and data warehouse. So we imagine that your training is co-located with a storage system where you're not decoupled. Then in this case what Alexio is good for is we can help you to speed up your storage system data access to a high-concurrent retraining workloads to increase your GPU utilization rate and to reduce your storage system pressure and access cost. Now if we're looking at the right to lower left one where I point a training data to the offline training platform to just give you an example. So the training inference cluster when it's not co-located with your storage GPU and then with your storage system or when your GPU is limited because your GPU is a limited resource, right? So most likely your data warehouse and your training platforms are not co-located. Then the data locality becomes an issue. So Alexio is actually best at resolving your data locality issue. So before you deploy Alexio on the lower left corner as an engineer you won't need to manually migrate as needed to copy to a local dist or copy from the one HDFS cluster to another HDFS cluster. But after you deploy in Alexio if you look on a lower left one you really only Alexio can access and then cache and then manage your data and demand without the need for your manual migration. So basically your offline training and inference can just tack into Alexio to get all those data you wanted. And then the same logic goes to from the right box to the upper left box except for it is for the training similar to the training migration that's for model migration for online inference cluster. So basically the gist of idea if you could look from the right box to the left two we help with migration and we speed up. So the idea is because most of the time your GPU resource is limited and it's most likely your data warehouse and training platform are not co-located. So I kind of went over all those problems we resolved earlier but just in case you want to take a snapshot of what we do. So compared to that data migration the problem Alexio has that accessing layer overall unified access and layer is we reduce the engineer overhead in customizing that data migration. So you eliminate your manual work. We also eliminate the process of your manually deleting those outdated data in your persistent layer. And then we also reduce the high pressure for those data nodes and then data throttling caught in flourish storage. And you also don't need to worry about the difference between your online platform on your offline platform. You basically just need to access Alexio as sort of a unified API. That's what we provide. So to on the commercial version they call it a transparent URI and you're just tapping to it. And then the problem we resolved for compared to your having that direct access so some people say what if I just have direct access? The problem is the GPU utilization. So give you an example. The ResNet 50 will get about 20% to 30% GPU utilization. Now if you use Alexio you will get about 80 to 90% GPU utilization to overcome the IO bottleneck and reduce the IO waiting for CPU and GPU and thus we increase the utilization. And we also save time by limiting those save money by those limiting those API costs. So each cost quite a bit especially in machine learning case. In those scenarios at the scale of machine learning you're talking about tens of thousands API costs. And then in each single machine learning training your API costs can easily go over a few thousands about tens of thousands. So that's compared to everyone you have. So this is where you compare accessing Alexio to accessing your cloud direct access. All right. Oh, this is to give you an example of some companies are already using Alexio. So in Microsoft case the user for training enable high GPU utilization that we talked about in cloud training. Shopee is similar case for GPU cluster without data migration. And then there is some other big tech companies so a lot of names we cannot mention because there are customers. We're seeing 250% performance gain for inference and for their online platforms. And then there's some other companies that are using Alexio for pre-processing Spark machine learning. They have the data in the cloud GPU on-prem so the Spark machine learning pre-processing go through Alexio and then load back to the on-cloud to continue their training journey. So our customers do try to customize to actually make the best use out of it. All right, I'm gonna pass the torch on to Lu who's the engineer who sort of work on Andorra project. Okay. Hey everyone. This is Josh Ming for introducing Alexio's data access layer. So I'm going to introduce Dara. So Dara is Alexio's next generation architecture for AI machine learning. So Dara is job for decentralized architecture. So like why would come with this Dara architecture? It's basically like just like AI machine learning is pushing their limit and our users are actually pushing our limit. So to fulfill their requirement we come with a next generation architecture. So for the motivations and benefits. So our users, they already use Alexio for single training like two billion files in a single training. And they tell us that, okay, I want 10 billion files in near turn and the user demand is still increasing especially in this fears competition like all the giant tech company they are competing for building the giant models like GPT4 and other things. They want to join this game and they want their model to be more accurate as possible. So they want to fit more and more data into their model. So they keep asking the infrastructure to be able to support that ability. Basically they want a limited scalability. And for the reliability part. Our users, they keep require us to provide a really reliable like data infrastructure. So one really concrete example is that our user directly ask us to provide 99.99% of availability which gives you a downtown only like 15 minutes per year. And even those top story system vendors they only provide like 99.9% which give you like 40 minutes like downtown per month that kind of SLAs. So they ask us that we as an application level we don't want to care about the data errors. So you guys handle the data errors yourself. So this gives Dora a high motivation to develop something that not have a single power failure and it's highly fault tolerant. And for performance wise like different application they have different performance requirements but basically they want the data IO not be your GPU utilization rate limitation. So that's a result like Dora implement all kinds of performance optimization from zero copy network transmission to support high concurrent rate in the training workloads. And our users also want like some fancy features for data governance. They want to use Alasio like for multi talents and also manage how the user going to use Alasio and all the resources. They want to have quota management and they all have like their specialized security model. Like many of our users they don't use the generalized security. They have their own like security model. And how can we be able to support all the different kinds of security model so that it's not limited to a certain like popular one but they have other company like customized ones. And that's the motivation for us to build a whole new architecture. So to understand like how Dora provide a high scalability and high availability I will first introduce the whole architecture. So Dora basically in the middle part is this between the application, the AI machine learning application on the left part and also the under storage on the right part. So it helps us to fetch data from under storage and catch it to places that closer to our AI machine learning applications. And all the middle part like Alasio client and Alasio cluster are the Dora architecture. And this architecture have three main major parts. The Alasio worker which is responsible for execute detail task. So it will help fetch the data from under storage and catch it locally and provide to the client. And we have the server tree which basically managing all the Alasio worker information. And we have Alasio client who gets the cluster information from cluster server tree periodically. It gets the task information from the AI machine learning application. And you have is over algorithm based on the affinity block location policy and consistent hashing algorithm. Basically it can base on the cluster information and task information to calculate, okay, which worker should execute my task? And it will directly go to that worker without contact something like master or other things. So directly calculate the worker and go to the worker and ask the information from the worker. So in this whole structure, there is no single power failure and it's highly scalable. I will talk about more details in later slides. So this architecture is supposed tens of billions of data. So it's high scalability comes from two highly scalable component. The first one is the server registry. And the service registry, remember for the architecture before that it key check of all the worker information. And the server registry is not in the critical IO path which means that when you read a data or write some data so it's not the client does not need to contact with the server registry in that part. So it will not be the IO bottleneck. But also we utilize highly scalable server service. For example, the Kubernetes ETCD which supports tens of thousands of also workers and with fine tuning, it can even support tens of thousands of a lot of workers. And on the other hand, on a lot of workers side is designed with easy addition and removal of nodes. So basically it's isolated, it's decentralized. So you can simply add more nodes when needed. And each worker nodes is support tens of millions on files. So with the two stability components together, we are able to support tens of billions of data. And on the other hand, this architecture does not come with single power failure. For server registry, we are using the highly available server service. So you have mechanism side, to be consensus algorithm data replication to ensure that the service is highly available and does not have for a single power failure. And on the other hand, the Alasio workers, killing any of the Alasio workers have limited impact. So we will basically like fall back to the under storage or relocate the job to other Alasio workers so that even if anything like happen unexpectedly, the whole system will still functioning. And from the user side, they may experience a little delay because the catch heat ratio job a little bit, but the whole system is still functioning. The user can still fulfill their IO requests. So on the other hand, that Alasio have like extensive experience in AI machine learning industry. And we have many company users that are working closely with us. So providing us with really valuable information like data pattern and traffic pattern. So especially the read traffic pattern. So analyzing those data pattern and traffic pattern, we can come up with our performance optimization approach. So our users like for the traffic pattern, so there are some similarity between like analytics and AI wallows. Like they all have sequential streaming read. But on the other hand, for AI wallows, we experience much higher concurrency of position read. So the read wallow, we experience much higher degree of concurrency and they are much more random risk. So for example, like AI machine learning, they are much more unstructured data. So the structured data like a table, they may only want to read one column from the table. And they may distribute the read for one column to multiple different threads. So which from the storage side where we look at the traffic pattern is pretty random. So that's why we need a good support for the high concurrent position read. And also because they may be only reading one small cell from the whole table. So each read can be quite small. So for arrow file, while user provided as the traffic pattern, the arrow file read can be as small as 4KB. And based on the read patterns and all the data patterns that provided by our users, we are able to come with some optimization. The one for sequential streaming read and the other for high concurrent position read. For the streaming read part, we develop geo copy data transmission. Basically it reduce the data copies as much as possible and improve the memory efficiency so that our read can have a higher throughput. So in fact, this optimization have proved to improve the large file sequential read performance by 34% to 50%. And on the other hand, for concurrent read, we implemented the high concurrent position read throughout the whole repo without many of the smart read optimizations. And with this approach, we can solve up to 100 times read amplification issues when you are using the traditional sequential read. And on the other hand, we improved on structured file parallel read up to 9% night times and also improve structured file position read from two to 15 times. And as we talked about earlier, that by just mean that Alasio provide the catchability to speed out the data access from cloud storage to our different applications. But on the other hand, Dora enable us to connect different like machine learning AI stages together and they move data from one kind of application, one kind of data platform to the other kind of application and platforms. So from data pre-processing, feature engineering model chaining to model inference. And this, so the reason that Alasio can connect different like application together is sense to is easy to use data access API. So it's compatible with the mainstream interfaces like poses, S3 and HDFS API. So data scientists and data engineers can use their familiar ways, like your familiar like techniques, like for example, you're assessing your local file system approach or assess the data in S3 and HDFS to assess Alasio. And also Alasio support connecting to different like storage renders. So when you're changing from one storage render to the other, you also do not need to modify your AI application and training squeeze. And one thing for the Dora architecture is that you can actually use the Dora client itself as a local catch solution. It's a standalone solution. So we call it Alasio Fuse SDK. So it's the poses local catch for remote data. It allows us to assess the remote data set, the S3 data set, HDFS data set, et cetera, just as the local data set in your local file system. I also have the built-in local catch management so that you can take the data from your data source and catch it locally, manage the data in vision, only catch the hot data that needed to be used by the AI machine learning applications. And so that compared to, so it also benefit from many of the Dora optimization that I just mentioned before, the zero copy to reduce data copies and improve the throughput and also the high concurrent precision read. So compared to S3 FS Fuse, which is the most popular S3 poses solution so that you can use S3 data just as your local data. Compared to them, in millions, billions of small file changing cases, we can achieve two to 20 times IO throughput improvement and up to nine times of metadata latency reduction. And we are now also like, so we try to see like what Dora can work in different AI machine learning stages. We try always try to expand the Dora's use cases and also leverage is like architecture, scalability, availability and advanced features. And one of the cooperation that we are cooperating with a giant tech corporation to try to see whether we can use Alasio to catch large language model responses. So large language model, so you guys may not be familiar to give you an example that Choi GPT ESL is a large language model. And we are working with those giant tech cooperation to see whether we can reduce the response latency and computational causes to actually speed out the large language model responses and reduce the computational costs like reduce the GPU and also the latency. Okay, that's it for our talk. So do you want to talk about the QR code? I want, okay, yeah, so in case since we didn't even get it running earlier. So there is a QR code that is a community survey. So if you're interested in getting in touch with us, just scan that and then just fill in your name basically email. And then I do have also our Slack channel there. That's usually where we're at. And then the bottom is our GitHub repo if you're interested in checking it out. So that's where at once you've done was a QR code. I think I should also flash back to where the two of our emails are because you're interested in getting to it. All right, and then we're open for questions. Yeah, well, we'll change to the slides there. Move too fast. Anybody have any questions? Yeah. I'm curious to know about the source of the optimization comparing to S3FS fuse. You're also using a fuse layer. So the inefficiencies aren't from fuse. So I'm assuming that it's coming from cache. Is that the main source of the efficiency? The improvement? So for this part, we do many of the investigation. I do think that for there are two parts like fuse part and the caching part and the fetching part. So there are three parts that in this workload. So we are not like initially compared with S3FS fuse just that we build out this architecture and then we see whether we can do some comparison.   But for S3FS fuse, they catch it as a single file. So this gives them inefficiency, especially when you're reading large files and other things. But the reason that I didn't show the large file comparisons because usually it's bottleneck by the, your NVNE or your this performance. But for small files, we do found that they may have some inefficiency, but I'm not quite sure what's the inefficiency of S3FS fuse. Maybe for the metadata part, for the gap file status, I do find that there are a lot of like, maybe some synchronization mechanism so that it makes it much slower. But we are divided, we mainly target for chaining part, the chaining is highly concurrent. So that's why we put a lot of effort to reduce the, to improve the concurrency, to reduce the synchronization, but also maintain the current needs. Thank you. I wanted to give somebody a chance again. I've got lots of questions. So would you say that you're optimized for read-only workload or can you write results back as well to have the same data set being shared by other consumers? So your question is that whether we support write and whether we support that, you write and then immediately read. Right, so for example, you said that if the efficiency in part comes from the metadata, that you're caching metadata, then maybe the backend storage doesn't have the correct metadata immediately available for other clients. So for any storage system, especially a caching system, a big issue that we need to solve is like, under storage they have the source of truth, like your metadata, your data. And how can we be able to synchronize with them? So in the Dora architecture, one good thing is that the application can provide us with several information, but basically they can provide us the file name and also timestamp when is the file being modified. And then for our caching system, we actually will check that, okay, whether it's that copy that you want, that version of data that you want. If it's not, we will invalidate and re-catch it from the under storage system. And for our write, we will write directly to the under storage. You can optionally write to the cache, but for that part, like the caching part, it's normally like, we also write a version to it so that when you read, if the application already know the version that you're going to use, so you know the version of the data. Okay, so it's write through for write and then cached on with an invalidate. Yeah, yeah.