and that really is what it's all about. I am from Las Vegas, Nevada, extracting the signal from the noise. It's theCUBE, covering IBM Edge 2015, brought to you by IBM. Okay, welcome back everyone. We are here live in Las Vegas for IBM Edge 2015. This is theCUBE, SiliconANGLE's flagship program. We go out to the events and extract the signal from the noise. I'm John Furrier, my co-host, Stu Miniman, our next guest is Stephanie Chiaras, Director of Scale Out Power Systems. Welcome to theCUBE. Thank you, John, happy to be here.  What's going on? What's the big innovation? What's the transformational product? What's some of the examples of success as you guys are seeing? So as you can tell from the Power portfolio, we've made drastic changes since April of last year. We've changed the strategy. We've fully embraced open. That's been a big part of our shift in power. My portfolio is the Scale Out portfolio when we introduced Power 8 technology in the Scale Out portfolio in our one and two socket systems, a big shift for us, critical for cloud deployments. And today we're very pleased to be launching our Pure Power offering, which is our next generation to Pure Flex. So it's a rapid to deploy, fast time to client value, converged infrastructure offering built with Power 8 technology, leverages open APIs to allow flexibility going forward. So it's really taking our strategy and bringing converge to the next level. People like the simplicity of the converged approach as they can drop it in, but flexibility is one of the key criteria as well as software capability. What's some of the innovations on the software side? Is it the threading, is it the sockets? What are some of the sort of tech? Yeah, we're very proud of this next generation in the sense that we feel it's redefining the value proposition of converged. Clearly clients look for converge for rapid time to deployment, deploy an operating system in hours, not days. And we're fully committed to that on day one. But again, in our commitment to open, we leverage APIs that allow flexibility, building upon things like open stack, industry standard types of open, so that your flexibility is not just in what components you pick on day one and arrive to you on day one, but allows flexibility. IT is changing too fast these days. Clients need to keep up, their needs change, they need to be able to leverage that investment that they've made. So Stephanie, when I look at the converged infrastructure marketplace, virtualization is really the primary workload that we see there. Last year when Lenovo took over the X86 piece of the platform, they also got Pureflex, which surprised a lot of us because if you look at that full stack, parts of it were IBM, parts of it were Lenovo, we were trying to squint through and try to understand that. Can you help walk us through now kind of the Pureflex portfolio versus the Purepower and how IBM looks at this whole market and how does virtualization fit into that? So we, when we divested our system X business to Lenovo, as you mentioned Pureflex was split. It was a heterogeneous architecture. We are fully committed jointly with Lenovo to preserve the investment that clients have currently made in Pureflex. Clearly going forward, we are focused on bringing converged infrastructure onto our power platform. So we will, our next generation of Purepower is built upon power eight. It will support heterogeneous operating systems including Linux workloads as well as AIX workloads continuing forward, but we won't support the X86 architectures. Lenovo will be continuing their portion of a converged infrastructure going forward that X86 based architectures. And we will be loving virtualization within our portfolio because it is absolutely, as you said, it's too critical to cloud. Our first GA-1 will be supporting Power VM and we'll be virtualizing across leveraging our Power VM. So really focused on secure deployments. Okay, so we were talking right before we started the interview here. You've got scale out in your title. When I think of scale out, we really think, you know, hyperscale and you've got the soft layer acquisition. Can you talk a little bit about how, you know, power, the relationship between power and soft layer and power in kind of the Purepower, you know, the solution that you've got. Yeah, so we, as we move forward, the announcement has been made that Power will move into the soft layer portfolio. So we will be deploying that in second quarter, so bringing all the capabilities of power into the soft layer portfolio. As we move forward, there's been a lot of discussion about hybrid cloud and how best for clients to choose how they deploy their systems of record, their system of engagement. And as we move forward on hybrid, whether or not that's on-premise or whether or not you leverage it off-premise. So take data. We clearly believe that data is how businesses are going to differentiate. Some of that data you want to keep in your house. You want to keep it on-premise and Purepower is a great way to deploy that in-house. If you want to then leverage your systems of engagement, you can then link to the power architecture, keeping that same architecture across and leverage your Linux-based systems of engagement on soft layer. So it's the ability to bring that, what's on-premise, what's off-premise, leverage a hybrid cloud deployment, keeping your systems of record at home. We clearly support AIX. We have a statement of direction for flexibility to support IBMI that's announced as well. And then you can link into your Linux workloads as well, be it on-premise or on software. So I've got to ask you, what are the top conversations that you have with customers? We could like, we're all into joining the conversation. We're having a conversation. What conversations are you in, top three with customers when you talk about the powers, the scale out power? So clearly our commitment to Linux is one. We have fully embraced Linux since April of last year. We now have, not only in our technology, I came from the chip design team in power. So we're very proud of the capabilities we have. Industry first, leveraging both Big Indian and Little Indian capabilities for Linux. So prior where we had really been focused on the segment of the Linux ecosystem that was Big Indian, a large portion of the Linux ecosystem, a huge portion of it. We're playing Big Indian, I didn't even repeat all I might not know. I will. So Big Indian and Little Indian separates how you store your data. It's really as simple as the order of your bits. Do you support the biggest bit first in your eight that you send out to your data or do you store it last? And it's critical. Switching between the Indianness is not simple. So if an application is written in one, you store your data that way and you run your application that way. But now power eight supports in the hardware by Indian support. So whether or not you have an application that runs Big Indian or Little Indian, it can run on power eight. And what this does for us is makes us very relevant in that Little Indian ecosystem. So now if you have a application that runs on Little Indian today, that can migrate over to power and you can leverage the capabilities of power eight technology. Not a lot of rewriting. So from the perspective of the software, it doesn't matter. That's right. Whether or not it's a scripting language, a scripting language will lift and move over if it's a CC++, it will require a recompile. But we've seen great ability to be able to do a recompile and move it over to power. So we can migrate it over and then we can optimize. There's just our commitment to Linux is different. Linux and what are the other two? And so I want to extend the Linux to open. Our commitment to open is really what I view as revolutionary. We've now made power the most open platform, server platform in the industry. Our open power ecosystem, which I'm sure you all know, are well aware of that we launched back. We're over 120 members strong now. It's not even two years old. Not even two years old. Not even two years old. We had a really good discussion with Ken this morning, actually. Oh, good, good. On the open tower, talk about Google and Rackspace and everything there. I'm actually going to be the open stack summit next week in Vancouver. Can you maybe help connect the dots for our audience as to how this solution fits with open stack because still putting open stack and converged is a little bit new. It is a bit new. And it comes back to that redefining the value proposition of converged. It's on day one, given the flexibility of component choice and ease of rapid deployment, but continuing that choice going forward as your needs change. So we'll be leveraging Power VC for your virtualization control and that leverages open stack APIs. Our hardware monitoring will be done leveraging Nagios, which is also open. So really it allows a client to take what's deployed on day one and then extend it and can leverage those APIs going forward. So Stephanie, one of the big challenges in open is how do you control how you contribute to all of these things? How do you manage the outcomes? We've talked to Angel Diaz quite a bit about the global IBM initiatives around some of that, but speak to your group. Nagios, I heard last year at the Open Networking Summit was an interesting one. How do you decide there's so many different pieces out there where IBM helps contribute and what your group's involved with? And Open is not just a buzzword now. It's really a new mode of development. That's really what it is. It's allowing innovators to contribute at all different levels and as you said, how you encourage them to feed back into that is critical. So we work very closely with our Linux Technology Center. We have a team working on Nagios and it's equally important that they deliver value to our portfolio but deliver back and contribute to that ecosystem in order to feed that forward. There's also a trend I want to get your comments on where you used to have to be a real specialist to have to chip work and or do hardware. Now the trend is software is the majority of the budget for engineering but hardware's still growing. There's a lot more commodity, I don't want to say commodity or building a hardware. There's still hardware solutions out there but you don't have to have like the engineering excellence. Is this a trend that power takes advantage of it? Is it the hacker culture now? Well, I think it really feeds into that discussion of open as well. So years back we had that whole optimized solutions right where you had hardware that was workload optimized systems. And the challenge with that is you went after a pinpoint workload and you built this entire hardware in order for that one workload and boy you had to pick right. With open power what we've done is put in some baseline technology such as CAPI which allows accelerator attachments to be linked directly into the processor and leverage data in a coherent fashion. Very different. So now you will leverage that CAPI linkage and you can have all the open power partners build accelerators for any workload so a community, not just IBM but a community of innovators can bring accelerators to the platform. So hardware optimization is hard to beat from a performance level. But boy it's a huge investment to make that happen as you mentioned. Well it's hard to build but if you can open it up where there's a platform that's enabling other people to plug in. I mean we love all the work going on and with Facebook and on Microsoft as you're donating all this stuff to open compute for instance. That's right. I mean open compute is like the home brew club for servers and data centers. It's like people are really hacking with the hardware. It is. And you know we've really been able to open up, invite the innovators to participate. It's about moving quickly and leveraging innovations quickly into the platform. And hardware has always had a hard time doing that just because as you said the life cycle is very long. It's very arduous to build a new processor or things like that. But now we've put in some technologies and capabilities that allow a huge community. It allows someone as an entrepreneur who can do some engineering. That's right. Get a use case up and running. Get some traction. Maybe some transformational impact. And then get someone to build that scale in Asia. So one of the things we're focused on is tools. So if you look at the open power ecosystem you have companies that do small accelerator work for their particular application but our team is very focused on providing them tools. So it's easy for them to build up those accelerators. Our partners with Xilinx and Altera and their tool set. And how do we enable them to do that quickly and easily? Yeah and it's a great opportunity too because the old days you'd have to build a factory do the chip designers. You'd be working months and months. Wait a minute that workload's out of business or the market shifted. Or you picked the wrong one. But that's absolutely right. I mean that was the old way, right? All that testing. Absolutely. And now it's all about differentiation. And how do you provide differentiation in a way that's adaptable and easy to change so that you can keep up with the changing IT world? Yeah and I think cost gets driven down. Values increases. And then the converged infrastructure component of it is packaging, wouldn't you say? So that's kind of what you guys do on the system side there. It is. It's about packaging and allowing infrastructure management that allows clients ease of management, lowers your operational costs. It's really focused around that level of infrastructure management. And then you can take things. It will fit into our pure systems portfolio. There's a statement of direction from that pure application team. And pure application really brings it to another level to an application platform. So while pure power is very much focused on an infrastructure platform, PureApp or pure application system is really built as an application platform. So when will the pure system stuff be in pure power? Pure application, it's actually the opposite. I'm sorry, pure systems. Yes. Capabilities. Pure power will be brought into the pure application portfolio as the baseline infrastructure underlying it. I'm getting all the skewers. But they're announcing a statement of direction for second half. Okay. And that'll be announced here at Edge as well. Stephanie, thanks for coming on theCUBE. We appreciate it. Love the energy. Love the technology you guys are doing. Give you the final word. Share with the folks who aren't here. What's going on at the event? What would you want to share them about the stuff you're working on and the customer activity here? What's the big sound bite? Yeah, one of the exciting things here, and I came from the lab now here, you're surrounded by clients. It's about hearing the change. We started this journey with power back in April of last year. And now we're really starting to see the traction around our Linux, right? There's, we have now doing Linux like we've never done Linux before. We're committed to open with our 120 open power partners as part of that foundation. It's seeing the traction and the energy come back. It's all about being relevant in the industry and power's there. I love that. Doing Linux like Linux never before was great. I'm going to get that sound bite. Stephanie, thanks for joining theCUBE. We'll be right back. Live here in Las Vegas for IBM Edge 2015. I'm John Furrier. Stu Miniman, we'll be right back.


Hall of Fame in baseball that has been postponed it is not happening. So Jeter and Walker will not be going in this year They'll be going in as part of next year's class and next year's class if you look at it The one person that seems most likely to get elected would be Kurt Schilling So that class and Jeter and Walker would go in together and there are others You know obviously at the late Nick Caffardo into the writers wings and things like that But they they will not have it. It will be next July 2021 So if you're a Yankee fan, and they were expecting the largest crowd ever in Cooperstown Well, it's gonna be July of next year buy your tickets now make your hotel hotel reservations now because it's not happening this year Kayla Greco Rosa


The SARS-CoV-2 propane-like proteins, PL-PRO, is of interest as an antiviral drug target. Here, the authors synthesize and characterize naphthalene-based inhibitors for PL-PRO and present the crystal structures of PL-PRO in its apo-state and with the bound inhibitors, which is of interest for further structure-based drug design efforts. This article was authored by Jersey Ossipiac, Sara Anazizzi, Steve Dworkin, and others.


The Ogun State government says it has recorded 15 new cases of the new Delta variant of the coronavirus within the month of July. Commissioner for Health Dr. Tommy Koka made this known in a statement issued in Abhiokuta. Dr. Koka lamented the rise in the number of cases within the month, compared with having only two cases in June. She explained that four patients are currently receiving treatments at one of the isolation centres in the state, which, according to her, was practically empty in the last one month. The commissioner who confirmed that two new deaths were equally recorded advised residents to adhere strictly to public health and social measures.


Pretty much ever since I started the channel, I've been looking for a replacement for Firefox. There are many reasons why this is true. For the last year or two or so, Firefox has been getting slower and slower and slower. It has failed to render many web pages for whatever reason Patreon won't load in it half the time for me. The banking website that I use won't load at all and has broken things when it does load. That's just two examples of several websites that just don't work well in Firefox. When you're a web browser, the one thing you have to do well is browse the web. Otherwise, you should call yourself something different. So because of these problems, I've been searching for an alternative to Firefox. I've not really found one. I've tried Brave, I've tried Qt browser, and despite really liking Qt browser, there are certain things in it that I just can't get past specifically the really bad ad blocking. And even though the ad blocking has gotten better in Qt browser, I still can't use it. Brave is just kind of horrible in my opinion.   So my quest has been going for almost a year at this point and I really do want to find something that is better than Firefox at doing the job a web browser is supposed to do. So one browser that I was never ever going to give a true chance to was Microsoft Edge and there are many reasons behind that. Most of them having to do with the fact that it has Microsoft in the title. I'm not a big fan of Microsoft software. I never have been specifically because Windows is trash and everything they do is an offshoot of Windows.  So I decided to install the beta of Microsoft Edge and give it a try and I really like it. I really, really do like it. Now I've been using it for a couple days now and I'm going to go through a few of the features that I truly enjoy and then I'll talk a little bit about whether or not I'm going to be maintaining my use of Microsoft Edge. So let's go ahead and jump in. So this is what my install of Microsoft Edge looks like and there are a few things here that I'm just going to point out that I really, really like. So the first one is vertical tabs built in out of the box. Now I'm not one that uses vertical tabs all that often you can do it in Firefox with an extension. They're fine. They have a lot of options with that extension and stuff like that. So they're really good. But with Microsoft Edge, for whatever reason, the fact that they're built in actually make them a little bit more usable and their functionality seems to be more native than with the extension in Firefox. Another thing that I really like about Microsoft Edge is that it puts tabs to sleep and by that I mean that if you don't go to a website or one of your tabs here along the side for a certain amount of time, it actually takes them out of memory so that they're not taking up resources and puts them to sleep. Why haven't we come up with that idea before? It's fantastic and now I understand that every browser or whatever has a certain mechanism for not loading up tabs that you're not using. But most of them, if tabs have already been loaded, they're in memory and that's where they stay until you close them. With these, you don't have to close them. You still have them open. They've just gone to sleep and you can tell that they've gone to sleep by being kind of grayed out here and that's really cool. Now I'm not on a system that has low memory or low resources. So it doesn't, it's not that big of a deal but I really truly enjoy that feature. Another thing that I really like in terms of tabs and stuff like that is that you can add the tabs that you have open to a collection so that if you close all those tabs, you can actually just reopen those tabs by opening up that collection. That's really cool. You can also create groups. So if I open up another tab here and go to say it in nine to five Linux or something, whatever, I can actually create a group, add tab to group and we'll just call this Linux and we can add this tab to group, the Linux group and now I have a tab here and I can actually collapse or at least I should be able to collapse that. I may have to actually enter. Yeah, you have to have another tab open but once you have another tab open, you can collapse that group so that they don't take up so much space. So if you're a tab hoarder, which I know a lot of people are, you can actually go through and group those tabs. Now, I don't know if like Chrome has this. I haven't used Chrome in ages. So it's possible that something like Chrome has this functionality, but this is great. I love that. I'm not really a tab hoarder, but I could see myself actually using this because I have certain tabs that I have open all the time. The same thing with collections. I have certain tabs that I open up every single time I open a browser. Just the fact that I can open all those tabs with one click is kind of awesome. Now, other than that, this is basically just Chrome. It does have some neat settings. So if I go up here to the settings panel, the settings they have gone through and completely rewritten from Chrome and it looks more kind of like what the Vivaldi settings do, but it's not quite as customizable as Vivaldi, but it has some really cool stuff. So you can go through and you can choose your themes here and stuff. You can go through and change the start homepage. Basically, all the stuff you think that you can do with a regular browser. And then there's some things for certain privacies and stuff not. I probably should change that district. But I mean, at least they're trying to be a privacy-focused browser. I doubt they succeed. The thing is, I don't care all that much. Yes, I enjoy my privacy. And yes, I want privacy on the internet, but I'm more interested in having a functional browser. And so far, the thing that I like most about Edge is that it's fast. I can go through and load up YouTube and it just loads. It's very, very quick. If I open up Firefox here and then go to YouTube, you may not have noticed that, but it's a little bit slower. And over the course of the day or whatever, you just notice it's getting slower and slower and slower. The more tabs you have open, the slower it gets. It's not a great experience. I mean, that wasn't there wasn't too slow. But like I said, this is the only tab basically I have open. Normally, how I use a browser, I have multiple instances. I have some on one tag. I have some on another tag or whatever, because I'm, you know, segregating my work in my reading or whatever that I do. And the more instances of Firefox that I had open, it was always, always going slower. And that's not a great thing because I want my web browser to be fast. Now, if Firefox had only been slow, I probably would have been okay with it. Like I can put up with websites that load just a little slower than everything else, just for the fact that I want to use open source software. I want to use an open source browser.  If it was just that, I would have been fine. I would have continued to use Firefox, but there's also websites out there that just won't render. And most of them are ones that I absolutely have to have Patreon, my bank, Facebook from time to time has just broken things in it for whatever reason. Now I don't use Facebook all that often anymore.  And I haven't had any of those problems in Microsoft Edge yet. So the thing is, is like I said, I wanted to support Firefox for the longest time, and I wanted to continue to use it. But the little things that it just doesn't do well that it needs to do well have driven me away. And surprisingly enough, it's the Microsoft Edge browser that has gained my support for the moment.  For one, you can't hide the extension icons up at the top unless you have the extensions menu enabled. I just want to hide all that stuff. I want to get rid of it. I don't need the extensions icons to show up at all unless I tell them to. So that's annoying. Another annoyance is that I can't actually go through and customize the CSS like I can with Firefox. I like customizing the Chrome around it. And you can't do that with Firefox because or with Microsoft Edge rather, because it's just not what they do, right? It's not Firefox is open. The Microsoft Edge browser is not even though it's based on open technology or whatever. So yeah, I've switched to Microsoft Edge. Whether or not I stay with it or not is going to be a good question because the thing is, I feel guilty. Like I'm a Linux guy. I'm a FOS guy. And I want to support Linux and open source software. I really truly do. And I'm willing to put up with some pain points in order to do so. So I feel guilty about not using Firefox and whether or not that guilt will eventually drive me back to using Firefox. I don't know. Maybe possibly we'll see. At the moment, I'm really enjoying Microsoft Edge. And it makes me feel a little dirty to say it. So in the comments below, let me know if you've tried Microsoft Edge, whether or not you liked it or let me hear your thoughts on my switching to it, because I'm sure there are some people out there who have very high, very low opinions of Microsoft and you'll want to share those, of course. So do that in the comment section below before I go. I'd like to let you know that you can support me on Patreon at patreon.com slash Linuxcast. You can support me on Twitter at the Linuxcast. And you can find all of the contact information that you'll need to contact me in the video description below. Before I go, I'd like to take a moment to thank my current patrons, Devon Chris, East Coast Web, Gen 2 is Fun 2, Marcus, Megalyn, Donnie, Sven, Jax and Knife and Tool. I got you kind of cut off there, Jax and Knife Tool. We'll fix that right now.  Thank you for supporting me. Mitchell, Mr. Fox, Arts Center American Camp. Thanks everybody for watching. I'll see you next time.


Is Facebook gaming better than Twitch FV versus Twitch streaming? A few months ago Facebook gaming was nothing. Almost overnight. It merged with Mixer since it's shutting down and it became a pretty big deal Now I still think that YouTube and Twitch are still way more popular than Facebook gaming But is it better than YouTube or Twitch? First, I'm going to talk about watching Facebook gaming to be honest Even if it was better, I still wouldn't watch it because there aren't any big streamers here yet Now in the future, that might change, but right now it's just not happening Facebook gaming might be a good option for you if you use Facebook a lot because it's always trying to rub their new stuff into people's faces This could be good for creators, but more on that later. Twitch does let you see streamers Just not as much as how much Facebook is pushing out their streamers content It's like they invest so much into this thing and their goal is to dominate the streaming industry as a whole Even if you use Facebook, I wouldn't watch Facebook gaming myself because of the low amount of popular streamers Yes, they're playing big stars to use it But like with Mixer, it's hard only watching one or two big ones who like for example Ninja and Stroud There are some good things though Like it's pretty easy and simple to browse games on Facebook gaming You can stream, view, play, basically a ton of things here. It's more Facebook than Twitch If that makes sense. You can browse your feed and comment on other clips and streams It's kind of has that Instagram format It's definitely more social than looking at a stream for long sessions The suggested rows are also pretty cool because you can browse streamers of all different types of games And it's just another option to look for someone The main take is basically that Facebook pushes your stream to more areas Whereas Twitch only shows us when people look at a game or follows a streamer In Facebook gaming, you can just go directly to your Facebook feed and find it if you add it to there And that's just another way that Facebook and Facebook gaming connect each other Now let's talk about game streaming. Is it easy to grow? How much money do streamers make? What is it like to stream on Facebook? Based on some videos, it's pretty easy to set it up You just need to fill out some info and customize your page and stuff Since it's still a new platform, it's less saturated for new streamers Plus with Facebook pushing out this platform more and more You'll get recognized pretty easy. Now this doesn't mean you should leave Twitch and just head to FB gaming And I think it's like this Even though you're the biggest streamer on you're not going to get a lot of views For example, when Ninja got like 10k compared to 50k on Twitch It also has a creator tab where people can find new creators and you can get featured in it Another thing I noticed is the vast majority of viewers were watching mobile games For example PUBG Mobile itself had like 100k and the next most popular game had like 10k Viewers. This might be the best platform if you're a mobile streamer So that's a benefit for them. On the money side of things probably the same as Twitch If not better because I heard that ads make a lot especially if you try to grow on Facebook itself There isn't an exact answer for everyone if you use Facebook a lot chances are you'll want to stream and watch there If you don't, Twitch is probably the right for you I'm not a streamer so I don't know much But this really seems like a good platform. The only thing holding it back is its popularity compared to Twitch and also some Polishing. Oh, yeah, it's a viable streaming platform and you can definitely get famous on it And it's quite simple to use. This is for now, but in the future all of this might change Due to Facebook growing so much more and more streamers know about this platform and some might even switch to it The massive growth due to how much money Facebook is putting into this thing is just massive Even though Twitch still has like 75% of game streaming viewership Facebook still has a chance to become as successful as Twitch. This won't happen soon But it might happen in a couple of years or so What are your thoughts on this and that's having a discussion about this in the comments below


The top four contributors to fatal crashes on Minnesota roads include speed, distractions, impaired driving, and failing to buckle up. But those are not the only reasons why people get into crashes, and some here in Bemidji are looking to learn how to avoid them. It's been a while since some of these people have been back to driving school. The Driver Improvement Program gives some incentive for those 55 and up attending the course. Take an eight hour class, and then every three years take a refresher class. Then every year they will get 10% off in their car insurance. Some like Jerry Coslucher come back each year. One thing he picked up this time around was how to position your hands at the bottom of the wheel. By nine and three and ten and four. Oh, she positioned her hands and fingers, you know. I position them to take the weight off my arms. Another thing to keep in mind, be mindful of how things change as you age. For Elizabeth Bergeron, this means her eyesight. I don't really like to drive at night. I prefer just to do my things during the daytime. But it might be to her advantage. The most dangerous time to drive is at dusk. The class also helps remind the drivers about rules and regulations that seem to be constantly changing. We don't have a way in Minnesota for people to get the new law changes. Laws regarding cell phone usage are being updated as quickly as the technology itself. Driving school instructor Russ Bly says that phones can be a big distraction for teens. It's hard to get them to set that down. But that's also a problem for the adult population. In just five minutes on Paul Bunyan Drive, Lakewood News spotted a few people making calls. Not illegal, but still distracting. One woman approaching a work zone didn't appear to be making a call. If you remember, why do crashes happen? Simple mistakes, drive or air. Dawn Bly says we need to get more deliberate with our driving. Take time to think about the task at hand. I think we can also bring down some of those crashes and some of those fatalities on Minnesota Roadways. More information on how to sign up for the courses can be found on our website at lptv.org backflash news. If you've enjoyed this segment of Lakewood News, please consider making a tax-deductible contribution to Lakeland Public Television.


Quick tips. It's time. You want to take us to Dale John? Yeah, here's a good one that I verified This is one of those tips will be known by some people but me become one of the five new things for others I like it. That's the whole idea. I created a new reminder on my iPhone running iOS 15.1 I wanted to type a precise time and it was expecting to see a keypad that would let me type in this case 18 hours 22 minutes the time that the International Space Station starts to become visible overhead Instead I got the revolving wheel that had increments of five minutes. How annoying Solution press on the revolving wheel hold for a moment and then let go there was no haptic feedback But the number pad appeared enabling me to create the reminder with precise time Yep And yeah, I verified this. I don't use reminders, but going in there I you know, I got to scratch my head over why they default to five minutes and not Minute minutes probably because it would be a pain if you know if your reminder is for 9 o'clock a.m But you want it to be 9 30 scrolling through 30 iteration or 30 Notches on the wheel is much worse, right? Like so. I think that's That would be my guess is they just decided from a user experience standpoint. Most people are doing meetings They probably could have done 0 15 30 45 and hit, you know 85 percent of the use cases, but they probably said well, let's get a little more granular But we don't need to be super granular, but yeah that that hold it down thing Um, I don't know that that's documented by apple anywhere, but that it's not just reminders It's anywhere that you see that time wheel You can hold down on it and do that and I forgot I had totally forgotten about it until this quick tip came in So this is great. Good stuff


This study found that youth diagnosed with MISc during the COVID-19 pandemic exhibited a more severe phenotype than those diagnosed earlier on. The authors identified several key differences between the two groups, including higher peak ferritin and anti-probenp levels, increased prevalence of liver enzyme abnormalities, high pulbuminemia, and thrombocytopenia. Additionally, the authors noted that patients in the later stages of the pandemic had a greater need for intensive care and treatment with biological agents. This article was authored by Thomas Rensson, Nils D. Falkett, Kimberly Amador, and others.


Thank you, everyone, for coming down. And all of you, or most of you, are either OpenStack users or OpenStack system administrators or technical support engineer. Has any of you managed to install OpenStack the full stack without hitting any problems? No. OK. So most of you may have hit at least one issue, be it with Neutron, NOAA, or any other issues. But if you are experimenting with OpenStack, you have enough time to sit back, relax, and troubleshoot. But that's not the story if you go to production with OpenStack. And if you hit an issue, you have the right to sleep. So you always end up sleeping like this. But you do have your right to sleep like this forever. So that means you have to do your homework and configure your production environment properly to meet the optimum performance. So throughout this session, me and Destin is going to explain to you a couple of issues that we worked to troubleshoot. And what was the approach that we have taken to troubleshoot it, and what was the root cause, and how did we arrive into a resolution, and how we are going to help you to save a lot of sleepless nights. So myself, my name is Sadeek. And I have been working forward, had technical support for more than nine years. And recently, I'm working as a cloud success architect. The name itself explains the role to make sure that each and every OpenStack deployment is going to be successful. So this is my colleague, Destin. So I will hand it over to Destin to continue with the rest of the session. Thanks, guys, for joining us. Again, like Sadeek, I am a cloud success architect for Red Hat. Sadeek and I both work as part of the support organization. So what we want to show you today is really from the experience of our broader support organization. Not a whole lot of time and 40 minutes to run through a lot of troubleshooting. So what we chose to do in this session was to highlight a couple of the key issues that we have seen come in. We're going to show you a couple of points where you may be able to make a proactive choice to make a difference. But a lot of this is really about the process as well and how you approach these situations and what we do in our support organization to try to make this as easy as possible for you. So as our contact information, feel free to email or tweet us. So start off with kind of what a problem looks like when it comes in. Sometimes you're going to get a user complaint of one of these flavors. Sometimes you're going to get an error that comes in. To look at each one of these, they all look like they could be different in some way. And these may have come from different sources, different log files, different messages from users, different experiences that they're having. And it's really, in our support organization, we see a lot of this. We have a lot of these types of messages coming in. And we really have to filter through and figure out, OK, where do we look for the information to actually tell us what the real problem is. And you can imagine, I mean, this gets noisy. This is a lot of explanations, a lot of different kinds of wording of the problems.  If you're deploying OpenStack, nobody rose their hand that they did a flawless deployment when Sadiq asked. You don't.  And you're always going to have some crazy list of messages that are going to try to point you in different directions. So I'm going to talk about one in particular that we found fairly interesting, because it pops up reasonably often. And it turns out, as you'll see, that we've discovered that we had a rabid WQ problem, thus my rabbit kind of falling over here in his troublesome state. So again, these kinds of issues are going to manifest in so many different ways. Narrowing it down to this is an interesting task for us to take on. In this particular case, I think the problem description in the one that I'm looking at in particular here was described as compute service on the compute nodes is stuck in a state of activating. What does that mean? Where do I even begin to start looking here? So we're digging into logs. Obviously, this is one of the first places we're going to look, trying to figure out what is important in here. And one of the things that we come across in the log files are these kind of nondescriptive timeout messages. So this is going to be common as you guys are troubleshooting things. You're going to see a lot of log messages that in the beginning are mixed into a bunch of other messages and trying to figure out what the actual important information is in that log file is going to be quite difficult. So in this case, we see our operation timeout. It says it's terminating. We've got a failed state. We've got scheduling a restart. And of course, this is even worse because it's going to keep trying to do this over and over again. So we end up with these massive log files. And if you're familiar at all with the sauce report tool, this is a way for us to collect data that we look at from the support side. It gives us kind of a snapshot of what's going on. You get things like this happening, and your logs can suddenly be hundreds and hundreds of megabytes per log file. So just generating a sauce report and trying to collect all the data can be troublesome on its own. So we take this. We're looking at this situation. We're trying to dig into it. We've tried to say, OK, well, something's hung up here. We're using too much of something's timing out. What's going on? Let's try rebooting the node. In this case, rebooting the node doesn't help. We actually run right into this problem, again, pretty quickly, even after rebooting the node. So our process is going to take this a bit further. OK, where can we start looking now to figure out where the source of this issue might be? A tool we commonly use when we can figure out a process that we're looking at is going to be sTrace. So in this case, digging into the sTrace, we get of the Nova Compute service here, we're starting to find these communication problems that are specifically pointing us to RabbitMQ. Now we have a place to go. We started off with nondescript errors. We're narrowing it down. Now we've got a little bit more of a direction to follow. So you'll see here in the error messages that have come up that we've got the ampqp. It says that the server's unreachable. I've highlighted the IP addresses just because when you're looking at these long lines, it's easy to overlook stuff in the log files. So what you'll notice is that we've stopped communicating with one of the messaging servers, we're reconnecting to another messaging server, and it shows that we're connecting to that one. And again, we're getting a lot of repetitive entries of this kind of thing in the log files once we've narrowed it down. So what we end up here, and I purposefully kind of left out the details, because Sadeq has got a nice technical dive into a second issue here that I think will be more valuable to you guys. But the process is always, this piece of information leads to another, leads to another, and we finally narrow it down. Always very time consuming. Strace gives us more logs, the logs give us some hits on some existing bug reports. The bug reports give us some links to some upstream communications. Again, yada, yada, yada. But what we finally get to is that we've run out of file descriptors. And why did this happen? Why are we running out of file descriptors? Well, we used a default. And the default that we brought into the product that came from upstream in this case is 1,024 file descriptors. And that might have been a reasonable enough starting point many years ago. But we know at the kind of scale that we need to be running OpenStack at, that our customers are using it at. We're going to exhaust 1,024 file descriptors for the messaging service very, very quickly. So we know this is absolutely a limit that's going to get to us. This is actually an upstream post on GitHub. Hopefully, I don't get too beat up here for bringing this out. But Michael's one of the, I believe, one of the maintainers of the RabbitMQ. So this is actually from a PuppetLabs GitHub conversation about commit. And this is kind of nice to dig into it because you actually get his description of what kind of problems you're going to see if you do run out of file descriptors and why this is an issue. If you look through this entire conversation, the recommendation from this thread actually ends up putting it at 16,000, which is a good starting point. It's a whole lot better than the default 1,024. What we have done on the Red Hat side is considered this kind of scale, again, that our customers are wanting to get at. And our recommendation is actually to go quite a bit beyond that 16,000. And we really want to push it to the max, and we recommend a 64K setting on this. The memory overhead of making an allocation like this to set the number of file descriptors that high is really quite low, given the scale of the systems that we're typically seeing being used. So it's worth it to go ahead, take that memory allocation, set the file descriptors higher, and that way you're proactively getting ahead of this problem. The good thing is we've learned from this process we know that these RabbitMQ file descriptors need to be higher. You're going to find in the current releases of our product that the default is set to the 64K value. So you're not likely to run into this in the product or in the upstream release in newer releases. But if you've had older releases running, and maybe you're starting to push the scale on those a little bit, it's potential that you could run into this issue, but it's really more of a description of the kinds of things you could run into across the board. It may be RabbitMQ, it may be HA proxy, it may be some other service that you're going to see these kinds of issues. And if you know ahead of time where to change these values, you're going to be in a much better position. So a note on this too, if you are actually looking at this specific issue or any of these traditional U-limit issues, be careful based on your implementation where you're making this change. If you take a look at our KCS solution, which I'll talk a bit about that in just a moment, you'll see that we have some careful descriptions in there about three different ways you might have changed depending on your deployment. So real quick on what we do at Red Hat when we discover information like this. We have this process called Knowledge-Centered Support. What we're really looking to do is take all of the knowledge that we gain during that troubleshooting process every time we're working with a customer. And actually while we are in the process of working with a customer, we are bringing in that knowledge into a new solution. Those solutions are internally referenced when they're on the rough side and we're still trying to figure out what they are. We try as quickly as possible to get those published to our actual customer portal. So you as a customer, you can go in, log into the portal, do your own keyword search, and you're going to find if you do some RabbitMQ searches about file descriptors, you're going to find this article posted that's going to give you these settings that we just talked about. And the same thing happens for every issue you run into. We're always trying to build this knowledge base. We're always trying to leverage this knowledge base. It's part of our support process to touch this first before we do anything else. Build on the knowledge that we have, and that way we get to the solution much, much quicker instead of going through this long troubleshooting process every single time. Again, the link to this particular solution is down there at the bottom, if that's at all something that you're interested in taking a look at. And with that, we're going to end that issue, and I'm going to pass it on to Sadiq, and he's going to give you a nice technical dive into a separate but similar issue. Thank you, Dustin. So Dustin clearly explained that the default value for RabbitMQ file descriptors are very low, and it cannot meet the messaging requirements for various open stack services. He also explained that there are various ways, a lot of symptoms that can manifest in too many different ways, which makes troubleshooting really difficult. And good, we identified the problem, and we also have a solution that increased the RabbitMQ file descriptors, and we increased that for the customer, and everyone involved started sleeping like this. But unfortunately, that didn't last for a very long time. After some days, we had a different issue from a different customer, and the symptoms started to manifest exactly like the problem one that we had, the Dustin explained. That's random failure while spawning large number of instances in batches. So you create 100 volumes, one or two volumes are going to end up in the failed state, or you do any task, 100 or 200 tasks, one or two of them are going to end up failure. So again, initially we thought that this should be very easy to troubleshoot, because we already know that there is a problem that Dustin explained, low number of RabbitMQ file descriptors. We thought this customer might be hitting the similar issue, but unfortunately, either he has reused the knowledge-based article, and he identified those problems before coming to Red Hat and solved on his own. And when we checked the customer environment, we already saw that the RabbitMQ file descriptors are $0.65. He has already done that work before coming to Red Hat. And again, our sleeping mode got changed into this so that we have another bottleneck here that we need to troubleshoot. So again, I mean, if you are going to spawn 200 instances or 300 instances in a batch, and five or six instances end up being failed, it's very difficult to chase behind that instance, which instance got failed, get the instance ID, and go through the various logs. We don't know whether the problem is with RabbitMQ. We don't know whether the problem with Keystone. We don't know whether the problem is to attach a Cinder volume to the instance. So it's very, very difficult to chase behind those instances. So we tried to chase behind the logs to see what happened, so we didn't get any clue to solve it. Then we tried different ways to understand what are the different ways this problem could manifest. And we were likely able to run a script for NovaList or CinderList or whatever command. And when we do that in a script 200 times or 300 times, we get an error message, unable to establish a connection to Keystone admin API. So we got a clear error message here to concentrate on, and we started asking Keystone why you are sending this error message. And unfortunately, Keystone told us, I'm innocent here. I categorically deny all allegations that you are making against me. But then why are you sending this error message? He has no answer. So we tried to debug Keystone to see where this error message is coming from. We enabled debug logging. And we went through Keystone logs a lot of time. And we also inspected the number of connections that is coming to Keystone. Incidentally, we also found that a lot of Keystone connections are stuck in close weight status. And we tried to concentrate on that to understand why this many connections are stuck into close weight status. But we didn't get any clue on. But we still, at the end, we thought it's not good to just concentrate on Keystone because Keystone may not be the culprit here. So again, then we took a step back and tried to understand how the environment works, how the connections are works. So on a default deployment using Red Hat Enterprise Linux director, we are going to get three controller nodes, controller one, controller two, and controller three. And we are going to run most of the OpenStack services active-active on these three nodes. Each and every API service, we do have an HAProxy load balancer. And the three nodes also act as MariaDB Galera database service. So suppose an end user is going to run NovaList, and he's getting a Keystone error message, an error message from Keystone. Then the first thing that the user contacts Keystone and gets a token. And once the user gets a token, then he's trying to connect Nova API. When he does that, he first hits the HAProxy VIP, then HAProxy selects, assume that all the three servers are the backend nodes for HAProxy. And HAProxy is going to run on all the nodes, but the VIP should be active only on one of the nodes. So VIPs are managed by Pismaker. Pismaker will move around VIP if one of the nodes goes down. So assume that the controller node two has the VIP, then the Nova API connection first reaches the controller node two. And controller node two chooses one of the backend node, backend Nova API service. Here, let's assume it has taken the controller one. And then Nova API service then makes a connection to Keystone to make sure to do the authentication. And again, the connection to Keystone goes through the HAProxy node. And let's assume that this HAProxy node that was selected was controller two. Again, the Keystone need to contact the database services. The MariaDB Galera service was running under HAProxy here. So each and every open stack services have the VIP configured as their database server IP address. So then controller two then contacts the database services via HAProxy. Then HAProxy chooses one of the database nodes to send that database request. So this is how it works. And we were trying to understand, during this, where the packet gets lost and where the error happens. So we tried to make a lot of educated guesses. So during troubleshooting, if multiple components are involved, we have to do, we have to make a lot of guesses. And we have to weigh and give priority to those guesses, which one is highly likely to be the culprit here. And so there are a lot of options. The first one could be that there are intermittent packet loss within the environment, something network related. If this happens, if the packets are intermittently lost, then it could cause problems like this. So we did a lot of network debugging, trying to understand whether any network problems are there. We were afraid. It's not there. Then the second option is whether the connection, is it a problem between end user and NOVA? Of course not, because the error message comes from Keystone. So it's not a failure between the end user and NOVA. Then it could be a problem with NOVA, the communication between NOVA and Keystone, of course. That is one possibility. And it should be a problem between Keystone and databases. So of course, it is at this time that we have two options right now to concentrate our troubleshooting. We tried to enable HAProxy debug logging to understand if there is any problem between these two communications. And when we enabled HAProxy communications, we saw a lot of messages, a lot of connections between the Keystone and MariaDB Galera database is being terminated with CD status. CD means it's a client timeout. Within HAProxy, we configure a client timeout. And if HAProxy cannot serve the request during that client timeout, HAProxy will close the client connection. So then we don't see any HAProxy error messages for communication between NOVA and Keystone. All the messages that we see are between Keystone and database. So it's at the time we tried to debug why these many client termination messages are coming for database services. And we then enabled HAProxy HTTP statistics page where we get a nice GUI which shows all N number of connections that is coming to be served by each and every proxy within the HAProxy. And finally, we tried to take a close look at the MariaDB Galera proxy. And we found that the MariaDB Galera has hit the maximum limit, maximum connection limit. That is 2,000. So this is a different route that we never can guess. So where does this 2,000 limit comes into effect? Then we had to investigate in that direction. So this is on the right side with the green background is what is the proxy configuration for MariaDB Galera. I have not defined a maximum connection of 2,000 there. And below that is the global section for HAProxy. I have defined the maximum connection as 40,000 there. So it's not 2,000. So again, then there is a section on the right side where you see a default value default for each proxy. Then there also I have in define a maximum connection 2,000. So let me just explain that the global section has a maximum connection that is the maximum number of connections all the proxies together can make. It's not about a single proxy. Only a single proxy has a limit of 2,000. Where this limit comes into picture is on the default section there is an invisible default 2,000 limit. So HAProxy basically assumes that this is the maximum limit each proxy is expected to make. This is hard coded in the code, the default limit. Either you need to change that value in the default section or you need to change it on a per proxy section. So this is not known to anyone. This is not clearly documented anywhere. And we had to clearly go to the code and understand from where this limit comes into picture. So OK, then why we didn't get a good error message from Keystone that the connection to the database is lost. So the problem is basically HAProxy is going to send the messages into a queue. And it will never sometimes close that connection on time. So I'm not getting very deep into that. But we thought that we have found the bottleneck that's HAProxy default maximum limit that is invisible. And we thought we will be happy and we can go ahead and sleep after this. But we got different questions from the customer. If 2,000 is not the right limit, what is the right limit for me? So again, we are already sleepless. Let's go and work on this tomorrow. For time being, you make it 10,000 or 20,000 or whatever you want. OK, so it took us more time to understand what should be the right maximum limit for MariaDB Galera for an OpenStack environment. This is what you need to give a clear, close look and implement in your OpenStack environment. Number of database connections depends on various factors. So how many workers are spawned by each API services? This is clearly dependent on the workers or API workers or OS API compute workers configuration that you make during the deployment. And we get a recommendation everywhere that number of CPUs that you have or number of cores that you have on your system is the optimum value for this number of workers. So let's step back. This is what you are going to see in each and every configuration files that you see with the blue background number of workers for OpenStack API services. The default will be the number of CPUs available. So let's just go into the next slide and take a closer look. If you have a 24-core system, how many database connections are going to be opened by each and every worker? So NOVA has actually three different workers that is OS compute API workers and the NOVA API workers and metadata workers. You configure all of these to be equal to the number of CPU cores, that is 24 into 3. Then just like that, you sometimes configure the number of neutron. So you sometimes configure the NOVA conductor has 24 into 1. And Keystone is 24 into 2. There are admin API workers and Keystone public API workers. So that means on a single node, you are going to have 264 number of workers on a typical, on a normal OpenStack deployment with the core services. And each API worker is going to open five long-lived connections to the database. It opens the connection and keeps the connection open forever. And there is no timeout, and it's a long-lived connection. It keeps that connection open every time. So that means on a single node, it's 64 workers in five database connections is 1320. And we have three controllers here. All of them are going to spawn this many number of workers. And there are also all the workers on all the nodes are going to open five long-lived database connections. That means a 24-core system is going to open 3960 number of connections. And on the right side, you see, it's not just long-lived connections. There are also short-lived connections by some of the services. We haven't really able to quantify anything about the short-lived connections, how much is open and how they are closed. But to be on the safer side, we say that you add 1024 to consider short-lived connections, other services that you may add in future, like Trove, Sahara, or anything like that. And assume that in this specific environment, this customer had 96-core system. So it's not really surprising that he has hit this limit very early enough, because obviously he has exhausted all the connections. All he doesn't have enough HAProxy connections to open connections to the database. So it's just not HAProxy that you need to configure with the maximum connections. There is also another variable within the database that everyone is familiar with, maximum underscore connections. So I hope you all know that this also need to be changed depending upon the value that you got. So if you want to get a better sleep, it's better you make the RabbitMQ file descriptors very high. And you also calculate, depending upon your environment, how many CPU cores are there, how many workers are spawned by each and every open stack API services. And you do this calculation and update the maximum number of MariaDB Galera descriptions. Then you sleep like a king or a baby like this. So again, we went into a stage. We created knowledge and support. We created an article for this to help each and every customer who may hit this in the future to help them. And that is knowledge and support, this kind of reactive support. If they hit the problem, they come to know this article and they solve it. And Dustin is going to speak about the next evolution on how we can make this reactive support a proactive support. So we see that sometimes the solutions are relatively simple and straightforward, even if it's elusive in the first example. A little bit difficult to troubleshoot where the problem was, but once you find that it's one simple value, you take a fairly liberal approach to setting it, set it high, and you're going to be in good shape. Sometimes you have other symptoms and they're going to require a lot more troubleshooting. The symptoms may look similar, but you've got a long way to go to actually find this new real problem. So again, this knowledge-centered support is really important to our processes at Red Hat because all of these new things that Sadiq discovered in troubleshooting this problem that looked similar ends up becoming yet another article that we can publish and that you can find as a customer that we can reference as support engineers to try to make sure that we are not spending the cycles doing all this troubleshooting again. And that has been extremely effective with us, very, very important for our support processes for a long time. The trouble is it's reactive. It's necessarily reactive. You only know to actually go find these things if you run into the problem. Hopefully part of the lesson learned here is never trust the defaults. And everybody should go and consider the scale of their environment, the kind of workloads they're going to be running, and consider what all of these tunables are. But there are a lot of them. There are a lot of places you can have these bottlenecks and these failures. So one of the things that we are moving towards, and this is kind of our next generation of improving our support model, is trying to get these issues away from the reactive and towards the proactive, enabling you as the customer, as the user, to try to find these problems before they occur. So this is just a quick little plug of the project that we're working on. It's in beta right now. It's called Red Hat Insights. And this is going to be a tool where you can have an agent that is going to help collect a lot of this information, compare it against our ever-growing knowledge base, and help find these issues before you ever run into them. It's going to be a really great tool for taking a look at things before you're doing deployments, but also along the way too, because we're always discovering new things. And as our knowledge base is growing, that's going to feed into the Insights system, and the Insights system is going to continue to be knowledgeable, so you may be able to get alerts and reports of new things that we discover that may help you prevent running into a problem in the future. So highly encourage you to check this out. We're going to be developing this more and more where lots and lots of new plugins are coming, more development across our product lines, and there's some good information available at the Insights website. Again, our contact information. Feel free to reach out. If anybody has any questions, we are welcome to come up to a microphone and speak. If not, you can certainly contact us anytime. Appreciate everybody joining today. Hi, how you guys doing? Good, great presentation. Great question. You talked about a knowledge base earlier in your presentation. Can you share what are you actually using as an underlying platform? Is it some open source platform for knowledge base or something that you guys have had for a long time at Red Hat that you just implemented? It's gone through a couple of iterations. Is my mic still on? Yeah, it's gone through a couple of iterations. Do you know what the current backing software is? It isn't, I believe it is an open source project. Yeah, it's going to be an open source project, but I exactly don't remember. Yeah, I can't remember. It changed a couple of years ago from one project to another. It's all tied into our internal support systems. So when you open a case, we get a lot of this backend information about your account and history, flags, and internal communications, but then it's all integrated between our view of the support system and your view of the portal. So we're trying to push the knowledge that we gain from the support case into the KCS solution and get it published to the portal as quickly as possible. I'm just curious if there is a quick open source knowledge base solution that people can use. Send me a message and let me get the answer to you. I don't know off the top of my head what the tool is. Any other questions? All right, thanks again, everybody. Appreciate it.


Hello, Kubcon, and welcome to Multicluster is easier than you think with LinkerD and Ambassador. We're going to be showing you how to Multicluster the easy way. And at the end, I'm going to give you a couple great use cases for Multicluster that you can go use on your own. Before we get started, let me introduce my co-presenter, Daniel Bryant. Daniel, why don't you tell us a little bit about yourself? Thanks, Thomas. Hello, everyone. Daniel Bryant, Product Architects Ambassador Labs here. I'm working on the Ambassador API Gateway Project for the past three or so years. Loving this technology, of course. Loving LinkerD, loving Service Mesh in general. Combination of Ambassador and LinkerD for Multicluster is super interesting. So looking forward to bringing you the demo today. And my name is Thomas Rampelberg. I'm a software engineer at Boyant, the creators of LinkerD, a CNCF Service Mesh. Before we get started, let me introduce the two projects that we're going to be using to build out our Multicluster solution. The first project is Ambassador. Ambassador is an API gateway. In Kubernetes, you need to have an Ingress Controller. Ambassador does that and quite a bit more. In fact, to get into production, you need a lot of things like authentication, rate limiting, request filters. And Ambassador does an absolutely fantastic job of all of those. It is open source, and I would recommend checking out their website to go and kick the tires on that if you've been looking for an Ingress Controller. I work on LinkerD, which is a Service Mesh. In fact, it is a ultra-light, ultra-fast, security-first Service Mesh for Kubernetes. You get observability, reliability, and security right out of the box, which is kind of all of the normal Service Mesh check boxes. Obviously, the thing we're going to be talking about today is Multicluster, and LinkerD makes that even easier than you can imagine. So let's talk a little bit about what we're going to go over. To do Multicluster the easy way, you really need to have four separate things. You need to understand how Service Discovery works and be able to discover services across clusters. You need to have cross-cluster access. You need to be able to get the requests from one cluster to the other. You need to be able to route the requests in an intelligent fashion, and my favorite part of Multicluster the easy way is profit. So to get started, Daniel, why don't you give us a little bit of a sneak peek into the demo and everything that we're going to be working on here? So welcome to the demo. Let me start at the end. This is actually what we're going to end up here. You can see I've got a command line terminal showing a East Kubernetes cluster. This is a GKE cluster running Frontend and Pod Info services. And you can see I've got a local Docker Desktop cluster down here on the context of West. I'm again running Frontend and Pod Info here, but I'm also running a Service Mirror of Pod Info East. So the Pod Info East service in my local Docker Desktop cluster is actually pointing to the GKE cluster. It's actually pointing to the mirrored service there. And you can see I've exposed using Ambassador, local host onto the Frontend service running in my Docker Desktop cluster. And because I've got some traffic splits set up using SMI, which I'll show you at the end, you can see as I'm actually making the request, I've got some refreshing going on the background against local host. We're hitting West and East versions of Pod Info service 50-50. That's what the traffic split is set up. So although I'm always hitting the Ambassador ingress instance running in my Docker Desktop cluster, sometimes the response is being served by the Pod Info service running in Docker Desktop, but sometimes 50% is also being served by the Pod Info service running in GKE, running remotely. And that's all thanks to the power of Service Mirroring. And it's all secure traffic between the two clusters. And we'll now run you through how you can actually set up your own. So for today's LinkerD in Ambassador multi-cluster demo, I'm going to be using two Kubernetes clusters. I've spun up one already to save time in GKE, using the G Cloud command line tool. I've spun that up in the US East region. I'm going to be calling that cluster the East cluster. That's my top right window here. I've also spun up a local Docker Desktop cluster. That one's going to be shown in the bottom right window here. And I'm going to be running my commands against the clusters using the different contexts of East and West. And I'm going to be running all the LinkerD commands and the Ambassador commands in my console on the left here. These clusters at the moment are both empty. So let's pop along to the Kate's initializer and use this to bootstrap the clusters. This is a free tool. You can install a bunch of stuff. Ambassador for Ingress, Prometheus for Monitoring, Yeager, Keynote, Argo CD, et cetera. You can select the configuration and auto-generate the YAML and simply apply it to your cluster. So for Docker Desktop, for example, my West cluster, I'd select Docker Desktop. Don't want to terminate TLS. Again, this is a toy example. You do want to look at TLS in production. Once I've selected my config, I can select more down here if I want. But today, let's keep it nice and simple. I go review and install, and voila. I get some kubectl apply commands I can run against my cluster. Now I've got some pre-configured commands I've done already because I want to put the contexts in, so I'm not going to be copying and pasting from here, but this is exactly what I would do. Let's go back and I'll show you with the GKE one. It's exactly the same. Select GKE. I'm going to be using an L4 low-pouncer. Don't want to terminate TLS in this toy example. Public hostname of star, and I review and install. And again, I just install that against my cluster. Let's actually do that now with my pre-configured cheat sheet. First, I'll run my watch commands on the Kubernetes clusters. At the top here, I'm going to be watching the east context, our GKE cluster. I'm looking for services in all names places. So as we add more things, you'll see more services appear here. And in the bottom window, I'll do the same for the context of west, my Docker desktop cluster, so you can see as I have more things here. Right, let's install ambassador in both clusters. With the east cluster now done, we can see in the top window, we've got our ambassador services in addition to our Kubernetes core services too. Let's now install ambassador on Docker desktop. Now we've installed ambassador on Docker desktop, so we have our ingresses set up in both clusters, both the remote cluster, GKE, and also local Docker desktop cluster. Let's now move on to the linkerd multi-cluster setup. Now the first thing we're going to do with our multi-cluster setup with the linkerd is to create a shared trust anchor. Public private keys between the two clusters that enables them to communicate securely. I'm going to use these steps here like man, you could use open SL, whatever you like, and generate my certificates into my directory here. Once that's done, let's run the linkerd install command in both our clusters, using a little bit of a cheat here to do both east and west at the same time. And you'll notice on the windows on the right, that as the linkerd gets installed, you'll see some of the services popping up here. This is all looking good now. You can see the linkerd namespace and the linkerd services, both in my GKE cluster and also my Docker desktop cluster down here. Let's do a quick check on the linkerd config. The status checks look good. That's one thing I do very much like about the linkerd CLI. You can run all the health checks and it presents everything in a very nice way. We can scroll up and down and look at all the various config parameters and know everything is good. Thanks Daniel. It is really cool to see what it takes to get ambassador up running on, not one cluster, but two clusters, as well as linkerd all set up and ready to go. Now, for those of you who are hopefully going to be trying this out at home, I would like to note that while it was, it's going to be pretty quick for you. It probably will be a little bit slower on your own clusters because I used a little bit of movie magic there and sped everything up. Great. So with all of the components for ambassador and linkerd setup, let's talk a little bit about what's required and what all of those components do. So to get multi cluster up the easy way, the first thing to dig into is service discovery. To talk a little bit about service discovery though, first let's go into what it means in Kubernetes. So in Kubernetes, I'm going to be using the pod info service here as an example. Kubernetes has a resource type called services. When a service is created inside that namespace, you get DNS for it automatically. I've got that up here on the slide. It's a pod info test SVC cluster local and the front end app that I've got listed here can address pod info directly. When it does that, the DNS response from core DNS returns a cluster IP. That cluster IP gets rewritten through the magic of kube proxy and ends up directly at the pod. This is great. It works out of the box. Kubernetes is fantastic on service discovery. But once we add another cluster, there's a little bit of a problem that we run into. Very specifically, you can't out of the box share resources between two Kubernetes clusters. How do we go and get the pod info service visible and addressable from the front end app? If the pod info is in the East cluster and front end is in the West cluster? Well, very, very, very naively. Here's your answer. Kube CTL context East gets the pod info YAML and pipe it into the West cluster. That's pretty much all you really need to do. Now there's a couple downsides to this and we're gonna go into why we've installed a component to do this, but that moves the pod info service over to the West cluster from the East cluster. And now there's going to be DNS that can be addressed for front end. But as I'm sure some of you have already thought through, one of the biggest problems there is, is that it isn't going to get updates. If pod info was changed, if it was deleted, you wouldn't get those notifications on West. And so in LinkerD, we've introduced a controller. Kubernetes has a controller patterns, deployments are managed through controllers, services are managed through controllers, and those controllers basically watch the API server for resources to be updated and then do something. And so the LinkerD service mirror watches the East clusters API server. And when it sees pod info created, deleted, updated, it goes and does that on the West cluster. That's pretty much all it does. The one important thing to point out here though is that it adds the East clusters name as a suffix to pod info so that you can directly address it. Great. So all we've done is tackle service discovery. We're going to go into a little bit about how the cross cluster access works next. But before we do that, Daniel, why don't you give us a little bit of a demo on what it takes to install the service mirror with LinkerD on a cluster? Let's now look at installing multi cluster in very much the same way. This is the command we'll be running here. I won't be installing the gateway. You'll notice here, install gateway equals false because we've already set up ambassador and we're going to be using ambassador as our gateway. With service discovery set up, we can now address services in the East cluster. But how do we actually get packets to go over there? To figure that out, let's talk a little bit about a resource type in Kubernetes that is kind of under the covers. Some folks don't spend very much time chatting about it and that is the endpoint resource. The endpoint resource in Kubernetes is kind of like this YAML that I've got up here. Really just the pod IP addresses for a service. So the endpoint controller watches services and takes the pod selector from those services, goes and looks up the pods and adds IP addresses to an endpoint object for that. Just like our service mirror is syncing the services across, we could just stop here and sync the pod or the endpoint resource across the clusters, attach that to our mirrored service and let everything go through. Unfortunately, that presents two problems. Problem number one is that pod IP addresses would need to be routable between cluster West and cluster East. If you can imagine a multicloud scenario where you're running a Kubernetes cluster in two different managed providers, that would be pretty difficult possible, but pretty difficult. The overlays configuration there would not be something I would want to do. Perhaps the bigger problem though is bandwidth, interestingly enough. The endpoints object here has all of the IP addresses for the pod info service in it. So anytime those are updated, it'll get synced across from the East cluster to the West cluster. Not only is that going to chew up a bunch of bandwidth, but it will also have some issues around keeping everything up to date. Imagine a pod IP going away in the East cluster and not getting updated in the West cluster, you potentially could have some lost packets. So what's the solution that we have to this problem? Well, if you've been waiting, the solution to this problem is actually ambassador. Why not use the same ambassador that you've got already managing your ingress for the multi-cluster communication? Ambassador already has a public IP address. It's routable from the West cluster. There's no need to pay for an extra load balancer and it can manage the traffic for you as part of its production setup already. So what do we need to do to get that all wired up? Well, all we need to do is tell our service mirror to instead of move all of the endpoints from the East cluster over to the West, we just grab the public IP address of the ambassador load balancer and stick that into the endpoints. One thing to note here is that when the service mirror syncs the pod info service from the East cluster to the West cluster, it also removes the pod selector to make sure that the built-in Kubernetes endpoint controller doesn't go and modify any endpoints for us. Pretty cool, right? That's it. Now all of our traffic from front end can get correctly addressed to pod info dash East and that will be forwarded by Kubernetes over to ambassador, which will finally pass it on to the pod info pods in the East cluster. I'd like to have one note here really quick around the routing because all of this traffic is going over the greater internet. Security is pretty important. Linkerty provides MTLS out of the box for both sides because there's proxies on the front end West cluster and ambassador East cluster, that communication across the internet is encrypted. Because it's MTLS, remember it's the mutual part, it's also authenticated. And so the ambassador only allows traffic from the West cluster or more specifically signed with the same, a certificate, signed with the same trust route into the cluster to get to pod info. And so your traffic is secure and access is also secured. So you don't need to worry about the security there. That said, security is a amazingly deep, deep, deep dive. And I've added a link to this slide so that you can go and take a look and go into all of the nitty gritty details if that's something that you're interested in. Fantastic. Daniel, can you show us what it takes to get ambassador set up as a gateway for all of this traffic and a multicluster solution? It looks good. This is probably the most interesting patch. Here we're patching the gateway to an open ports and ambassador to allow Linkerty to talk through and sort of to the mirrored services on either side of the cluster. That all looks good. You can see now lots of extra things are spinning up there and also the timing is nudging the window a little bit to display slightly strange. We'll just now check the rollout is working successfully. Here's nice command line cheat we can use to do that. Everything is successfully rolled out, looking great. And we can also check the multicluster aspect from the Linkerty CLI as well. I'll now just get all the Linkerty multicluster config. Everything is looking good. Everything's up and running here. And now I can link the two clusters with this command here. You can see I use the Linkerty command here from the context of the East linking in the West and actually a cube cuddling apply into the Docker desktop cluster. That all looks good. Let's now run our checks. Once again, context West check multicluster. You'll notice here we can now see the cluster East gateway ambassador ambassador. That's our GKE cluster is now configured. Linkerty running in Docker desktop in the West cluster now access the gateway of the GKE cluster. Now to make the display more interesting, I'll stop all namespaces and we'll now switch to the test namespace as we set up some services to play around with the service mirroring. I still can't believe how easy it is to get ambassador setup so that it works as a gateway for multicluster. It's pretty awesome. Thank you, Daniel. Now that we've got the gateway setup, we've figured out how to do service discovery across clusters. We have showed how to get packets across securely and now let's talk a little bit about request routing and why that matters. Being able to address services in other clusters is cool. As you'll see here, we're directly addressing pod info East, but you know what would be even cooler being able to shift the traffic over without needing to change code or even restarting anything. Isn't that the cloud native dream after all? So far, we have had a bit of a theme. We have built on top of Kubernetes primitives and only Kubernetes primitives. And the reason for that is that the Kubernetes ecosystem can compose and build on top of those very easily. We've used services, we've used endpoints, we've used load balancers and we've used ingress controllers. All of those are standard off the shelf components. This is perhaps the first non-standard component though it's part of the service mesh interface spec and what it does is allow us to split traffic. And because all of our service mirroring is using services, all it does is split traffic between two services. So here in our spec, we have said that any traffic addressed to pod info should be split 50% to the pod info service on the West cluster because we're gonna be talking about the West cluster here and 50% to the pod info East service on the East cluster. All you need to do is modify this Kubernetes resource and all of the traffic coming out of the front end application addressed to pod info will get split 50-50. And this is how it ends up looking. Being able to modify the eventual destination of the traffic to pod info really opens up a bunch of interesting possibilities. And because this is built on top of Kubernetes primitives, it opens up the world outside of service meshes and API gateways. Daniel, do you think you can install the front end and pod info services on your clusters in West and East and show us how the traffic splitting actually works in the real world? What I'm doing is I'm just watching in the namespace of test. This is the East cluster, the GKE cluster at the top here and at the bottom, I'm again looking for services in the namespace test, but this is in my Docker desktop environment. Now I'll run some simple script to see if the buoyant team, to go to the LinkedIn GitHub, download some services, some example services, some of them based on pod info from Stefan Pradam and some other simple apps as well, which we can use to demonstrate the service mirroring between Docker desktop cluster and the remote GKE cluster. This is gonna take a minute to do install. With our deployment complete, we can now see in both clusters we have the front end service and the pod info service there. GKE cluster at the top here and then the Docker desktop cluster in the bottom here. Let's now expose a very simple mapping in the ambassador gateway just to map the prefix of slash. I'm exposing Docker desktop on localhost. So localhost slash at the root, we're gonna map to the front end service running in the test namespace on port 8080. Let's do that now. Now if I pop along to the app browser and do localhost slash, voila, we are now looking at the pod info or the front end and pod info services running in my Docker desktop, west local cluster. All looks good. Now let's do some service mirroring. So if I clear my terminal and I'll copy and paste in this command here, I'll use the linkity multicuster export service to add some annotations to my services. You can see I'm configuring the gateway names ambassador and the gateway namespaces as ambassador to this overriding the defaults of the linkity gateway but that is nice and straightforward, hopefully. And we should see in my Docker desktop cluster in the bottom right here, we've now got a service named pod info east. This is effectively a mirror of the pod info running remotely in the east cluster in GKE. If we want more info on that, I can run a command here. I can basically attempt to get the endpoints for the pod info east running in my Docker desktop cluster. And I can also get the ambassador at global low banter IP address as well and compare them. And there you go. You can see that the west pod info east service endpoint IP is equal to the gateway IP in our east cluster. So basically the Docker desktop pod info east service mirror is pointing at the gateway of the GKE cluster, the east cluster. Now we're running some traffic generation in the back end as part of the demo. So you can do all the cool, regular stuff you can with linkity. You can get your stats to see what kind of latency and requests are going through. You can also look at the actual, you can do a tap and look at the traffic too, which is quite nice. You can see here also, I'll just pause it a second, TLS true. Although ambassadors are not running TLS exposed in this toy example, because we've got our shared trust anchor, we're of course encrypting traffic between the clusters courtesy of linkity, which is great too. Linkity command line is fantastic as is the dashboard for debugging, poking around the services, frequently find myself in there looking to make sure I've got my traffic config all set up correctly. Now I could simply expose the pod info east via my west gateway via my west ambassador, but it's making it more interesting and we're going to be using the SMI, the service mesh interface config to do some traffic splits. So if you can see here, I'm going to apply in my Docker desktop cluster, a traffic split on pod info across the pod info that's running in the Docker desktop west cluster. And I'm going to also split 50 50 to the pod info east service. And that is going to mirror into the east based, the GKE based pod info service. Let's apply this now. All looks good. If I pop back to my browser and do a refresh, oh, we're already seeing east. Perfect. And there we go. We see some west, we obviously see some east. Looks good. How cool is that? We've now gone through getting multicluster set up the easy way. And that's pretty much it. We've shown what you need to do to get service discovery so that you can see services from one cluster and another. We've outlined what it takes to get cross cluster access and make it secure. And we've shown everything you can use to do request routing. So the most important step of all is right now, profit, where we sit back in our chairs and talk about all of the amazing things that we've enabled with this. So what have we enabled? Well, one of the most interesting use cases for me is cluster isolation. Imagine that you're managing credit card numbers and that's a compliant environment that you need to have audited. Instead of having one big Kubernetes cluster that has everything in it, you now can go and stick that into a very small cluster and make your auditor love you because they have a smaller surface area to check now. You've got an insecure cluster that you can open up access to the rest of the company on and you don't need to go and put some crazy policies in place. Another use case for this is disaster recovery. Imagine a very important service failing. The front end can now get automatically redirected to your backup cluster or perhaps more interesting. The backup cluster can be tested with real traffic during normal operations. So instead of having to hope and pray that your backup cluster is going to work, you can go and test your disaster recovery during the day at any time that you want. Perhaps the most exciting use case for me though is development. I know we're virtual, so I can't see anybody's hands raised, but I'm gonna raise my hand. How many of us have worked on microservices and had it get to the point where it doesn't fit on your laptop anymore? I sure have.  You can use this multi-cluster to go have a shared cluster now. So any services you're not actively working on, just go use them in a remote shared cluster and you can work on your service locally. And that's pretty much all you need. So that's everything that we've got for multi-cluster. Just a couple more slides here to talk about. First off, I'd like to introduce the Linkardee Community Anchor Program. We really like to get the community involved and especially if you're using Ambassador in Linkardee, we would love to know about it. So jump onto that link down at the bottom of the slide. If you're interested in becoming a cloud native expert and getting a little bit of help from us to tell your story. Also, join both of our communities. I've got links to our GitHub and Slack accounts. Daniel and I both are Slack junkies and are sitting around waiting for your questions with bated breath. So please dive into those. And then finally, here are links to everything that we've gone through so far today. I would really love to see everyone here go through the tutorials, deep dives and get up and running with multi-cluster on your own clusters. Thank you so much and have a great rest of your coupon.


from Six Sister Steph and today we are making a delicious orange punch but first let me explain my look for the month of October we decided we'd be festive and so you'll see all of the Six Sisters in our videos dressed up to be to celebrate Halloween so you can guess who I am later but so we are going to make orange party punch and this is perfect for Halloween parties or just any gathering whatsoever it's good all times of the year but to start out you'll just want a big bowl or a punch bowl and I've got a two liter of sprite and if you want to cut this down you can cut it in half if it if you won't be serving as many people it's pretty easy it'll taste good however you do it a little frothy so after you've added your sprite I've got a can of orange juice concentrate that I have thought in my fridge if it's frozen it won't mix as well and so you'll just pour that in that gives it the orange flavor then I've got a 46 ounce can of pineapple juice and you pour that in it's really easy so you'll just mix this up just to get everything combined and then you're going to top it with a few scoops of orange sherbet you could also use lime that would still be festive for Halloween so and it might be easier depending on how many you're serving to actually serve the scoops on top of the individual cups but it's just up to you these will kind of melt they're sort of like flavorful ice cubes so it'll add more orange flavor to your drink and make it look extra festive I guess so I usually do about five scoops but you can do as much or as little as you want some people like to mix it in others just leave a floating on top and there you have it our orange party punch and you can find more recipes like this on our blogs six sisters stuff calm happy Halloween


Hi everyone, Sean with jaspyscasebreaks.com here doing the random number block randomizer for Tops Chrome Black Pick Your Team 3 random number block, Chicago White Sox, Seattle Mariners, Tampa Bay Rays, Chicago Cubs, St. Louis Cardinals, and Cincinnati Reds. So here is our, actually as all random number blocks, every spot gets a random number from this break, zero through nine. I think the number three will get all cards with three being the first number next to the slash on the left side of the slash. So for example, if the card is 13 out of 99, three out of 25, 43 out of 49, 93 out of 99, and so on. Number three will get that card. All redemption cards will go to the zero spot. Even if it is a known one of one, it will still go to the zero spot. All in person one of ones will go to the one spot. With this break, there are unnumbered cards. So these cards that are unnumbered in the set, including autographs, relics, base cards, etc. All unnumbered cards in this block of teams will be randomized to one person in the one lot at the end of the break. So there you go. Here is our list of customers from Alexander Weprin down to Vinny Vichy last spot mojo bought straight up and then four spots that were one through our tops update filler and our numbers zero through nine dice roll here to randomize both lists and pair them up one and one two times snake eyes so one two so Alexander down to Vinny and one and two on the numbers one two from zero down to four so let's make this easier to read Alexander Weprin with zero Eric Jock with two Matt McLaughlin with eight Jimmy Wang with nine Steven Savinsky with five Vinny Vichy with three Jeremy Felecker with seven Johnny Ho with one Logan Callister with six and Vinny Vichy with four so Alexander Weprin would get any and all redemption to this break even known one of one redemptions while Johnny Ho would get in person one of ones so that is the random number block randomizer four tops chrome black pick your team number three random number block chicago white socks mariners rays cubs cardinals and reds pick your team three tops chrome black coming up next i'm sean jaspyscasebreaks.com thank you very much hanging out and i'll see you next time