Hello, welcome to my KubeCon session, Breaking Your Kubernetes Cluster with Networking. My name is Thomas Graf. I'm CTO and co-founder of ISOvalent. I'm also one of the co-maintainers of Cilium, a CNI plugin for Kubernetes. And in this role as being one of the maintainers, I had the chance, together with the rest of our team, to work with many of you on networking-related topics and as part of this, we learned a couple of things around networking on what could go wrong. So in today's session, we'll kind of dive into them and give you an opportunity to learn from what we have experienced over the last couple of years. It's quite obvious that networking is everywhere. We take networking for granted every day in our life. We have Wi-Fi connectivity all the time. We kind of get this immediate itch if internet connectivity is out. So networking is here, networking connects everything. We're taking for granted it has to work. If the network is not available, it usually leads to very serious consequences. And at least I saw a couple of mentions in network, or for networking, even in the KubeCon title slide, so the previous one, let's have a quick look. So when I saw this slide, I saw a couple of things. I saw like an app team scheduling 5,000 services in there. I saw like pulling on KubeProxy. I saw KubeProxy trying to pull up IP tables, trying to kind of make it scale, trying to kind of implement intent from the app team. Hey, I need the app team wants me to schedule 5,000 services. I need those packets low balance.  I'm even seeing like a person looking away, kind of resembling the platform team, ignoring crashing code DNS parts, which are affecting the app team. And even the Moto 4 Watch together is kind of networking related. It made me think of like all the different components involved in Kubernetes networking, CNI, CNI chaining, KubeProxy, Ingress, Core DNS, Service Mesh, and Cloud Networking, all coming together and trying to forward our packets together. So let's dive in. Kubernetes networking, the dark side. So Kubernetes networking is simple and I think it has a lot of great advantages and in general it works really, really well. There are a couple of dark sides to it and that's what we will kind of enlighten today. Obviously we cannot have a networking session without a special appearance from DNS, very obvious. If you like DNS stories, you will definitely get enough of them today. But before we dive into concrete example, a bit of context. Where are the stories coming from? First of all, as I mentioned, I'm a cylinder maintainer. So these are stories from our users. If you have never heard of Cilium, Cilium is a CNI plugin among other things. It is EVPF based and it provides networking, security and observability. And don't worry, this talk is not going to be about Cilium. I will leave it at this. If you want to learn more about Cilium, check out Cilium.io. It has pointers to all the source code repositories, docs and so on. You will learn everything that you want to learn on that website. All right, so let's dive in. Before we go into stories themselves, we'll do a very, very quick one-on-one on Kubernetes networking. Because if you boil it down, it can really be simple. Obviously this is slightly oversimplified and in some cases it can be a bit more complex than this. But in general, these are the assumptions that Kubernetes networking is making. First of all, all pods have an IP. And second, all pods can talk to each other. This is typically called a flat layer three network. And then in general, this is no longer always true, but in general pod citers are allocated to nodes and then nodes allocate out of that cider to the pods. So in this example, we have 10, 0, 1, 0 slash 24 and the local parts and that nodes will get IPs out of that range. Then if you want any sort of load balancing, you will use Kubernetes services and you can use DNS for service discovery and network policy for segmentation. So in general, it is as simple as this, which I think on its own has a lot of merits and a lot of benefits. And most of the stories we'll hear today are actually not about failures of this model. It's about details and about like small little pieces that you can get wrong. So first of all, of course, DNS. Kubernetes DNS, as we learned, it's used for service discovery. It's usually implemented with core DNS, although you can of course implement it in another way. It's often run as a multi-replicate deployment, but you can also run it as a demon set. So you have a DNS responder on every node and it typically needs no app changes. So KubeNet will inject a resolve.conf into the pod correctly. And it looks simple, or in general it is a simple model. If you believe everything I've just said, you probably have never seen this. It's not DNS. There is no way it's DNS. It was DNS. If you are new to DevOps, this is the first thing you should learn. It is always DNS. In fact, if you remember this graphic, if you have this graphic handy, you probably get a free level of seniority in DevOps right away. So let's look at a first, I have a couple here. First, I think this is a very common one, the end-dots default.  This search parameter is a list of names and these names will be appended at the end of each name. If the name that you look up is not fully qualified. This is what allows you to just look up the service name without the namespace and the service suffix and so on. This is what makes this work. And the second important one here is end-dots, number of dots. This defaults to five in Kubernetes. End-dots basically says that if you have less than five dots in your name, it will always append the search list to it. And if you have no clue what I was just talking about, maybe a quick demo on how this looks like will help you. So let's switch over to our Kubernetes cluster here. I have a couple of parts running and I will run a Hubble. Hubble is Syllium's observability layer based on eBPF. And we'll run this and this will, in a second, this will show all the DNS lookups that we're making. And in the upper window, I'm basically just cube control exacting into the client. And then I'm running curl Google.com. And this has succeeded. So Google has returned 301. And you can see here, in fact, this was Google.com responding. And then in the lower window, so you see a bunch of output. We can actually scroll back up, like lots and lots of lookups were made. Let's scroll all the way here to here to the start. So here was the first lookup. So it actually did a DNS query for Google.com.default.svt.cluster.local. And it was actually for 4A. So this was IPv6. And then the second one was done, the same name, but for IPv4. When the DNS server is responding, not found or not existent and so on. I can look at all of these names. So basically go through the search list and tries all of them until let's go all the way down until the very end, it actually does the lookup that we want. So Google.com. And again, it does this twice for IPv4 and IPv6. And you see Google's DNS actually returning here and returning IPv6 and IPv4 address of Google. Great. So what we just learned is that for any non-FQDN lookup, you're not really doing one or two lookups for IPv4 and IPv6, but you're actually doing five, internal five if you have five search entries or four plus the fully qualified one per address family. So basically 10. So we just did curl.google.com, but we actually did 10 DNS lookups. This is not ideal. It will lead to a massive number of DNS lookups. So please educate yourself about end dots. You want to understand this and change this default. And it actually plays into the next problem that we have, DNS rate limiting. Most cloud providers actually do rate limit DNS. For example, AWS limits DNS to 1,000 packets per second per ENI. This sounds like a lot, but think of last slide, you'll have lots and lots of lookups. It's actually very hard to notice. So you've actually very likely been limited, but you never knew. It leads to random connectivity errors because it will happen and it will fix itself. And it's also often hidden because P99 latency measurements often don't cover DNS so that it's not shown. Often if you have latency metrics in your dashboards, it will only show TCP handshake metrics, like how long does it take for TCP connection to be established or how long does my HTTP request response latency or what's the latency between the request response and so on. It often hides these DNS latency. So hard to notice. The way we found this is with the Hubble observability layer that we just saw in the demo. Don't worry, we'll have more DNS, but for now we'll switch over to network policy. Network policy, if you have never heard about it, very easy, declares who can talk to whom. So let's say you have a front-end pod and you have a back-end pod. With network policy, you can basically inject or start a firewall in Kubernetes and then define roles. For example, front-end can talk to backend and everything else should be denied. So one of the most common failures that we are seeing here is something like this. User wants to allow front-end talking to backend, writes a simple network policy. Network policy spec, pod selector match labels at front-end, egress to pod selector match labels at back-end. Cool, easy, straightforward. What could go wrong? This is allowing front-end to back-end, easy. Well, did you think about DNS? Often DNS is overlooked. So if you add an egress network policy, you will automatically put egress of that pod into default deny, which will block DNS. So you have to allow DNS. It's surprising how many users are being bitten by this. The problem is, and yes, with some CNIs, including some, you can define global policies to define this cluster-wide, but with standard network policy, you have to allow this for every namespace separately. So as soon as you go from default allow into default deny, you have to think of DNS. So what you need is something closer to this, where you're not only allowing front-end to back-end, but you also now allow to code DNS. You can now either edit your YAML or, and I will demonstrate this real quick, we have actually made our network policy editor available for everybody. So this is a free offering that you can use. You can access it via editor.cylum.io. And it's basically a visualizer and editor for network policies. So I've loaded the policy that we have looked at. So you can see there is like a pod selector app front-end, and it allows in the namespace to app equals back-end. Great, so you see the green arrow, the green, yeah, the green arrow here. And you can also see that Kubernetes DNS is actually still red. So we can go in and actually allow DNS. Great, so it actually added the YAML. We could now download this and apply it. Kubernetes services and load balancing. Couple of things here that will bite you, the first one, and probably the one that you've heard before is simply scale. So if you're running Qproxy in default configuration, you will be using IP tables. It uses linear lists of rules. So as you grow your number of services and endpoints, latency will go up. I've taken a screenshot here from a QCon talk of 2019, a team member of ours presented like a BPF implementation and also showed numbers comparing BPF, IPvS, and IP tables. And you can see that BPF and IPvS both have basically fixed cost, no matter how many services you're running, whereas the IP tables-based implementation will simply show increased latency as you scale up your number of services. And it's not actually observable only like 5,000 services or something like this. I think there was a claim earlier on by another QCon talk that you need to run a lot of services. 1,000 services is not that much and you already start to see the difference. Service loopback, this is essentially a part talking to itself via a service. So very common if you have like a front-end and you have a front-end service and the front-end is talking to the front-end service. This will lead to packets being routed back into the same pod. With many CNIs, this will just fail silently. And the reason for this is because Linux or Linux networking, the actual kernel only accepts network packets where the source and the destination IP are the same if that forwarding is happening on the loopback, the LO device. For Kubernetes networking that is not the case, packets will go out, they will go to the cluster IP of the front-end and they will be translated and will go back in. This will lead to random connections breaking. Linux will simply silently drop them. It's worse because often you're running multiple replicas of front-end, which means that the maturity of connections will succeed. And then depending on how many replicas you're running only some of them will fail. Some CNIs, including Syllium, I'm not quite sure which with other ones as well, do support this specifically. So they will translate the source IP to a different IP to fix or to work around this limitation of the Linux kernel. All right, the last one here on the service load balancing side is a manual attack around external IP, which basically allows you to redirect any traffic with the help of an external IP service. And before I go into lots of explanations, I will just simply demo this. Let's go back into our cluster. And to make sure we have a bit of visibility, I'm actually going to run Hubble again and I'm going to limit the visibility to a particular client part. And I'm going to curl this IP. And actually let's curl this IP from my laptop first here. So this IP is basically an IP of Google.com. So I've curled the IP here and Google is responding. So let's run Hubble again and let's curl this IP from my client. I'm actually expecting the same. So the same output should happen. So let's see what we see here. Huh, this doesn't look like Google at all. Like what is this? So this is like some output, like some emojis even here, like your successor running a JSON server. Definitely something else. So let's look at Hubble. And Hubble is actually telling me that we are seeing packets from like the client going to EchoA, which is kind of, is that another pod? Yeah, so this is another pod running here. So I can look at this service here that I have injected. So I have this EchoA service which is a type load balancer. It has an external IP in what I did. I've simply added the external IP of, or one of the Google IPs. And this will then load balance to EchoA, like the, according to the pod selector. So I have successfully managed to, to steal or packets intended for a Google.com IP and redirect that to one of my local pods. This is of course fixed since I'm deliberately running an unfixed version here that to act to still demonstrate this. All right, CRDs at scale. CRD is customer source definitions. They allow you to create custom objects in Kubernetes stored in a CD of the API server, can be created, watched, deleted, are often misused for anything, configuration state storage and so on. Sounds good so far. How does that affect the network? CRDs are often watched. This means that you are watching for changes on them. To illustrate what that means and how that can bite you. Simple example here. Let's assume you have a CRD and it's on average 50 kilobytes in size. That's already quite big. Obviously you can change the math here and you will still be, have a similar effect. So let's assume 50 kilobytes per CRD on average and 5,000 nodes. And let's also assume this CRD is updated by each node every 10 minutes. 10 minutes is not that frequent. That could be like a heartbeat or something or like just kind of a safety update or something. That already means that we have one update per 10 minutes per 5,000 nodes. That's eight updates per second. This gets worse. Eight updates per second times 50 kilobytes times 5,000 watchers, 5,000 nodes. That's 16 gigabits per second. Or two gigabyte per second for the non-networking nerds. This is the amount of traffic. Worst case that if you're running a single API server that your API server has to push out on the networking side. Your API server, the VM running this probably doesn't have that type of network connectivity. You can make this worse. Assume a perfect storm. Demon set updates to CRD on startup. Something that's very common. And somebody for some reason deletes all the parts of that demon set. And let's say that actually takes 10 seconds while all the parts are being deleted. All of these parts will come up at the same time or around the same time. They will each update the CRD. This will lead to 900 plus gigabit per second of network traffic. Obviously, there's no network that will do this. Like you won't be able to find a box doing this. So this will simply mean like your APIs or will be busy just pushing out these updates for a very, very long time and it will stall everything. This is a major cause for scale bottlenecks, often in application code that uses CRDs in this way. All right, last topic here, CNI configuration wars. And this is getting a bit into the weeds, but I think it's interesting to know. First of all, CNI basics. So Kubelet reads CNI configuration from at C, C9, at D or similar, it can be different depending on your Kubernetes distribution. Often, CNI plugins are deployed as demon sets and they drop in their configuration file. For example, 05-cylinderconf. First file in alphabetical order in that directory wins. So Kubelet will take that file, read that configuration and then invoke that CNI plugin. So for every part scheduled, it will do a CNI add. For every part before it disappears, it will do the CNI delete. And very important, the node is not ready until a CNI configuration is found and valid. This prevents parts from getting scheduled before a node can actually provide networking. Sounds simple, sounds cool, sounds easy. Let's look where this can go wrong. So first of all, the uninstall leftover surprise. So as we learned, CNI plugins typically drop the CNI configuration as they get deployed. It can be done with a post start hook or any container and so on. But CNI plugins cannot remove the CNI configuration on pre-start or in some other way, because if they would, then every time you would restart a CNI pod, another configuration file in the CNI directory could win. So if you would schedule a new pod while your primary CNI is being restarted, that pod would get scheduled with a different CNI. This is, you don't want this. Most likely that pod will not have network connectivity. So this means that CNI plugins will simply leave the configuration behind when you remove them. So when you just delete the pods, the configuration will leave behind. This means in a lot of cases, if you uninstall a CNI, you basically leave a non-functioning networking in your cluster behind and you need to fix it up manually. The bootstrap race. So users deploy a CNI by a demon set, typically with system node critical, so they get scheduled first. But another CNI plugin is already pre-installed. This is basically the standard on any managed Kubernetes offering. Because of that pre-installed CNI, the node is immediately marked ready. So pods can schedule right away. The CNI of the, or the demon set of the CNI that you want then races to be scheduled first on that new node. It will probably win because of system node critical. But if that race is lost, the intended plugin or like your primary CNI plugin will never see the CNI ad. So again, the random new pods will have no connectivity. Bit of a bonus complication here, scheduled doesn't mean running. So even if the CNI pod is scheduled first and starts running first, it's not guaranteed to run to the point where the CNI configuration gets dropped or written to disk before the first pod is scheduled. So you might still have a race in there. In the end, the most reliable way to make sure this never happens is to make sure that you have a CNI configuration for your primary CNI written as you want it when the node comes up. Last but not least, the asymmetric cleanup. CNI configuration acts as present, like for some plugin, some pods get scheduled, they start running. New CNI configuration file is written. You then delete the pods to restart them. You know what happens? The old CNI will never be invoked. On delete only the new one will be invoked, but that new CNI plugin actually has no clue about these existing pods. This means that routes, interfaces and other resources are simply leaked. Often this is not an immediate problem right away, but it often bites you a couple of weeks later on because you will have leftover IPs, leftover routes, like laying around in your nodes. Very, very common. All right, so this was a quick warp through lots of failure stories as you have learned. We have obviously have many more, but only so much time. I wanna take a bit of a kind of a summary at the end and cover three lessons learned overall, like and kind of give you best practices. Obviously you can avoid the specific mistakes or like learn from them, but in general I think we have learned three things. Shiny objects are cool and Kubernetes is one of these shiny, cool objects. You can build fantastic stuff with Kubernetes and it's awesome to build it and it can be highly effective, but keep it simple, like don't over complicate it. In moments of complications and in moments of disaster, simplicity really helps. I think that's tip number one, it's obviously very generic, but I think it's in particular true for networking. Second one, maybe even more important, visibility matters. So connectivity itself is not enough. Like when we talk about networking, we often talk about benchmarks and connectivity and maybe latency. Yes, that's important as well, but what really matters on day two and beyond is to have visibility into what's going on in our work. We've quickly looked at Hubble, that's only the tip of the iceberg obviously. There's many other solutions out there as well. In general, don't just look at the connectivity angle. Don't just look at maybe some shiny benchmarks, even though that matters as well. Look at the visibility. It will really help you when issues hit the ground. Last year, I think this is a little bit biased, get yourself some superpowers. Asylum is not the only project using EVPF. There's more and more projects in the cloud native space leveraging EVPF and it's leading to innovation that is truly awesome for you as a user. It leads to Kubernetes native solutions where instead of mapping this awesome new shiny Kubernetes objects and concepts into all technology to really map that into something that is intelligent and that can benefit from these concepts the most. If you want to learn more about EVPF, I highly recommend you dig into one layer deeper, learn a bit about the concepts, check out the many awesome EVPF projects, tracing, security, networking, visibility, observability and so on. EVPF, I always a great starting point to get started. With that, I would like to thank you very much for listening to me and I really hope that you have learned a thing or two. If you want to contact me, feel free to do so on Twitter. If you want to learn more about Scyllium, feel free to go to Scyllium.io or check out the Scyllium GitHub project. Thank you very much.


Mr. Vice President, how many times have you ever spoken to your son about his overseas business dealings? I've never spoken to my son about drug abuse. Ask the right questions. The media has never turned out a chance to smear the Republicans with whatever rumor or half-baked scandal that was presented to them. We've seen them discredit themselves over and over when they prematurely blast out a story that was crafted to fit a narrative. For example, Covington, Jussie Smollett, the Russian spy story, or more recently, the story about a girl who claimed bullies cut her dreads off. But then it turned out that she made the whole thing up. That didn't stop them from breathlessly reporting the story to advance their narrative. Yeah, it turns out she made up the whole thing, and of course the media wasn't nearly as interested in reporting the truth. Which is exactly why we call them the drive-by media. I often pose the question to people on Twitter, when was the last time you remember our media pursuing a Democrat scandal? People rarely respond, but if they do, they always point to Clinton without realizing that the media defended Clinton. They attacked the whistleblower, and they smeared the independent counsel that was investigating Bill Clinton. When someone drives as one House Judiciary Committee member put this some weeks ago, a truck bomb up to the steps of the Capitol and just dumps it on them. Now, this is politically a very violent action for Ken Starr to leave this on them weeks before an election. I'm looking for a moment that the Starr report is a balanced, judicious presentation. It's not. It is a partisan prosecutor with some zealous aides who's trying to make a case against a guy he despises. Did this report have to be that detailed, that explicit? I mean, you cringe when you read it. Does Congress need it? Do the American people need to hear it? Should the president be subjected to that kind of embarrassment? What is Ken Starr do now? And do you think he'll be investigated? This report, it is so salacious. It is so graphic. There will be many people who will feel it's disgusting. Wonder what they'll tell their children. It could be a backlash against Ken Starr. It seems that when a Republican is in office, the media loves investigations and holding truths to power. But when a Democrat takes office, they change their mission to defending that president and discrediting their critics. Even though we have a Republican administration, we're seeing that strategy play out right now. As negative attention has focused on the Democrat candidate Joe Biden, the media has turned the tables and began demonizing and discrediting the critics. Instead of digging into this story and holding truths to power, they're simply circling the wagons and attacking anybody who speaks of this scandal as some sort of crazy conspiracy theorist or just plain wacko. They're not only dealing with impeachment, but they're dealing with this new line of attack regarding his son, which of course has been Disproven. Disproven. Repeatedly. Repeatedly. Disproven. Disproven. Repeatedly. Repeatedly. Repeatedly. When? It needs to be investigated before it can be disproven. We've heard left-wing activists in the media repeating this claim over and over, but we still have no idea why Joe Biden's son was given a position on the board of an energy company for which he had zero experience and paid $600,000 a year. Are we supposed to believe that it's just a coincidence that he gets this position while his father is vice president? Are we supposed to just move on when that same father uses his office as vice president to get the prosecutor who was investigating that company fired? Wrong. What makes things even worse for the media is that when Joe Biden was asked about this curious arrangement, Joe Biden lied and claimed that he had never spoken to his son about the company. Mr. Vice President, how many times have you ever spoken to your son about his overseas business dealings? I've never spoken to my son about his overseas business dealings. Ask the right question. We now know that's just not true. Hunter admitted that they spoke about the company during an interview in the New Yorker. Not only do we know that Joe did talk to his son about this company, but now a new photo has surfaced showing Joe, his son, and Devon Archer who served on the board of that very Ukrainian company. Biden lied, yet there seems to be zero interest in the media to hold truth to power. Or how about Joe and Hunter's dealings in China where once again we see Hunter getting paid big bucks and sitting on the board of a company where he has zero experience. No smoke there, not at all. I mean everybody knows that no serious journalist would want to look into this story. How do you do that?  That's funny because I remember MSNBC, CNN, and the other networks claiming that Trump called immigrants animals in an infestation. That's been debunked since 2017, yet keeps appearing and reporting in the media. I'm still wondering what facts have been disproven. Joe Biden lied. That's a fact. His son was getting paid big bucks to be on the board of a company he had zero experience in. That's a fact. How many of you have gotten paid for a job, much less the big bucks for a non-entry level job that you had no experience in? Why was Hunter on that board and what part did Joe Biden play in that? Any serious real journalist would ask that question. But these fakes only seem interested in deflecting and burying the story. You can find the same sort of horrific clown show over at CNN where they brought on disgraced liar Andrew McCabe to bash Trump and his supporters who might have questions about Joe Biden's dealings in Ukraine and China. And keep in mind, this is a guy that has somehow, up to this point, avoided prison time for lying to investigators. You were one of the top officials who made sure the Russia investigation was opened. You were at the top levels. They're investigating you and your colleagues. Do you find Bill Barr's travels problematic? Top levels. They're investigating you and your colleagues. Do you find... As she's talking about McCabe's role in opening the erroneous and now disproven Russian collusion investigation into Trump, the light on him fades and he seems to turn to a black shadow. How poetic. Is Bill Barr doing this for the sole purpose of pursuing Trump's conspiracies, debunked multiple conspiracy theories being spread by this president and his allies in the right wing media? What a surprise. The Democrat related scandal with tons of unanswered questions is simply a conspiracy theory. The now thoroughly debunked conspiracy theories about Trump being a Russian agent and stealing the election still apparently a topic to constantly reference and keep alive. This is just more proof of what we're up against. The Democrats in their media will stop at nothing to make sure that Trump either loses the election or never even gets a chance to run. And that's kind of scary because if these people get power back, watch out because they're not going to stop at Trump. They're going to come for all of us. If you want to help me in this fight against the DNC media, please hit that like button, share this video and subscribe. If you want to help this channel survive and grow, consider subscribing to me on Patreon or Subscribestar. You can also send a donation over Paypal. You can find all the links in the description or the pinned comment. Thanks for watching and keep coming back.


Hi everyone Joe for jazby's casebreaks.com coming at you with a nice one here 2023 Panini impeccable football three box Pick your team three Lot of great stuff here a very big. Thanks to This group right here Thanks everybody who got their team straight up Appreciate it Jason K.  Oh Joe with the Ravens Before I pulled the Panthers and Colts for that random number block. We did the We did that randomizer in this different video. There's everybody else right here. Thanks everyone for getting in By the way, there were a couple of you Maybe Ed and Aaron maybe When we initially posted pick your team three we said 30% off we intended 30% off But for some reason the first couple teams picked up were only 10% off So I sent you the difference as a gift card code break credit to jazby's casebreaks.com Hopefully that's okay. Everyone else. I think the 30% off applied correctly All right, there were the results of the serial number randomizer. There it is Set that aside next to me right over here and Good luck, and I did not see any more impeccable football in the back So I think this is it for now Until until we find some more Replenish our supply of blank cards. I like that Or are we gonna lead off with a nice base card here? I think so really uses blank card with the jazby stickers on front Right there Team list right here ready to go It is nice nice Anthony Richardson to start things off to 25. That'll be for the Colts. That's a number block team So that'll go if you're not familiar with how the random number block works. We go by that digit right there So number nine that'll be for Greg So Greg will get nine out of whatever 19 out of whatever 29 39 49 59 6 9 etc. Etc. Etc. Etc. We got Russell Wilson 46 out of 50 Where does he end up now as a Raiders fan? you know if he ends up Getting released Denver eats the money and if he signs for the vet minimum I wouldn't mind the Raiders taking a flyer on this guy that goes to Paul in the Broncos Alan's taking jets Back up Aaron. I think he I think Russell Wilson wants to be a starter though. He still fancies himself a starter We'll see what happens. It could be a crazy off-season 7 out of 75 Isaiah Pacheco We'll go to Robert and the Chiefs We've got a four out of five Michael Wilson dual relic and on card autograph That's gonna go to Kevin and the Cardinals Kevin. I don't know if this is gonna fit in a 180 top loader Check first might have to be a slider box, or is it a perfect fit? I'd be a perfect fit actually It's a perfect fit 180 top loader and Out of fives and under get the train whistle Kevin All aboard the big hit Express Nice Who's Russell Wilson he wants to win two more rings in the next five series Cortland Sutton three-color patch autograph 34 out of 75 another Bronco for Paul my rivals Russell Wilson said that Duncan. I mean I like the confidence I mean, I suppose if he ends up in a right the right situation. He could be Here is Brian Dawkins. God bless. Thanks, Brian 14 out of 25 Aaron with the Eagles to fly Eagles fly. I mean he might stumble into one To in the next five years. I mean that's so I like his confidence Here's Sean Clifford 32 out of 99 Landscape rookie auto for the Packers that'll be from Mark Lance and Back here be John Robinson. Nice dual relic and autograph 35 out of 35 nice RPA We've got Justin Beard picked up the ATL straight out. There he goes Justin Right Duncan. There you go. Yeah, Russell Wilson might might win a couple Super Bowls in the next five years as a backup Did he say he wanted to do that as a starter if he did not clarify then I guess that would still count All right, there's the case it a silver piece some fine fine silver I think they still have gold right golds are one-of-one, but the silvers are obviously far more common We'll do that last Looks like we have oh, I thought it was a redemption hiding. There is not a redemption hiding there Okay next box. Good luck everybody There's Justin Jefferson 29 out of 50. What are the Vikings do at quarterback? I'm sure if they're terribly thrilled with the rookies that they have there Might just be good backups, but maybe not starboard quality. Do they resign Kirk Cousins? Is he feeling better? Who knows that's gonna be for Douglas in the Vikings Antonio Gibson 44 as 75 commanders will most likely have a new quarterback Brett with the commanders We got DJ Moore 22 as 75 will he have a new quarterback throwing 10 a lot of quarterback questions Ed has the Bears He did have a good connection with Justin Fields. I feel like this year They decided to go Justin Fields. What if they get Marvin Harrison, Jr. On the other side of DJ Moore some offensive line help stick with Justin Fields Who knows that could be the winning formula Got more Broncos Clinton Portis 43 I 75 And that's Paul with my rivals the Broncos for the Jets we got Joe Nameth who just Who just puts Broadway on there Three out of 15. Thanks, Joe. What a what a character Andy Owens with the Jets You think he did that for all these cars all 15 maybe we might have to send this back to Joe and be like Might want to put your name on there too, sir. Mr. Name it. All right, then we got Dorian Thompson Robinson 96 out of 99 for the Browns Jared And then back here is 10 out of 25 Zay flowers nice the the end of Ravens Jason K last spot last Bob Mojo, man 70% of the time last spot Mojo hits 100% of the time We might have to up that to 80 Nice. All right Last box. We've had a solid break. We could use a bit of a We could use a bit of a monster here, I think Solid but we know we know what the ceiling of impeccable can be Jimmy saying Russell Wilson won't win without an elite defense and a running game Two things that the Raiders have maybe not elite defense, but it has improved significantly last year And they've got a pretty pretty good running game Josh shake-ups a mirror white Admore running Okay onwards Got Curtis Samuel 46 out of 75 for Brett and the commanders Devonta Smith 31 out of 50 for Aaron and the Eagles and he guesses on that redemption Hey speaking of my Raiders rookie autograph Michael mayor 14 out of 49 Tight end for Russell Wilson throw to I don't mind Russell. I mind the contract But if he signs for a vet minimum You know, I take a flyer Who's got my Raiders that'll be Ryan Hanks with my Raiders. There you go. Really like this guy Michael mayor We got a tank tank bigby eight out of 35 landscape rookie auto for the Jaguars Matthew with the Jags 28 out of 49 nice break for the Broncos Paul the Broncos. That's Marvin Mims There's Puka Nukua 7 out of 99.  We've done three breaks of court Kings today And we added some more to the site as well Jasmine's case breaks on com. All right, no one guessing the redemption, huh? Well guess window closed I wasn't gonna give away a little break credit. We got elegance rookie helmet and team logo auto K E And Kendra Miller who I think is a Is a saint. Yes Alan with the saints when the saints go marching in There you go ain't There you go. All right. Let's see what we got in here. Good luck, everybody Joe Montana Joe cool 10 out of 25 The Hall of Fame Stamped into that Troy ounce of fine fine silver That'll be for Ed P and the 49ers Very nice 10 out of 25 Excellent. What a break ladies and gentlemen impeccable really solid stuff. Thanks everybody for getting in on it That was pick your team three. I'm Joe. I'll see you next time for the next football break. Bye. Bye


Hello everyone, welcome to the Jenkins Weekly Infrastructure Team Meeting. We are the 1st of August 2023. So today, round table, we have myself, Damien Duportal, Hervé Le Meur, Marc White, Bruno Varton, Kevin Martin, Stéphane Merle, six of us are there. So I hope everyone is doing good and let's get started. So the weekly 2.417 is out. At least the package and the Docker image will be published soon. I've created the tag and the Docker image soon. I just had a health attack before because when I published the tag, I didn't saw the updated version on Jenkins.io. I believe it's the Fastly cache unless I missed something, but there might be something to be careful about. No, Jenkins.io hasn't, the changelog hasn't been merged yet, we're a little bit behind there. The thing is, I took that the page, there is a Fastly uncache during the packaging step in the release process that should update the version here on the download page. I, if there's no changelog entry, so if you, well, at least my experience has been, unless there's a page regeneration event like a commit that arrives on this site, we don't get new content that they can update. Okay, that explains. Okay, so it's out and we are waiting for the changelog as usual, as usual. Right, is there anything else on that release? Oh, okay. Announcement, we had a security release last week. It was announced Tuesday, so we already discussed that last during the meeting. We had a combination of core release, core security releases for both LTS and weekly last week. Just a reminder, so that's why we had to cancel last week's weekly release because it was part of that security release. And next week we have the weekly meeting, but in two weeks, if I'm not mistaken, that will be the 15th of August, which is not the working day in front and some European countries. So my proposal is that we cancel the 15th of August meeting. Is that okay for everyone? Yes. No objection? Yes, no objection. I'll be out of the office, not for assumption day, but I'm out of the office. The French people here cannot object because they won't be walking, of course. Weekly meeting cancelled, but I assume we will still have a weekly release as usual. Is that correct? Still up? Yes, yes, but that means Kevin will have to deal with merging the change log and adapting it because I won't be available. Just in case, Kevin, I will be around because I'm not expected to work for my day-to-day service job, but I will have a task to run on the internet. I will have my machine, so I will be around if you have an issue or need any help that day. You can absolutely ask for help if anything is wrong or if you have questions for that day. No worries on this one. I will be in front of the computer, that's why. Sounds good, Daniel. Thanks, I appreciate it. I think it should be okay merging and everything like that, but yeah, I'll keep that in mind for sure, but hopefully you enjoy your time. Cool, thanks. At least I will need to be there for creating the tag on the docker release unless someone is able to automate that until then, but no emergency. That's all for announcement for me. Do you have something else on announcement folks? Nope, okay. Let's have a look at the upcoming calendar. So the version 2.418 Jenkins next week will be the 8th of August. I already forgot the next LTS, as usual. Cool, let me search. You don't need to search 2.414.1, releasing 23rd of August. So release candidate 9th of August. Let's see, August, I didn't. Release is 23 of August. Did I catch correctly the version number? You did, exactly, 2.414.1. Thanks, Marc. Something else about upcoming release? No, the security backports are already done, and there was just a regression reported today in the UI that I'm not sure if a backport will be done or not, but certainly we're watching actively for any surprises or bugs that might justify backports. Cool, thanks Marc for the detail. No plan security release and no next major event as far as I can tell. Anything else on the calendar for you folks? Cool. Okay, so let's jump on the topics about the tasks that we were able to finish. I haven't had time to sort or classify the items, but let me know that first we had a bunch of issues where a contributor failed to log in to Artifactory or to manually release their plugin, and they received a 401 HTTP error, meaning no authentication is either not provided or wrong. And the symptoms are the same for everyone, including our Marc Wait here. When you log in to AccountStream, Kinsayo, or LDAP, or JIRA, or any services, it works. So it's not you, you're not crazy, you didn't misconfigure your Maven settings. It's because there is an issue that we still don't know the root cause, most probably a change in the Artifactory configuration that we are not aware of. But Artifactory tries to compare the password, the LDAP password you are passing as credential, to a local password, which is either empty or a generated random value. We don't know why this happens. So we are working, there has been a new issue. We had a lot of these issues right now. We have a bus factor in order to solve that problem, you need an Artifactory administrator. So Daniel Beck and I are administrator for sure. So I'm the bus factor here mainly, given the low availability of Daniel. The trick is there is a setting on the user that has to be set to disable internal password, which forced Artifactory to request password check on the LDAP. That was the default behavior. So I've opened an issue to track that. We have to contact GFrog to ask them why did it suddenly happen? Did they have a recommendation? Right now I'm fitting to changing when a user opened an issue. So if you have that issue, please open on the LDAP. I've started to search how to batch update the users. But the problem is any new user created have the setting disabled. So the batch won't fix on long term. Any new contributor joining Artifactory will have the same problem sooner or later. So we need to ask GFrog for a solution. Maybe that's a feature request. So we can define a default behavior.  I don't know. But right now it's annoying. And we will have to treat the user one by one. So sorry for the inconvenience. If you have an emergency issue, please open an help desk issue. Is there any question, suggestion, clarification, objection on that topic? No, okay. So thanks for the person who triaged the issues. Next issue, plugin not found on plugin site. So we had a contributor who created a plugin and they were really eager to see that published on plugins, which happened after two or three hours. The reason is when you publish a new plugin, you have to wait for the build to kick. The build runs every three or four hours, as far as I can tell. So their plugin has been is now visible. Thanks to the manual built trigger by Alex. Thanks, Alex. No more issue on that topic. It wasn't immediately done by the previous build because there has been a 502 error during the plugin generation. We have to diagnose that part. An issue has been opened by Alex on that topic. So thanks for that. So the user problem has been solved. That's why the issue is closed. And we'll have to work on the HTTP error. A long running issue, artifact caching proxy is unreliable. So thanks, Basile and Dere for completing that task. So now the acceptance test harness is using the ACP caching system. And we haven't seen any regression due to that system. So thanks for that work. That confirms that the network changes we did are not suffering the overlap issue because we haven't seen any TCP connection error on Azure since at least a few weeks. Any questions so far? Objection clarification? Nope. Okay. So let's continue. Stefan, can you give us a head up on the not toleration and taint for IRM CPU not pool on Azure? Yes. We did create a new not pool with a taint and not selector enable to make sure that no non IRM pods were scheduled on that not pool. It went smoothly and we checked that everything is working fine with Javadoc that we moved from the old IRM 64 not pool to the new one with the taint and it worked nicely. We also had to deal with Datadog and Falco to make sure that that pool got those two demon sets and we just had to use the taint toleration toward the toleration for the pods to be able to spawn on that. It's working fine and I just added and move migrates the reports dot Jenkins dot IO from Intel to that not pool IRM 64 as it was also an engineering pod so it was okay. That one is another issue so no worries. I'm not sure there's an issue. There is another issue for migrating workloads these are two separate topics. So thanks for the work two additions. So as Stefan said we had to update the demon sets that we deploy Datadog and Falco. So we have agents of these services running on the new machines even despite the taint supply to the new not pool. Nice thing is that all the Azure internal components you see the cube proxy here but there is a bunch of Azure services used to mount persistent volumes. All of these systems were already working despite the taint so it looks like they really did their their homework properly and their managed service are really working as expected. That's really nice on AKS. Please note that's not the case on Amazon so that's why I'm I mean how in the with the user experience in Azure I'm really happy with AKS so far. And the last one the funny one is that Datadog has a second components named cluster agents which is an highly available service that we run on each cluster and that one has been running one of its replicas inside IRM while the other replica was on Intel for month which wasn't a problem that showed that everything was working properly but as an additional proof that Stefan did his homework properly with the taint now that component is fully working on Intel as expected. So thanks for the work Stefan, thanks for the reporting on that issue. Next issue, Jenkins server enabled to download plugin from the updates Jenkins site or website so I guess that was a user level network issue. Yes okay so that person was facing issue for accessing one of the download mirror they were redirected to. So thanks Mark for completing the issue there wasn't anything on our home but at least we had to check that the mirror wasn't done which is not the case and our redirector wasn't done either which is not the case. We haven't heard back from the user but there is nothing else we can do they have to contact the ARCHON university which was the mirror because ASER the network is denying requests to that server or the university denied requests from that network. Well thanks for reporting sorry about the inconvenience you have to contact them or change the mirror. Next we had 88 build failing due to denied outbound requests during tests so that one was an old issue I forgot to close two weeks ago so I closed it. We are allowing external requests to the internet from the Jenkins CI agents only for outbound HTTPS and now HKP protocol which is used to retrieve public GPG key which is done by a lot of the Docker files inside the ATH and the request from them. We finished the reporting on the last miles of the Kubernetes 1.25 update so the issue 1.26 for Kubernetes update has been created. I need to do a last path but most of the changes we did on the two past Kubernetes upgrade have been reported to the task list for the upcoming 1.26. The deadline for 1.26 will be October so we have a bit of time ideally if we could target under August or September that will be better. Any questions so far? Clarification? Objection? Okay I'm continuing. We had three account issues so person unable to have their account. Some of these requests were due to either we have two main issues on account Jenkins IO today if I exclude the people who tried their Jenkins account inside our platform. First we have the person who had an old account they didn't use which was part of the security problem with the LDAP in 2021 and this account as part of the security response had their email disabled and set to the username so when they try to reset their password they cannot. It requires a manual intervention. For this user's reminder, don't blindly change the email immediately. Check that the user isn't the plugin maintainer or did not already worked with Gira. Most of the time they already commented on Gira so you can confirm that the email is the proper one. Yes Mark? Okay so just for precision when we had that account outage we did not alter anyone's email address. What the problem was they had set it themselves to that flawed address and when we attempted to do the notification they could not be notified because we had no email address for them so we actually did not reset anyone's email address as far as I know when we were dealing with that incident in 2021. Okay so that might have been a 2020 incident then. One of the LDAP security incidents before I was able to act on that area Olivier did a batch on the LDAP to remove this email. Oh okay so I didn't remember that but all right. So I have the date wrong you're correct that's not the 2021 all of these accounts haven't been used since 2019 or 2018 in that case so that's why. Okay but thanks a good point Mark and so probably one the takeaway is when you see such an account without a valid email double check the account on Gira because this user most of the time connected to Gira a few years ago and Gira keeps a proxy user with the proper email or you can also use artifactory or all let's say all GitHub issue you can find the user but don't blindly use the email asked by the user to avoid any unwanted takeover. Well and thanks for your point on Gira as a cached copy I had forgotten about that that's a really good place to check if if they don't have an accurate email address or they don't have a useful email address on accounts.jankins.io but they do on Gira like you said we can go to that one and use it and trust that that was their address at one point in the past good thanks and the other kind of issues are users that are trying to create an account from a public IP aggressive public IP which is in our deny list so they are classified as spam and we can see these errors inside our data dog monitor so in these cases I tend to create the accounts manually I don't know if a user really are really doing what they expect but yeah that doesn't kill. Any question objection clarification oh okay cool. I closed us non-planned a request from a user who reported an invalid certificate for that's the generic domain name we use as the main entry point for the public IPv4 on our public cluster. All of the services are consumed with a domain name such as get jankins.io which is most of the time is a C name to that sub domain name there is no reason for asking for a certificate because there is no reason for accessing that's domain name so I've asked the user never answered one of the most probable reasons is that this user might be running inside an organization with let's say some kind of appliance of security appliances that tend to not respect the HTTP host protocol headers most of the time they for instance they are case sensitive while they shouldn't as far as the RFC which when they have the CNM redirection breaks the HTTP redirections most of the time or worse it's their DNS that doesn't follow RFC in any case there is nothing we can do and we shouldn't publish any service behind that domain name so since we didn't have a use case because there might be a use case that will require that in the future I'm not an absolute person but right now without any problem statement I decided to close it as a not planned and no action required from us if there is an objection or if I forgot something don't hesitate we can reopen with a problem statement and we might work on this is there any question need for clarification on this one no okay so let's continue with the work in progress issues every can you give us a status on the migration of the update center updates to another cloud and AWS as it is today we discussed it together and we came to the conclusion we should use the mirror bit charts so we get an apache to server and mirror system already include we want to use an apache to server and not engines for example because the update center is generating a lot of access configuration file we have to keep so the goal here is to mount these the access into the apache to services and add the redirection HTTP redirection to the bucket on cloud flare we might have to put the main name with cloud flare to be able to use it for enabling public token buckets but it remain to be we have to check that okay be used for redirecting to cloud flare hosted error to buckets so reminder that means each time a Jenkins server will request the update center json like today on updates Jenkins ios slash whatever depending on the URL you are eating you might have a first redirection to the json file itself and with that new architecture once the request goes directly to request the json file as survey stated you will be redirected to a cloud flare public bucket that will be on another domain name but still HTTPS valid that will take care of handling the agress download we will start with one or two buckets which of course won't be stored inside our infrastructure so we don't have to pay for the outbound bandwidth and we benefit from cloud flare infrastructure the second reason of that additional redirection is that then we will be able to set up additional mirrors cloud flare or something else in areas where the network require proximity such as in behind the great firewall of china cloud flare for instance allows us to create additional buckets that could be on the china network that one will be a great help for our remote users is there any other question on that topic or is that everything is clear and we can continue proceeding on that no okay let's continue then thanks survey for the report stefan your turn applications oh yeah you were right there is an issue um yes that's that's uh all the um the services that may uh take advantage of running on an higher m64 in order to to save some cash and uh i did this morning uh this afternoon sorry migrate reports dot Jenkins dot io and it went well yes let's see that a few to look in and to check cool um i believe uh wiki will be the next candidate oh no i'm not sure i would have to check that so we have just a note we have two kinds of services the one that like javadok and reports that already consume on a third party docker image such as the official ngenix image for these two this image on the docker already provide iron 64 declination so for this one it's easy we can just change the toleration selector and that works out of the box for images such as wiki we build these images ourselves so that means we will need to update our docker image build process to support multi architecture if we want them to work on iron 64 yes i i mentioned wiki stefan because i was thinking about ngenix because it's built on top of the official ngenix image so we may be kind of easy to have also then i am yes i i remember that so that's the idea so thanks stefan so you will need to find the other services that we don't that are using images uh we know that about cluster agent for instance yes um but yeah i don't have anything else to add on that topic is there any clarification question on this one okay so let's continue with jenkin ci failing for jenkin's plugins so the contributor answered after our request so it looks like in their case the plugin repository they are using is an unexpected one because it's not an original repository it's forked from the company's organization initially they forked their repository to the jenkin ci organization that pattern looks like to be of consequences inside ci in jenkin ci in the way we configure ci jenkin ci job scanning when the contributor sumo anema despite being an administrator of that fork inside jenkin ci when he creates a pull request i forgot the pull request which is not from a fork of him then is not considered as a maintainer so any change to the jenkin file is not taken into account in the pull request build that's a security concern that's a security mechanism on building in jenkin ci but here we are on the edge case we might have to update the job configuration to cover this case but it's not an easy one to solve so at least they were able to have to use the method provided by team when you have such case the contributor of the plugin should open the pull request from their own fork not from the either the original external repository neither from a branch on the jenkin ci plugin repository that might be a bug on the github organization scanning plugin but that also could and most probably is a misconfiguration on ci jenkin ci changing this configuration is not uh uneventful because it will trigger a reload and the scanning of all plugins we check two hours and most probably it's the api rate limit with the consequence of booking all those of builds so if you do this please don't do it during a bomb build or ath build so the user described their problem so we will need to find a solution or tell them to not use it and then update the documentation for developers uh if we cannot have an easy solution uh bruno and uh and kevin we will let you know the outcome of that issue in the future so i'm mentioning this to you because you have it under your radar no expectation from you or anyone on the documentation team right now uh but next week we should see more clearly the next steps for this one is there any question objection clarification for this one if we cannot find the proper configuration on ci jenkins io and if not a bug on the ci scanning plugin then we'll have to document this for developers next issue plugin site build commonly fail on infra ci when accessing plugin jenkins io so i haven't had time to look on this one that has been opened by alex we have to to check the build logs and see what caused the error um check the the login Jenkins because it keeps only five builds and okay uh is the build kept because when we see that kind of when i checked before it wasn't really okay so can you add a note on the issue to tell alex if you see another error like this one if you can keep the build so we it won't be deleted by the builder cleaner um i'm not totally sure where to look in terms of access logs to find these errors because plugin jenkins ci is inside fastly uh so i'm not sure if they have access log for that and i'm not sure if it's using fastly because it's a json file i'm not sure if it's static file cached by fastly or if it does it does directly it's the plugins.origin.genkins.io back-end service inside our cluster if it's inside our cluster we will be able to uh to check the access log at the ingress level if it's in fastly i don't know so we have to work on this um it's when it's when building on this one instance that's uh request ardent it's yam who's calling when you were building the plugin inside infra on ci it's calling yarn is calling the there is a there is a gs file called a gs script called by yam when building which uh requests the plugin site for more instance okay i'm not sure to see why it would have an influence you were i i thought you were saying that if the quest came from more cluster or instance we might have more visibility on them that's why okay yeah it now it's more the server side that was uh if we if there has been an http502 issued it's a server side error so we should have an access log somewhere with an error message to point why did it fail to answer the request and that if the error would have been a four or something that would have been client side and in that case yarn would have been the cause but that's not the case and the error appears intermittently so is it fastly issued is it on house i don't know but we need more logs in order to diagnose this one i tried to run a build with plugin.origin that Jenkins.io but it failed because uh plugin.origin.io is hyper restricted yes uh why do you want to use plugins origin Jenkins.io for that it was a test it was a test okay okay that we in any case we could use plugin.origin. Jenkins.io because it's not a big bandwidth it's just from a build but yeah it wants to to use what the end users see and you don't want to create specificities like this one especially when using a cd and in front of your website um yeah so uh because if we have the json file served served on origin and it's a generated file then in that case that means we can have it somewhere else and consume it directly instead of having to use it through the website if you have to do that kind of bypass um yeah do you think we could create a data dog monitor to alert us when that build fails since infra ci has data dog plugin installed and set up and is streaming its data to data dog easy because at least we will be alerted and that will uh that will at least allow us to go and manually kick the build so and user won't be waiting too much for their plugins what do you think like the account app alert i've put in place we can easily target uh patterning uh specific service logs okay i'm not sure if it's a log that we want here it's more build status in the log when the pattern is matching the log of the plugin sites the build of the plugin site we can then create an alert from that okay makes sense but to be quite honest i will prefer a more generic monitor that say a the build failed because the main branch of that one should always be successful if it fails that should be a call for action from the server team but yes eventually having a second monitor with the log pattern will help for that specific help desk issue any question clarification okay so next topic hh builds are become a responsive so that's um we changed one week ago the kind of instances uh used by the hh builds inside saying ti jenkin sayo we switch from spot to non-spot instances which which are way more expensive but the benefit here is that that should fix most of the unstabilities especially when the dreadful message uh i don't remember but yeah could not connect to a name of the agent that message most of the time is because the agent was reclaimed by the spot instance every and i did discovered and so and we reported how to check if a machine was evicted due to spot reclaim which is a reason for the build the job being unstable um what we saw is that there are there is a retry it's retry two time for each steps but most of the time the probability of spot eviction of the second v8 spawn agent is really high so i've asked if we see better stability but i haven't checked myself we should check the history of the builds and also the cost increase per week to see the impact because most probably we will have to keep the hh agents using non-spot but in that case we will have to either change the instance type to have the proper instance do we need instance with that size do we need something with the different cost we need to do the reanalysis so i propose that we start with metrics and based on the metric we take a decision on the upcoming week that we will report on the issue most probably we will have to keep the non-spot the non-spot instance is there someone who want to work on this one and is there a need for clarification as well okay i will take it then we have to measure success build success rate success rate without spot and cost impact because right now as far as i remember the the cost was 10 time cheaper using spots so right i would be inclined to say let's retry four or five time until it work but not sure if it will change the global build stability it will increase most probably the build time when you are in a spot spot eviction time window but at least the build will continue until it works maybe we will have to wait a bit more time next issue assess artifactory bandwidth reduction option so there will be tomorrow we have to announce officially but the the time window has been fixed by mark and high because will be bus available so tomorrow Wednesday the second of august at 2 30 p.m utc there will be a brown out of more or less one hour on artifactory where we will disable and remove any virtual repository pointing to the maven repo one artifacts the consequence of that for us will be that the maven builds inside cnc in saio inside a g it abaction for the cd process and for any contributor will fall back when not finding an artifact directly on the maven central repository the benefit of that operation is that we will be able to test if we can build most of our biggest project without having to mirror these artifacts the brown out is a one hour session and then we will roll back and we will conclude if we saw catastrophic failures or if it worked as expected jfrog last week during our meeting validated that principle because that that should allow us to gain at least 10 to 12 terabyte of data not consume but that instance so that will help a lot and jfrog conclude that if that's work and if we can persist that setting they will be okay for us with our bandwidth consumption after that change that mean no need to go further unless they require that but right now they were quite happy with that change and as they told us if that is implemented on the low and take its premise then that should be okay that doesn't mean we don't have to track the other mirrored repository because most of these mirrored repository have the artifact store sooner later on the central repository such as gig it for instance but at least this one is far far away from the other so the brown out that will be the first edition and then we will make conclusion most probably that brown out if it takes his premises we will plan another brown out in two weeks that will last at least one full day and finally if everything goes as expected we will plan persisting that's that setting in the future is there any question on this one okay please note that the immediate consequence is that there will be soon later today or tomorrow morning last last deadline a change in the settings xml of our cagin kinsayo acp systems the builds will exclude acp from being used when trying to reach to download artifact from what any repository which id is central that should not have any impact now but during the brown out it will be heavily used that means any artifact downloaded from the maven central and not from our artifactory inside xml kinsayo won't be cached at all and they will be redownload every time that might increase the bit some of the builds we will assess the consequences of that later the priority is to decrease the outbound bandwidth on the artifactory and then we will think about a way to cache this artifact either using acp or another method no more acp for central artifactory for now any question clarification objection okay next step just a minute i heard the word noise in my house that's the cat who blocks something sorry i had me worried uh next issue done great to http when there is no trailing slash so our security officer um following a security report communicated to us that when you try to reach jenkin sayo with a suffix slash something there is a two http redirection the first one that goes to http which then redirect itself to https again we don't want https to http done great connection uh the setting is in fastly i don't know much more so we need to fix this one it's not emergency but that's important enough in terms of security so we should find a solution for this one it's not blocking anyone it's just security reporting so we have to look on the fastly setups and most probably change uh an attribute on the terraform fastly project any question okay uh any volunteer is uh welcome to check this um another issue during the security release that one should be easy to fix uh since the migration of the trusted NCI controller from one VM to another the new VM has uh not a localist name of controller for both machines which leads to human error and mistakes because you think you are performing an action on one of the machine while in fact you are connected to the other machine so the goal of that issue is to help the human operator that we are and that security people are to avoid uh this problem so at least on the the common prompt we want to have the full qualified name that will be controller that trusted that c i jenkin sayo or private controller c i jenkin sayo and ideally if we could add uh custom colors at least for the root user that will help so that might be a bit of linux and a bit of puppet to purchase the linux setup uh that's a quick one uh but that will help a lot the security teams is there any question or people who want to try this one i will probably try yeah okay it's okay for you absolutely stefan is volunteer uh disallow issue creation in event project type that's me being late it's a gira admin uh a step as far as i can tell i haven't looked uh on this one in detail so no expectation here i need to take it or any other gira ad aws decrease cost for summer 2023 uh that's an epic top level issue that we keep on while we are walking that's our top priority uh half most of this work is done by rv with the date center the rest is taken by me around the pkg machine migrating to azure uh we'll discuss about that on the boom 2022 another linux foundation issue mark is taking this he's exchanging for tickets with the linux foundation so no action required from the team but we keep it on the milestones matomo github repo no member of the team was able to work on this so let's delay ubuntu 22 upgrade campaign uh for that one i've started checking pkg genkin sayo which is still on the same virtual machine that we want to get rid of as update center um most probably we will start with only an apache service based on the mirror orbit charts that rv mentioned earlier which serve static files and that's all that service pkg origin genkin sayo uh as the package indexes which is then procced by the fastly cdn so that service has a low usage but is really really important and so the goal is to move it from aws to azure cluster so then we should be able to not depend on a remote virtual machine for generating packages during the core releases so work in progress on this one the most complicated part for me is to find every core release steps that interact with that machine and there are lots not mentioning that also the plugin updates is also interacting with that machine sometimes for update center sometimes for package sometimes for both so right now i'm trying to access the different elements and i will report with diagram here finally last one no work was done on remove ip restriction on bounce or migrate to vpn i've created as we will see with the issue to try each the issue about third ci cleanup and the whole network cleanup we are inside that area for now i won't be able to work and i don't think none of us will be able so i propose that this issue is delayed of at least two weeks no objection okay i've i've bored everyone so now let's have a look at the issue to try age do we have new important issues do you have topics that you want to mention new issue new things okay so i'm selecting try age issues yeah we have a lot of try age issue i will only look at the last one um artifactory the one i mentioned at the beginning of the meeting so what should we do for the artifactory 401 issues


What's up, everyone? It's your girl, Kenzie here from Layton Sports Guards, and I'm ripping a personal for Alistar G. We've got a 2022 Top Star Wars Chrome Galaxy Hobby. All right. Good luck. First Star Wars of the day. See a sketch. Do you follow Mama Breaks on Instagram? I think so, yeah. She posted a video the other day. I guess she had a booth and she had an open box of Galaxy, you know, and some lady and her husband came and bought one pack. They called me one of one Super Fractor Darth Vader from one pack. Really? Yeah, it was, yeah. Starkfighter on the gameplay refractor. Luke Skywalker on indoor refractor. It was crazy. I'm happy with that. Yeah, insane. Got Darth Vader. This was the Super Fractor of this one. Yeah, wow. From one pack. Of a loose box. Yeah, that's insane. God didn't mean to intrude refractor. There Obi-Wan Kenobi and Boga. Omega refractor. That's refractor. Wedge and Tilly's refractor. Refractor. You've got Luke Skywalker. Assembles his lightsaber. Let's see some color here. Got an auto coming up. Got Yoda and Luke Skywalker refractor. Back. Auto. It's going to be blue of Cursaiton. 39-150. Kerry Jones as the voice actor. On the blue. There we've got Bush disguise refractor. A card game of chance refractor. Come on, let's heat up here. And some number cards and parallels. We've got atomic cracked ice of Sand Trooper atop a dooback. 75 of 150. That's a sweet looking card. Ben Kenobi refractor. X-wings in formation refractor. Vintage refractor. Bounty hunters. Answer the emperor's call refractor. The smuggler Han Solo refractor. Jedi outcast on the gameplay refractor. Patrolling the cantina refractor. Lando Calrissian refractor. Last pack here. Princess Leia Organa refractor. Alrighty, Alistar. That will do it for your galaxy chrome star wars. Thanks as always for hopping in. We'll get these right out to you.


All righty, fantastic. You are still watching Hashtag Hawaii in the morning right here with me, Brian Sarko. Remember, the segment is all about entrepreneurs and Hashtag Entrepreneurship Tuesday. And we bring you a lot of great conversations and insights on young and upcoming businessmen who have managed to be in the field and gathered some few lessons here and there to help you back at home. The person who has a vision of starting yourself a business for yourself, that is, you will find all the hacks and all the tips right here. Well, away from that, you can find us on our social media platforms. Facebook, Instagram, and Twitter is at Hawaii254Channel. Remember, on Instagram, there's an underscore and Minds. Why do I keep on saying Minds? Anyways, I'll find another term. Minds is at Brian Sarko 101. And joining me right now on set is a very powerful gentleman. I always say powerful, okay? An incredibly powerful gentleman from AISTO who will share with us the journey and the times of why did they venture into AISTO and literally why literally upper products. And without much further ado, I'm going to let him introduce himself. Good morning. Thank you so much for coming through, bro. Good morning. You're welcome. How are you? My name is Jeff. I'm a representative of AISTO. AISTO, we started in 2019. Okay. So, AISTO meaning it's Apple products. Yeah, we deal only with Apple products. Okay.  No, no, no. I'm on behalf of AISTO team, the whole team. Okay. So, meaning it's a big team. It's a whole team. Okay. Why Apple products? Apple products, first of all, Apple products are, it's a global product. Okay. Yeah. And in terms of markets, the prices for Apple products are dictated globally. So, sorry, sorry, sorry. They're prices are dictated globally. Okay. So, they're not dictated by the local market. So, it cuts across all types of, all our target customers. Okay. Yeah. There's no inflation or, yeah. Okay. When you speak of Apple, it's one of like the major brands as compared to like Samsung. A lot of people would want to associate to themselves with Apple products. Even from the iPads to what are they called, to the MacBooks, to the mobile cell phones. And right now, if you check out the market, it's quite a competitive market from even the advancements from the fast, fast, fast, fast Apple products. What were the fast Apple products, by the way, from your experience? Because you are handling the brand. Do you have like maybe a license? Did they issue with your license to maybe sell their products? Yeah, we started as taken. Okay. And then we majorly focused on the Apple products. Now, we were branded to iStore. Okay. Which was early last year, around early last year. So, yeah, now we were branded as authorized resellers of Apple products across, let's say globally. Okay. Because we've got across East Africa, countrywide, yeah. Okay. If somebody back at home maybe who's watching and they want to venture into Apple products, for example, the issue of license, because I understand it's a very stringent and competitive market. How do they go about it? It's a simple process. Just go to the website and with the right documents and back up your documents with what you do. Just be genuine and it's easy. Okay. Yeah. Okay. I've seen on your profile, you've mentioned about tradins. Yeah. What does that mean for a person who's watching and they are interested to know like, your business allows people to maybe do that kind of a business away from like your normal business. Yeah. Tradin is basically a concept where you bring the phone you're using to get the phone you want. Okay. So you bring your phone, we check it via our softwares, then we value your phone depending on your condition. Yeah. Of course, with authenticity because there's a form you have to fill and you have to produce your eligible documents, identity documents, either passport or national identity card. Okay. So then after valuing your phone, the phone you want to purchase, they value it, you balance with the value of your phone, then you top up the amount remaining. Wow, interesting. Yeah. So at your plus, do you have like people coming in? But you've mentioned it's only Apple products. Yeah. So it means if somebody wants to trade in with you, they must come up with, they must bring an Apple. Yeah. It's an Apple product for an Apple product. Okay. Yeah. All right. Interesting. Now, let's talk about the competition because I understand in Kenya right now, there's a lot of companies including Techno, some of the biggest brands. Samsung, there's even, is it called, a company is a product of Techno. There's one it is keeping me though. There's so many including Nokia, which was, it's still a Windows one before. How is the competition in the market, especially for, if you compare Apple, Samsung and the rest of the brands? Of course, this core, new phones are coming out every day. With new technologies, but for Apple, the products in terms of software and hardware, they are designed to meet the present needs. It's more of like we lead the full, I'm just saying. Then if you look at the watches, the earpods, then also the genuity of the products. Yeah. Interesting. And we can't do this without talking about COVID-19 because I remember when the pandemic struck, there are so many businesses that were rendered still, or they came to a standstill because of the pandemic. How did the pandemic affect your business? Just like most businesses, people could not operate normally because there was no physical interaction with clients at that time because of the lockdown. So we had to find a way of marketing better. So that's why we put much energy into social media. That's when we decided to make reels and sponsor them, which actually worked because we realized we have potential on social media and it's such a powerful platform for our products to sell, which we use up to date. Do you have maybe some of the biggest brands, not even brands, some of the influencers because you mentioned about sponsors. For somebody who is watching back at home and maybe they want to sponsor their ad online, how can they do it so that they attract maybe clients or customers like you mentioned for you? For example, on Instagram, there's usually an option. If you have a business account, there's an option of sponsoring your ad and there are different packages depending on the duration of time. You mentioned that your market has captured across East Africa. For example, somebody from Tanzania, they want to arrive at your shop. How do you do that? How do you deliver products to them? Because I believe this should be a very secure business that actually delivers products to people. If it's cutting across all the countries like you mentioned in South Africa, then I'm sure there's a way that you guys conduct your business. People outside of the country, we communicate with them, they send their details, they confirm the product first, then we package the product, then we send them through agencies like Wells Fargo, which they still warranty in case the product does not get to them. Or in case of damage, there's warranty on both sides from the transportation agency from our side. So that means you guys will be able to cater for that cost. And if the person who's doing the delivery gets an accident, how does that happen? Now that we have to come into terms with the transportation agency. Interesting. Now let's come back to Nairobi. I want to basically now come down to the customers who go for Apple products because you realize actually anybody who has an Apple product, they want to show it off.  I don't know, there's just something about people who have iPhones. They like holding it, they like showing it off. Why is that, by the way? Having an Apple product, the way Apple products are designed, first of all from the marketing, it's based on simplicity. Then there's the differentiation strategy. Apple products are so different from the other products and they distinguish themselves. Now talk about maybe a feature, because I understand you've got one for yourself. What makes it more compatible for you, maybe for somebody who's back at home and they're wondering if I want to have an Apple product, what is the advantage of having one? Apple products come with so many features. First of all, they seamlessly products work together from the earpods to the Apple watches to the phones to the MacBooks. The one you've got is an Apple watch? Yeah, it's an Apple watch. Okay, nice. So it's just a watch, because I've seen those that even have a feature where you're able to pick a phone call. Yeah, you can communicate with the watch. How does that happen? You just connect the watch to your phone and if you can't reach your phone, let's say you're in a meeting or you're driving, you just pick it from the watch and it will just communicate easily. In case you have earpods, you just click the earpods like this. I'll show you. Sure, sure, sure. You're wearing one in this case. So when it rings, it rings from the phone to the watch and it announces to the earpods. So you just tap, communicate, then you tap to... That's like a Bluetooth connection. Yeah, that's a Bluetooth connection. Okay, interesting. Some of the technological advancements from it may be because you guys, you deal with the product just to maybe tell our viewer about it. What are some of the advancements that have come with that? Especially because you said you deal with all the products. Which one mainly for you as a retailer? I'll say Macbooks and phones. Yeah. Macbooks and mobile phones. And iPads. Mostly. Yeah. What are some of the features that make it compatible for use? Do you have even friendly users for people who are disabled? Because I believe there's people who are blind and they want to have an Apple product. Are there user-friendly apps or software that make them available? For Apple, for example, for Apple, for the Apple music you have the Apple family. Okay. Where you can create a group of not only family but friends. You can create a group. And let's say I'm paying, I'll pay for the subscription, which you can be able to access all of you. So in terms of family, you can also share the family groups. Also photos, you can share the family. Yeah. Okay. Now, who are some of the buyers mostly for this product? Because, like I mentioned before, everybody who has such a brand, they always want to show it off. Is it a product cut for specific people with a specific amount of money or something? No, not really. We cut across all types of customers. Yeah. Okay. We don't have a target like, we don't say high end, middle, low. No. Yeah. We do business with all types of clients. Yeah, depending on their budgets, that is. Okay. Yeah. In trade-ins, for example, I want to trade a different product. Maybe product take a time, but I don't know if we still, if you guys still have an archive of maybe the first, first products that came on the market. If maybe somebody wants to trade in a very old product, because right now you mentioned there's even the latest, which is the latest development of the one you've got? The one you've got is on? The latest is on, this is an 11 Pro. Okay. The latest one is on 30 Pro Max. Yeah. And I heard the manufacturing of 14 on. Yeah, there's a release date on September, around 14th. Okay. Yeah. Then it will be available in the market from, say, September or October. Okay. All right. For any young person who is watching back at home and they want to venture into this business of selling your product or your kind of brand, is it cost-friendly for them? Yes. Uh-huh. It just starts from the way you market yourself, the way you market your products, how you want to come into the market, your target market, because you just can come like, you want to access all types of clients. Uh-huh. You have to start from somewhere. Okay. Yeah. Now, from your experience working with the, you mentioned its eyes too. Yeah. How is the experience working with people and how many people do you got at your organization? Uh-huh. We have a team of like 10. Okay. 10 people. Uh-huh. It's good because everyone has their strengths. Okay. Uh-huh. We do, we work as a team generally. Uh-huh. Is that maybe a selection or criterion that you guys used to have people come on board as professionals, even for companies? Because I understand even for, even for a media company before they hire you as a reporter, you must have some certain level of CV or a work experience that resonates with the company, like for your company and your representative. Like how did you even get to work with them, which is literally a high-end company, yeah? Yeah. I just saw them online. Okay. Then I personally approached them. Then I was told this is expected of you. If you're willing to start, give us the date you're willing to start then. Okay. Yeah. There's no really a certain criteria as long as you're willing to learn. Okay. And you've done your research as well and you know about funds, all Apple products, how they work, how they sell. Okay. Yeah. Now, for a person who is job-hunting, maybe they want to come to work with you. Do you guys accept CVs? Yeah, we do. Because there's also an HR. Okay. Yeah. If there's a vacancy, because people come, people leave, if there's a vacancy, we advertise on our social media pages. Okay. Yeah. There's usually a certain pressure for young people. I'd say young, let me not say a certain age, to have that brand of gadgets, including even the iMarx and the iPods that you mentioned, and even some of the latest brand. And before that, I'd like to maybe even know, how much, like, which is the most expensive and which is the most affordable one for, maybe a person who wants to maybe get one for themselves? It really depends on how much you're willing to spend. Wow. Okay. Yeah. So which is the most expensive from, you said you've got an 11 Pro. Yeah. How much does that cost? This costs around 75,000. Yeah. So it comes with the space, storage, user-friendly features. It's basically just camera and... Different processors, storage as well, different types of cameras, different features as well, for every series of phones, because there's a 12 series, 13 series, 11 series. Okay. Yeah. Interesting. Describe your relationship with your clients, because I believe in every market, every business, and US is a business as well, you have that rapport, that grounded rapport with your clients. There's those that come constantly, there's those that come and disappear, they come after years for you, in that market, in that business. How is your relationship with your client, if you can describe it? Our relationship is so nice, because first of all, we sell genuine products. Okay. And if there's any case of warranty from a phone, we always communicate, we always available, all of us, we always available, such as if a client wants to reach you, they'll always find you. And if they can't find you, there are numbers on our seats, which are always on. So, we build trust with them. But how do you build even trust with a client?  For your brand, are you the rudest guys? Not really. Because your brand is like, we are the biggest brand, so you can't tell us anything. No, for me to sell your new clients, for me to sell to you, I have to convince you. I have to take you through the features. I have to show you, you're getting something that is worth your money. And if you confirm the product is genuine, you'll give us referrals. That's how it works. We build trust, you get, you create a circle, circle, create a circle, they tell the other groups, other groups come, come, come, and all. I'm being totally wind up, we are outside. But what is the biggest lesson that you guys have learned from your business and managing such kind of a business? Because I believe this is one of the most rooted out for business even in their tech world. What is one of the biggest lessons that you guys have learned that you love to pass across to anybody who's watching back at home from a personal level as a brand and also from you as a company as well? I'll say consistency. Consistency really works. Wake up every day, keep pushing your products, keep pushing your products, you'll find clients. You won't struggle finding the clients. Okay. Yeah. Alrighty. If maybe somebody wants to access you and get to know your market, where you located at? We're located at B Hit Hours. Along May Avenue. Okay. First floor room 4, B Hit Hours, first floor room 4. Okay. Yeah. So if somebody wants to send a CV or maybe reach you or call, there's a number. Yeah, there's a number, there's a number on the bio. Do you mind saying it, Amma, you're good? 0704-839-296. There's also 0707-879-446. There's also 0706-60843. Okay. Yeah. So your final shots before we close it up. Any lesson you've learned personally? Yeah. No, I'll just quote Steve Jobs. The team.  Okay. Yeah. Who's the current founder of your company? Because I remember it started with Steve Jobs. Then Steve, is it Steve Jobs or Steve Jacobs? Steve Jobs. Steve Jobs. Steve Jobs. And then it went to Tim Cook, right? Yeah. Is he still the CEO or founder of Apple? Yeah, he still is. Okay. Yeah. So that's all. I've been told our time is out. And we have been speaking to Jeff Marier. He's a representative from iSTOO. And he's just trying to share insights on the influence of the business to them themselves as an individual and even as a brand. And you're back at home. If you want to venture into that business, I'm sure you've gotten a few hugs here and there on how to go about that. And right about now, we are heading on a short commercial break. Really commercial? Why am I just a break? When we come back, we'll be winding up. All right. I'm being told that we have finished the show. So we're not coming back with drum. So continue chatting with us on our social media platforms, Facebook, Instagram, and Twitter at Y254Channel. Remember, on Instagram, as I always say, there's an underscore there. My social media platforms is at BrianSaco101. Remember to keep the conversation going through the hashtag why in the morning today being entrepreneurship Tuesday. Thank you so much for watching. Like I always say, we love you for watching. And see you next time right here on Y254Channel.


Harry Potter has a code of invisibility, so can scientists really make such a clock? Now before I tell the story, I want to emphasise that invisibility is not the same as stealth. Stealth technology has been around for quite some time. A stealth object either absorbs or deflects radar away from the sensor so the sensor cannot detect the object by radar. So a stealth object is in fact a dark object, it is not invisible. By invisibility what we mean is perfect transparency as we have seen in many movies. Now I can assure you that perfect transparency is much more difficult to realise than stealth. Now in the last century, science writers have the vision that maybe one day a man can drink some chemicals and transform them to become invisible. Now such chemicals doesn't exist. Even if they do, I would suggest that you don't try that because modifying your body chemically is very dangerous and by drinking something you cannot make your clothes go away. Now in more modern concepts there is a lady, a superhero called the invisible woman. She is way smarter than the invisible man. She doesn't try to drink something. What she tries to do is to create a force field to guide light around herself so that you cannot see her. The question is, can scientists really make something that can make light go around the body so that you cannot see that body? So can light travel in a curve in the first place? Now the answer is yes it is. There is a natural phenomenon called mirage which indeed light do travel in a curve due to thermal gradient of air and that creates an optical illusion. In the year 2006 British scientists invented a new method called transformation optics. And with that method they designed invisibility clocks which really can guide light around an object so that you do not see that object. Now technically speaking you can say that that is in fact a mirage effect or you can say that they have achieved what the invisible woman wanted to do. Now this is regarded as a very major big through in science. However there is no free lunch. It turns out that nothing in nature can do that job. You have to make those material by yourself using really advanced technology. But if you can do that the same idea can actually allow us to design not only clocks for light but also clocks for other kinds of waves such as sound wave, heat wave and even seismic wave. Now these are some examples that made in Europe and United States that can cloak sound waves. But that is not the end of the story. Now it turns out that for the invisibility clock that was designed what it works is by excluding light from a region. So that the object or the body being clocked inside the clock is totally blinded by the clock. What it means is that if a reporter has this kind of clock then you cannot see him but he sees nothing. Now in order to get around that problem we in Hong Kong invented another way. What we do is that we design what we call an anti-object and we use that anti-object to cancel the light scattered by an object. It is like 1 plus minus 1 equal to 0 so that an outside viewer cannot see the object and you cannot see the anti-object. So the object becomes nothing. But you see that you have done that but invisibility is just one form of illusion. Can we go one step further? For example can I make an apple look like a banana? Well the answer is yes. Let me show you an example that we have done. So in the next slide you are going to see that on the left hand side I am going to show you the scattering pattern of a spoon. And on the right hand side you see the scattering pattern of a cup. Now in the middle I designed something which is a material where it put next to the spoon. Actually it makes the spoon look like a cup for someone far away so that you do not see that clock. You do not see the spoon but instead you see a cup instead. Now but up to now all the design we have we use for passive materials. If you allow me to use power and energy I can also design active sources which can emit waves. They can transform one object and make it look like another. Say for example I can make the example so I can make an apple look like a banana. So in some way I will say in the past it is chemists who made most of the materials. Right now physicists can also use computers to design new materials according to laws of nature. And some of these materials may help us to go one step closer to the technologies that you use to see in science fiction. Thank you.