Welcome to the Jenkins documentation office hours. Today is March 16th, and this is the EU, U.S. edition. Today we have myself and Mark Waite, and if anyone else joins, we'll be sure to add them and welcome them as we go. On the agenda today, we have some action items of blog posts that have been recently published. Just quick updates on the LTS release last week and our weekly releases. Documentation transition from Java 11 to Java 17. This is something we've been discussing for a few weeks and we'll revisit. Adding books to jengans.io, as this has become something that's popping up more and more now and we want to figure out alignment and standards for that. Improving end-of-life notifications and how we can better provide users with this information. How can we increase the visibility of this? How can we make sure that everything is seamless possible because it's hard sometimes and some things take people by surprise? There's the prep for CentOS 7 end-of-life which ties into the end-of-life notifications. This is something that we've been discussing for a bit now. And so potentially is gonna be removed from this sentence and we have moved Docs Office Hours an hour earlier in the day to account for the daily savings time in the US and more importantly, making sure that this is available and accessible for our compatriots and the new. This is, it's way later in the day than it needs to be and we want to keep that in mind to be respectful at our own time. Anything else on NeveMark or is there anything? No other topics. Those look like good topics. Okay, wonderful. And we have, okay. So we'll start with the action items. So two recent blog posts by Bruno Verrachtin both regarding the Minigen which is an awesome little controller machine. If you have not seen it yet or if you weren't at Fosom, definitely take the time to check these out. Bruno gives a background about what Minigen actually is and what it's capable of and has written another blog post about how it ties into Risk Me and other architecture and usage. He gives some background on how this came to be. It's just a really nice insight into what is possible with Jenkins and just the really out there uses that can come from it. And kudos to Bruno for putting all this together and making this. This is a lot of just kind of, I would say passion work at this point, but yeah, hi Bruno, welcome. Thank you. Thank you so much for the kind words. Of course. Sorry for being late. Ah, no worries at all, no. But yeah, these are really great blog posts and I think they do a really great job of just showcasing how unique and different each Jenkins usage can be and just the different ways that we can achieve the same goals. And thank you so much for the review and the suggestions and corrections on. That was much needed. Thank you. No worries at all. Always happy to help. The next post was just an update from Mark about our Alasian partnership and sponsorship. So we are gonna continue to have Jira available for Jenkins as Alasian is continuing to provide that for us. And so we will continue to have Jenkins Jira tracking. There was an update this past Saturday. That's now done of course. And everything should be working as it was before. Quick update. So the Jenkins awards voting period is officially open as of last Wednesday. The recent blog post here just shows what the awards, the Jenkins awards are and the nominees for them. You can use the Google form attached or LinkedIn the sheet here to vote. You do need a GitHub account but this is where you can go and make your voice heard. And I've also linked it here in the docs notes just so that if anyone is looking for it they can get to it very easily. And again, these voting will be open until March 28th and the winners will be announced at CDCon this year. So if you haven't yet, please by all means take a minute, check out the voting. And there are additional issues that show the awards and a little bit more information about them. So if you are curious, you can get some more info about that as well. And finally, we did publish our February newsletter last week. So this has now, this is now live on the blog highlighting a bunch of awesome things that we got to do in January, February, including FOSM 2023, which we have a separate blog post from our attendees and participants that show some insights and just some great reflection on FOSM, which was really exciting to be back in person this year. So thank you to everyone who attended and helped out with the Jenkins booth. Next up, just a quick update on the releases we've had recently. So last week we had LTS 2.38, 7.1 and we also had 2.375.4 and 2.394 coming up, 2.394 being the weekly, but the security team actually assisted in putting a lot of these together and putting them, putting together the change log and release notes for these releases. So a huge thanks to them for all their assistance with that. And yeah, this week we've also released 2.395 successfully and something that we noted, we have not seen any regressions for the new releases as of yet. So we're doing really well. This is great to see. Next up on the list here, we have the documentation transition from Java 11 to Java 17 at the point of Debian 12's release. Now I want to be clear, Debian 12's release is not going to include Java 11 but Java 17 instead. However, we're going to look to transition the documentation regardless of the Debian release to make sure that we're giving everyone the best baseline possible to work from providing the correct resources and instructions based on what seems to work best. Java 11 will still be supported. That's not getting dropped until 2024 at the very earliest. And the Java 17 is fully supported as of last August or September. I want to September. So there's no reason not to and it provides extra further functionalities and testing that just doesn't exist in Java 11. So it's a great time to make the change because of the Debian release, but the change is going to happen either way. It'll also be part of the Windows and Linux install docs because Java 17 will be the baseline we're using for that. I've emailed Tim Chacombe, so he's aware of the transition and on board with it. And one of the things that I just thought of was that we need to also make sure that any issues that are present in Java 17 are taken care of before we transition the docs so that if we do provide that instruction, no surprises. But this will happen sometime in April or May. So coming up soon, but we still have time to figure out what that's gonna look like. And again, adding books to jenkins.io is something that we've seen a lot more. So Kevin, actually before we go off that topic, there's a reminder. You said something there that was crucial and I learned of an issue yesterday that was specific to Java 17. And so you highlighted that we need to be sensitive to, we need to continue watching for Java 17 specific issues because they might persuade us to delay, upgrade of change of that documentation. Right now, I think the issue is resolved, but it was a particularly complicated issue that Basil Crow had to investigate that was specific to Java 17. He's investigated it. I don't know yet if the fix is released. But we've got to be continue monitoring bug reports to see if there are any that are specific to Java 17 that might cause us to say, oh, we need to delay this a little bit. Right now, I think we're on track April or May. We make that transition and announce it because people have still used Java 11. Right, of course. And Java 11, as we said, isn't gonna be dropped until 2024. So there's still lots of time left for Java 11 usage. Right. So Mark, was it in the Jenkins core? Of course it was. No, it was not. No, actually it was not. It was in the job DSL plugin. And so job DSL plugin, that's a rather obscure case, but it's got 40,000 plus installations. And so it's a very popular plugin. It's especially popular in very large installations, right? So there are companies that run huge Jenkins controllers and job DSL is very important for them. So they would have been blocked from going to Java 17 because this job DSL bug made it simply not work on Java 17. Maybe it's too early. Maybe it's not necessary, but do we have a Jira Epic or something in place somewhere in the documentation where we could list all the problems we have with JDK 17? We do. There's a, what Basel had created is a set of three phases for the Java 17 project. Just like he created a five phase, epics for each of those three phases. And yeah, there are epics for each of the five phases that we went through for Java 11. So we have an epic that we can attach there. It's a good suggestion. Let me put the... Yeah, thank you.  Of course, Basel does that kind of thing.  Your question was a very good question. It's just a delight to say, yes, that was considered and yes, we've got a way to do it. But I haven't checked to see if this job DSL bug is actually attached to the Java 17 Epic because that may have been missed. And if it was missed, it'd be a good thing to attach it. Thank you. Yeah, very good question. Thank you. Yeah, that's great to know. And thanks, Bruno, for bringing that up. I didn't think of it myself. So that's fantastic knowledge. Thank you, Vols. Okay.  I'll put the link to the bug report in. And let's see. So it was that a job DSL bug was specific to JDK 17 fixed by Basel in this thing. ER 1264, good. And so, and that's, there's an issue where was it that was, that has a Jenkins issue associated with it and we can link to that Jenkins issue and see if that Jenkins issue is correctly attached into the Epic. And it is, it's attached to Java 17 support, phase one support Java 17, good. But we should put that Epic because we don't know how bugs will arrive and not arrive. So when we announced this change through a blog post, for instance, we should probably put a link to that Epic so that people can see it saying, hey, look, we've switched. And here's the, here's the point of pride that shows you can go here if you think that there's something that doesn't support Java 17. And that's exactly like we had for Java 11 too, if anything. Right. Yeah. So whereas Java 11 had five phases, the Java 17 thing only has three phases. But yeah, it's the same exact deal. Fantastic. Thank you very much, Mark, for all the additional context. That helps a lot. And Bruno for a sparking conversation. Like I said, I had no idea if this was great. Anything else on that one, Mark? Bruno? That's for me. Thank you. That's it. Okay, thank you. So the next topic on the agenda is adding books to Jenkins.io. This is something that's been coming up a lot more recently. Folks are looking to contribute to Jenkins.io and adding a book is a pretty easy way to go about contributing. Right now, the books page looks like this. So we have just all the books laid out. The numbering in the list is gonna be removed pending my pull request. I've submitted a request to just get rid of the number order so that it's more general and not giving any sort of appearance of ranking. But what I've been seeing to is that with the books emissions, there's potential to update the layout or provide a little bit less or more succinct book summary here. Some of these can be a little bit longer, but the idea is that we wanna drive people to interact with that book itself, not give them the book on the page. So I was looking around and I did find this example. So OpenStack has a books resource page and it's really clean, really simple. It's just a cover image title, the author or publisher and then a really, really short summary of what the book's about. And then Link takes you either to the packed in these cases page for it or in other cases it can take you just right to the Amazon page. Let's see what that one's about doing in this one. Or this one takes you to the rescaler. So it's just a nice simple way to list the books and something like this I think would be a little more aligned with what we want for the books page. And then the how to add this, how to add a book entry is something that needs to be added to the Jenkins contributing guide. Once we've determined all of this and once we have decided that this is what we want it to look like, this is how long we should have the summaries be, et cetera. Once we have that we'll be putting that into the contributing guide so that it's available for everyone. Kevin, I finally had the time today to have a look at that. I was intrigued because I wanted to know more about what could I find in a book about Jenkins. So I think the first thing that's struck me that how am I supposed to go directly from the front page to this book page? I didn't find it. I had to use a search bar. It's a secret right now, Bruno. Oh, that's why. Okay, perfect, fine with me. Sorry, that's really embarrassing to say but it was an experiment launched to see how it would pan out and the experiment has turned into actually quite a useful thing, right? I think there's interest here but I don't think it's ready for prime time yet, particularly because of the numbers on the top left, the one, two, three, et cetera because they mistakenly lead people to think we've somehow ranked these or prioritized them. And I was almost ready to buy something but the thing is the latest one is not a subject I'd like to address as soon as possible. I would have been Jenkins to up and running if it wasn't from 2018, 2018, sorry because so much things have changed in Jenkins lately that I'd like something from last month. Well, and that hints that we may someday on this page want something that's sorts by published date or sorts by author or sorts by me, by publisher, right? Those are all valid questions and none of those things are represented on this page yet. Yes, but it's a project. Okay, I didn't get that earlier. Thank you. No worries. Thank you very much Bruno for bringing all that up. Yeah, and this was actually part of Hacktoberfest if I'm not mistaken initially and something that Chris Stern had originally pitched and followed through on. So, and this was around the time that Gavin Mogan was working on the web components for the header and footer. So I think, and this is just me speculating on what I can see from in front of me. I think it was just like a perfect timing of the book's page was created, but then the web, the components were created and they just didn't have each other. So we have the lovely components, the header and footer, but unfortunately it's not listed in the about link like it was intended to. So right now it's there, but you're right. It's not really readily available as a navigation point. So something we need to fix. But yeah, like Mark said, wanna make sure that everything is impartial and in line with how we wanna present the books. I wanna make sure that the formatting and text is okay. These number lists that are not formatted as a numbered list kind of take away from that. So there are a couple things that need to be adjusted, but yeah, this is something that I've noticed coming in a lot more of and with the amount of response people are providing for this, I think it would be very good to just have that kind of guidelines and the contributing guide too. And I think time or date of publication or the publication date and stuff like that is absolutely crucial when we're putting these books on the page because we do wanna make sure that it's relevant information up to date, nothing misinformed or perhaps referring to older versions that are simply not supported. So the books do need to be vetted a bit better. How we go about doing that is another question, but that's again something we can absolutely figure out as we go along. This isn't make or break to the Jenkins experience, thankfully, so we can take our time a little bit, but yeah, anything else on the books topic from Mark or Bruno? Nothing from me. Okay, thank you. Next up on the list is the improvement of end of life notifications and in tandem with that, the prep for the Cent07 end of life. So the way it's in right now is right now, when things reach end of life, we don't necessarily have a way to communicate that properly to our user base. We want to change that, of course, and make sure that users are notified if anything that they're using is gonna be hitting their end of life sooner than later. We have some examples right now. So for instance, Ubuntu 18.04 is one of those candidates. The blue ocean container page, the Cent07 container image, the Arch Linux agent container image. There are several things that we can consider coming up on end of life that these are things we want to notify people of and give them the best chance to take care of before it becomes end of life. We need to figure out how that limitation would be delivered and look and how it would give that information, but we do have this really nice site, endoflife.date that is, it's got an API guide, so it's open and interactable. So that could be something that is potentially automated down the line. It also has end of life for most things that you run into contact with while using Jenkins, including Jenkins itself. So lots of great info there, but utilizing that, leveraging that would be a good way to get this information out. We also are considering blog posts, other announcements within meetings, just multiple avenues to get this information out and deliver it to the user base. So, Kevin, you just reminded me there's something that needs to be fixed on the end of life.date page and I'm not sure how we explore that with them. You're right there, that's the perfect page. Notice that it shows 2.375, that's correct. And it is in fact a valid version, but 2.387 is not listed here. So we're missing the current LTS. Right. And I'm not even sure how to present it here because 375.4 is the current LTS, but we expect no further security fixes in it. So it's actually therefore, for their definition, end of life, the next one should be inserted. I'll insert it and see if we can get this right, but then I've got to have a conversation with the maintainers of this site to automate that release detection logic. I don't know how they do it. Okay, well, that's good. We caught that then. Make sure this is all up to date, because yeah, cause 2.387.1 also came out on the same day, but maybe the release order was just slightly different. Who knows? You know, I suspect that it's that their page is not correctly configured to automatically generate the next LTS version, and it may have to be a manual process. So this looks an awful lot like an ASCII.Doc page. I think it actually is ASCII.Doc, but it's a very simple to edit format, but the editing was required actual editing, not just a machine that ran a program. Got it. Okay, that makes way more sense. Good to know. Okay, great. Thank you very much, Mark. So, and in tandem with the notifications, CentOS 7 is nearing end of life next June. It's been a maintenance mode since 2020, however, and it is looking at older versions of Git and SSH. So two things that are crucial to plugins working and operating properly, that it just does not necessarily have. It's not supported by the Jenkins RPM installer and not maintained in its own container image at this point. The end of life is June, 2024. So again, like there's several reasons why pushing this to end of life now or sooner than later is the right call. This is something that Mark's been proposing and will be, if not, he's already created a JEP4. And so we're gonna capture the end of life in CentOS 7 and the Epic, announce the deprecation before it happens, making sure to stop running automated tests in CentOS 7 and just in general, making sure that we can remove all the CentOS 7 information from the documentation. This has started. There have been a few places where Rocky Alma 8.9 are suggested as opposed to CentOS 7 and there are a couple other options here. I forget what they are off the top of my head right now, but there are multiple alternatives that we can recommend instead of CentOS 7 or utilize instead of CentOS 7. And so making sure that the correct versions and up-to-date options are present in the documentations is one of the next big tasks as well. Mark, was there anything else on the CentOS 7 end of life information? No, I haven't submitted the Jenkins Enhance for proposal yet, but I will. It's coming. Okay, no worries, no issue there. Yeah, and the last thing on the agenda, again, we're just gonna move the docs off this hour as to an earlier time, one hour earlier, making sure that this is readily available for everyone in the EU, US. If we stick to crazy daylight savings time here on my East Coast, it's gonna be way too late and I don't feel comfortable with that for the whole rest of the world. So we're gonna adjust that and today's the first day, so yeah. So now it's gonna be at, it's gonna remain at one o'clock my time, EST, that's not UTC, yeah, Bruno, maybe you can help me one time. No, I don't think so, I'm as lost as you are. And yesterday I discovered that in Phoenix, Arizona, they don't change with summertime and wintertime. I didn't know that, which is kind of funny to me, and 10 days from now, it will be one hour later in France, so it will be about 7.30 p.m. So the time is fine with me for this week and even in two weeks from now, that's perfectly perfect. Thanks a lot for thinking of us European folks. Yeah, of course. Bruno, would it be better for you if we moved it even one hour earlier so it was right after Google Summer of Code office hours? Yes, but you need some time to eat. If it's 1 p.m., Kevin, I don't know. My schedule is so weird, don't go off my eating habits. My eating habits are not normal. Please don't worry about me. Oh, really? And back-to-back meetings is actually healthier for me than having an hour gap between them. Let me look for just a minute to see if it would work for me in the average case. So office hours for, okay, today, GSOC office hours. Okay, it would collide with advocacy and outreach if we moved it the hour earlier. Advocacy and outreach happens twice a week on Thursday, so I think it's probably better to just leave it at the current time. Okay, fine with me. As long as that's okay with you, Bruno. It is.  Oh, yeah, I'm the one suggesting it. I definitely am okay with it. And I already made sure it doesn't conflict with anything from me as well, so yeah, we can just keep it at that time and just make life easier for everyone involved. Now that you have more of the time, I have to come every week. Tom! I mean, you don't have to, you're more than welcome to do what you need to do. Of course, no, no, it's a pleasure. It's a pleasure to be here and spend some quality time with you. That's cool. Oh, thank you, Bruno. I appreciate it and I'm glad you have fun here. That's nice. Thank you. Okay, done. Awesome. That takes care of everything on the agenda. I just realized we're over time by a couple of minutes, so we'll call things here. Again, thanks for joining. I appreciate it. The video will be available in 24 to 48 hours and take care and until next week. Thanks a lot. Bye-bye. Bye.


All right, let's get going Thanks everyone for coming To my presentation on real-time tuning I'm Grazian Krishan And to give you a bit of context about me. I work for an eye here in Austin and I Makes hardware and software for test and measurement systems I'm part of the real-time OS group Where we use preempt RT based kernels and we build our distro based on open embedded in Yachto We mainly work with 32-bit arm systems and Intel systems Mostly item class some zeons up to 20 cores max. So pretty small systems I also happen to be the maintainer for the Linux kernel shipping on RT hardware at an eye So in this presentation, I'm gonna try to cover really quickly just the slider to on what real-time is Mentioned the tools you can use to work with real-time systems and hopefully spend most of my time Talking about some of the tuning knobs you can use to get better RT performance out of your system some of the safety nets you might need to remove for the same reasons and some of the gachas we encountered around the way and things to avoid I'm not gonna have time to go into any implementation details about preempt RT or anything like that or do in-depth review of tools There's a lot of good presentations out there on the tools to work with our team So at a very high level real-time is really all of all it is about is having a deterministic response to a stimulus if an event happens we want the system as a whole to act in a Bounded and predictable amount of time. We call that latency Events can be asynchronous a sensor exceeding a threshold Something happening in the real world or it can be synchronous clock driven events periodic tasks that happen, but the the main thing about it is this concept of latency and responding in a bounded and predictable amount of time having max latency Real-time since those systems today can get really complex That's actually one of the reasons preempt RT is good at it because you can use the full inner sector system to do Really complex sensors like cameras and ML in your control loop and stuff like that but the the same idea of having a bounded latency and Response to something happening holds It it can be catastrophic in some cases to miss your latency that line might run over somebody So if it's all about measuring latency what kind of tools do you use for that like as mentioned before you can This systems can get really complex.  So my my recommendation is to actually take a layer approach and the first thing we do and we do this on every kernel upgrade We are on cyclic test at its core. It's It's a pretty simple test. It basically Sponsor bus bunch of threads and then you do this measurement loop where you take a time time stamp You sleep for a period of time take another time stamp and from that you can compute the latency the wake-up latency of that thread and And you can do more interesting statistic like histograms and stuff like that It's also important to simulate the load you expect or actually Maybe even stress it beyond the load you expect on the system And you can use tools like IPerf to saturate your network link or like FIO to do this scale Hackbench is a good tool to just stress the scheduler with background tests There's a lot more Things like this I Also like to plot the histogram that cyclic test produces a text histogram. I like to plot it After a while you start to see patterns if you plot the histogram you get from cyclic test It's a useful thing to plot. It's just a simple Python script One thing that cyclic test doesn't measure a cyclic test measures a lot of things. It can measure Preemption being disabled interest being disabled and so on but it's usually driven by a local CPU timer So it's not going to include your IO latency. So it's a good idea to write a set of tests that Include your IO latency in this measurement loop and and you can loop back your IO your analog out your analog in and And do kind of the same measurement loop where you read the inputs do some processing Update the output and see how long that takes or measure how many of these loops you can do per second We have a set of tests like these I'll come back to this graph they're really useful to find regressions when we do kernel upgrades and and Check our drivers that they still work and have good RT performance. I'll come back to this graph in the gacha section But ultimately you have to test the whole system latency And you have to do it for multiple days weeks months and and stress the system beyond It's expected operating point to be to have any kind of certainty that in a complex system like this It's going to behave correctly in all situations There's a bunch of other tools Cyclic test and hack bench or part of our detest is a lot more in there RTVL is another combination of cyclic tests in a load and some scripting The Linux test project has a section for real-time RTLA I'm gonna plug Daniels talk on Thursday it's taking a white box approach to Detecting where your latencies are coming from using tracing for that. It's really interesting. I encourage you to see that So then once you know you have some latencies in your system, how do you go about debugging it? I'm just gonna mention the tools here. The top three are my favorite tools use that trace trace command in kernel Chalk all the time It's it's really convenient To interact with the trace file system directly for example for you can just start the trace from your application It's you by writing to files. I use trace command to extract the trace and then the post filtering And Kurno shark just to see what's going on. It's it's my bread and butter tool There's other tools that are useful. I mentioned LTD and G there There's a bunch of other tracing frameworks that I have no personal experience with but you can find talks on them Perf is really useful for Debugging the hot pads in your application and kind of figure out where where the CPU is spending the time It also has a really useful mode where you can def Results from previous versions so you can figure out where your regressions are coming from There's new tools now based on BPF like BPF trace and the BPF compiler collection There's they're really useful if you want to do custom instrumentation custom tracing And don't discard if you are close to your hardware team and have access to a scope and like some digital lines Sometimes it can be very useful to have an external reference. You can check Against and and figure out how long something took Like I said Steve and among other people did a bunch of presentation on these tools you can find the recordings from previous DLCs So now how do you go about tuning with a system to get the best performance out of it? The first step is of course getting since I'm talking about preempt RT is getting a preempt RT kernel Install most of the distros these days provide one if you want to build one from source You can use the stable RT or the develop RT branches For now I put it for now in brackets there because the preempt RT patches It's on its way upstream and maybe by the end of the year you won't need to use a separate branch There's also birds of a feather I want to plug the Stephen is going to do on Friday And I think there's going to be a lot more details on on the status of the preempt RT patch and where things stand as of now One thing I wanted to mention on this slide is if this is all you do if you just install a Preempt RT kernel and expect better performance. You're going to be really disappointed If you don't do any of the tuning it's actually probably going to behave worse You'd real-time systems have to be designed from the ground up to have real-time performance So basically what you need to do is identify your RT workloads in in your application in your system and then Decide on a scheduling policy and priorities or if you're using schedule on things like runtime deadline and period and on the right-hand side of that slide I kind of Listed the various scheduling policies that are available and kind of what preempts what let's get that line will preempt schedule and so on So my recommendation is to use scat 5 or scat deadline if you have for your critical loads And for scat 5 fall you need to pick a priority in the one to well It goes from 1 to 99. Don't use 99 It's used by the kernel migration threads and can get you in trouble if you use it I'll pick something from 1 to 98 or if you're using a scat deadline. Yeah, you need to provide those parameters for each of your threads Don't forget to adjust the priorities of your interrupts The ones that are part of your RT loop, you're probably gonna want them higher priority and the rest You're gonna want to have lower priority for them as same goes for kernel threads and so on And use everything that runs for everything that runs in the background use scat other Or lower priority So it won't interfere with your RT loads The other way you can isolate your RT workloads from Everything else that's happening on the system is by partitioning your CPUs It's it is a good idea to designate some CPUs as housekeeping CPUs and and that's where you are going to want to move all your background like workloads or like things like the kernel word queue threads and Same goes for interrupts most of them you can Affinitize to a certain core and it's a good idea to do that depending on if they're part of your real-time loop or not And for for real really sensitive workloads, you might want to actually isolate the CPU even from the scheduler take and For that you can enable full dynamic takes if you if you set the config no hurts full option in your kernel and then by setting The I sold CPU and no hurts full kernel Parameters at booth you can isolate that CPU from from the kernel scheduler and then you can explicitly assign a Tread to it and that allows you to run a real-time tread Basically at 100% of CPU utilization without interference from anything else and you can get some really good latency numbers by doing that I forgot to mention on a previous slide, but you can set the scheduling priority and Scheduling policy using tools like chrt or the sketch these calls for partitioning CPUs CPU sets is a it's a good tool The secret version one actually works better for RT because you have thread level granularity Compared to secret version two Yes, Stephen Yes, that's the next slide. Thank you Perfect segue Yes, so the other thing you need to do is Delegate your RCU callbacks to threads then then you can move to your housekeeping CPUs by Specifying that RCU node to be CPU list at boot And that's how you do it Other things you need to worry about don't do memory allocations from your real-time context Do they all your allocations up front if possible? Allocations can cause page faults and obviously that's a bad thing for your for latency Also consider resolving Symbols are startup like if you have shared libraries you want to resolve all the symbols of startups So you don't get dynamic like behavior later that that it gets your latency and After you do all that it's a good idea to lock all your pages for your real-time processing memory Using unlock all you can specify current and future flags Those that will basically keep your code and data pages from being Paged out and and introduce big latencies The delay VM instead timer It's a good idea if you're using the CPU isolation feature and you want to eliminate all latency sources The other thing you need to check is your clock sources, and you can check that under that is fast path For Intel base CPUs you really want to use TSE all the other timing sources have big latencies by comparison And it's also a good idea if you're doing tracing to check what trace clock is used just to make sure your data makes sense The other big knob that it's a good idea to turn off to improve your your RT performance is power management Going into sleep states or power states can can introduce large latencies and So it's a good idea to disable CPU frequency scaling and the CPU Freak driver Disable power management. You can do that boot for P states and C states. You can do things at runtime for C states By the way, all the slides are already uploaded. So you can you can read this a leisure later And Where it comes to firmware bias on Intel Same idea disable P states C states high-portreading up until relatively recently It was the recommendation was to disable it because you can get interference from the sibling core With core scheduling, I believe it's possible now to leave it Enable and and with careful design you can you can make use of SMT and not Pay the penalty Turbo boost can introduce variations in latency. It's debatable if you can leave it on or not Memory corrections this there's some like deep level some memory correction that can add a bunch of Of latency It's a good idea to set it at the lowest functional level possible It's also a good idea to disable peripherals that they are not using because they're just going to generate interrupts and stuff And increase your latency Legacy hardware sometimes can be actually implemented in in bias via like SMI interrupts And I'll get back to those system management interrupts I'll get back to that in the catcher section and there might be other options in your bias that you need to tweak and test And see if they what kind of impact they have depending on who your bias bias vendor is So let's talk a bit about safety nets probably the biggest one is RT throttling That is the feature that reserves a certain percent configurable percentage of CPU for non real-time tests So if you're if your real-time thread goes above a certain percentage, you will get throttled Obviously this has a huge impact on latency We actually disabled that on our systems by default and we're telling people to rely on just a Hardware watchdog to catch runaway RT tests that that might consume all your CPUs or Or yeah, and and educate we try to educate our users from the get go to design their applications where they allow a Headroom for other background tests to run on a CPU If you're on Intel and you've proven that your TSC clock is stable and you can use it It's a good idea to disable the watchdog Because again, it can interfere with things and thing goes for like soft and hard lockups after you validate your system It's stable and runs good. You can disable those and again for for machine machine check errors you Basically, you can ignore the corrected errors because there's another kernel thread that runs in the background and adds noise And especially if you're trying to do full dynamic takes and CPU isolation It's gonna prevent a system from from being able to disable the scheduler tick Some safety nest for memory you might consider disabling memory over commit that That that will make it where like allocations will basically fail with Like malloc will fail of returning null and if your application is written correctly Then it can handle that as opposed to allocating as much memory as you want and then finding out later that you exceeded a physical Space available and womb killer getting involved and speaking of the womb killer It's also a good idea to prioritize Processes you want to kill and processes you don't want killed like for example, if you echo minus 17 for whatever is your RT process that they'll make it unkillable and It's a good idea, right? You also should be deciding what to do on Out of memory situation and their recommendation is to actually make the system reboot rather than Rely on the home killer to kill a random process and then your your the system will continue running But you don't know in what degraded state it is So that can create problems. It's it's often better to reboot and get back to a good state These I hesitate to put up But if your system is completely disconnected from network and you put the epoxy in your USB slots You can consider disabling the mitigations They do have a pretty big performance impact in certain on certain workloads and there's a handy kernel parameter They can just set mitigations off and getting back that performance. But like I said, be really careful with this So now for some gotchas things to avoid I already mentioned system management interrupts these are basically hardware Former hardware interrupts that are handling former in bios or ufi on Intel and their highly priority. They're unmaskable They're used for things like temperature management legacy hardware emulation patching hardware box in some cases The trouble with them is the transitions to the system management mode via this system management interrupts is completely Like Linus doesn't even know about it.  It's gonna see large latency SMIs and SMM are excited to see specific, but there are a similar Cons similar privileged modes on on other architectures like the secure monitor mode on arm there's a good wiki page by the Real-time collaborative project that talks about this and and shows you what tools you should be using to try to detect This SMIs happening and there's not much you can do. I mean you can if you can figure out what the Situation was that triggered the SMI you might be able to avoid it Worst case scenario you might need to select a different piece of hardware that doesn't suffer from from a lot of these Speaking of interrupts this are this is on the kernel side and this is the graph I showed at the beginning that was showing the degradation when I was doing that a analog in test analog out loop test So when we upgraded to from 414 RT to 510 RT, we discovered this drop in performance. You see there And we couldn't figure out of it first while what happened But basically when the preempt RT patch was rewritten for upstream. I believe if I'm Explained this correctly you can only have now a single software a cube per core and There is the bottom Half lock it's getting basically Both software accused and and force-threaded interrupts use this bottom half lock And this will cause basically latencies in your force-threaded interrupts If you don't request interrupts specifically with request threaded interrupt Which doesn't take that bottom half lock lock. So after we switch to our driver to Request threaded RQ directly as opposed to just relying on preempt RT force-threading your interrupt handler We gain back the performance. In fact 510 is much better than 414 if you look at that graph It's the latency spread is the jitter in the latency is much smaller and performance It's basically the max performance is about the same Another interesting one is MMIO CPU stalls. So this is a great cyclic test histogram that I took and a couple years it's a couple years old now And in the background, I was just accessing a TPM chip And I was seeing this huge 400 microsecond added latency tail on the histogram and The reason for that is if you look at this happened to be an Apollo Lake system If you look at this CPU diagram or the system diagram for that You can see the core where your code is executing up there on the left and the TPM chip is all the way on the right And there's this long path that goes through all the IO this IO fabric blocks. They are all Different bass widths and different frequencies Which means there's a bunch of buffering happening in the way to transition from all these IO fabrics and at the end you go through a fast SPI a boss To access that TPM chip and what happens is there's this common pattern where There's a bunch of register writes happening in a row through to this TPM chip to configure it And then there's a status register read and if the read is right after all those rights that happen in a row What happened is the rights will get buffered along the way in the fabric But the read because of the architecture ordering guarantees has to wait for all those rights to actually Propagate all the way to the hardware the rights to take effect and then you get your status register back But then the net result is your CPU will stall waiting for all those rights to go through all those bosses and To all the way to your TPM chip And this can happen with a lot of peripherals with we've also seen it with an internet Phi And we discovered that by accident by bumping an internet cable while we were running Cyclic test and we saw the latency spike and it's kind of the same pattern Basically the fight cheap is trying to do the link-negotiations for internet so there's a bunch of rights to that cheap and then a read that will stall everything and Obviously, this is bad because you don't want your RT application to fail just because you bumped into an internet cable that Not even part of your RT like you might not even use network communication here at the application But it's just background load I don't have a good solution for this other than test for it and try to avoid it in We have a couple ugly patches that are really not upstream of all where In a couple drivers where we saw a bunch of these rights and a read happening We basically added a delay to let the to give time to for those rights to propagate to the device and now stall the CPU Another big one after you solve all the tuning knobs and all that Probably the most common one I see when I debug RT applications priority inversions, so just to explain this If you have three threads there like a high priority thread age medium priority thread low priority thread and let's say the low priority thread starts running first and it acquires a lock or some other mutual exclusive racers and Doing its thing and then the high priority thread comes along and preempts the low priority thread And that's all fine. That's what we expect If you're using the FIFO scheduler But at some point age once the same mutual exclusive resource or lock so it blocks because Alice holding that lock The problem is that in the meantime you can have this medium priority thread That's completely unrelated to anything that that becomes runnable and because It is the highest priority after the our age thread blocks It is the highest priority thread on the system So it will get scheduled in and it can run as long as he wants and it creates this unbounded latency in your high priority thread Who cannot grab the lock because Alice holding it by L cannot run because the m thread is preempting it So this is really bad and it can lead to high latencies. This is kind of how it looks in So in Colonel shark, especially if you write the test to the specific area produce this But you can see the low The order is inverted by the low priority thread on top there medium high And you can see how the high thread gets blocked and then your medium thread can run forever And only at the end of that the lock the low priority track and get scheduled in and release the lock and The high priority thread gets to run the the solution implementing preempt RT for this is called priority inheritance and The gist of it is basically at the point where age blocks on the lock in order to prevent this priority inversion the priority of the L thread the low priority thread gets boosted to the priority of the high thread and That makes it where then this Interfering medium priority threat cannot come in and and scrub your day So low priority thread gets to run a high priority for just the right amount of time to release that lock And then the high priority track can continue and this is how it looks in colonel shark You see there like the high priority thread running blocks on the lock the little priority track gets boosted runs for a bit releases the log and And so on and you can see it in the trace events too if you can't read it So it is a common problem and it's the it's Unfortunately, it's common because there's there's a lack of priority inheritance support for locking primitives with the leap it read Glee C library the only Primitive that that has priority inheritance is the p-tread mutex and you have to use attributes to Enable it all the other priority all the other primitives in leap p-tread don't have priority inheritance And even worse we discovered a couple weeks ago, and we're trying to do a quarterly release the p-tread reader writer lock will actually live lock in user space remember we disable RT throttling so The p-tread reader writer lock got rewritten to be more performant So it does a lot more in user space trying to acquire the lock But that what it ends up happening you end up with this like high priority RT Threads spinning in user space trying to compete for that lock and it lives live locks your system So that's really bad. We don't have a solution for it I think we have a reproducing case now that we can send upstream and we're we're gonna work on See if we can do anything about it, and there's also no way to set priority inheritance on the standard mutex There's no way to set the attribute on it. So I Talked about a partial solution a couple years ago at ELC we Darren Hart and me and the RT folks worked on Implementing a conditional variable that has priority inheritance support and there's also PI mutex I'm taking suggestions if you know about our libraries that work well with RT that implement POSIX locks Yeah, it's not great And to make matters worse you can actually have Priority inversions with interrupts So this is a real case we encountered we had the watchdog functionality that was implemented in a CPLD That part makes sense. It's the CPLD is doing power sequencing on that board anyway So why not do a watchdog in it? Problem is it's on an I2C bus Which is fine for configuring and if you just want the watchdog to reset your system that all works great But there's there's a use case where we wanted to fire an interrupt and the reason you might want to do that on a real-time system Is because you want to put the IO in a safe state you might have like huge machinery attached to your controller and You don't want that thing to do crazy things while your controller is rebooting So you fire interrupt and there's a the sequence that puts IO on a safe state The problem is Even though the watchdog interrupt was configured to have high priority It requires an I2C transfer to acknowledge the interrupt and the I2C interrupt at that time was low priority in our system So and some unrelated me priority interrupt can basically ruin your day so you need to watch out for this and And kind of audit your peripherals and how they're connected what buses they use and what interrupts priorities you said And since I promise tricks in the presentation title This is a trick that I use a lot I Talked about priority inversions how do you go about finding locks in your application that don't have priority inversion and a really convenient way to do this is To basically patch the few tech sees call There's there's a piece of code at the beginning that checks for all the priority inheritance Few tech operations and you can add the default case in there That just sends a segmentation fault to your process if if it's an RT thread But it didn't call into the few tech sees call with the PI few text operation and This works a trade it like if you run your application gdb You can see the exact spot where where that call was made from get a nice Stack trace and you can see the p-trap Primitive that caused it in this case. It was a barrier They use few takes weight and and it was from an RT thread. So it's gonna suffer for priority inversions So in summary, I Hopefully gave you at least a starting point to what real-time tools to use and and What are useful and Some of the tuning knobs and and some of the safety nets you can remove And hopefully some of the things to avoid so you won't get trapped by them So that's all the content I have Are there any questions I can help answer there's two from the virtual session all right The first one is Bear with me. I am not in this field. I am interested in knowing if there's any Sorry, I'm interested to know if there's any built system built route Yachto, etc That integrates or makes it easy to apply most of these settings for you as an a template Instead of having to go through the checklist for any new system I Am not sure I know Red Hat has real-time offerings and and the out of disturb vendors. I Don't remember seeing anything yachto related to this I mean, it's a good idea in general But yeah, I'm not aware of something that exists out of out of the box unless like I said you use You contract with redhead or use one of their That's true Yeah Sorry, I don't know your name. You were saying that they can check the RPM spec package, right? That's using redhead and and that can be a good starting point for you And I should mention all our stuff is on GitHub. I mean if you if you want to look at our stuff You can take a look there. It's get up slash and I There's one more virtual question Does Muzi have complete support for PI Convars and mutex's. Yeah, I think it's muscle I Remember those interest from the muscle man there. We don't use muscles. So I'm not sure I can answer the question but I know they were cognizant of RT applications and and they were trying to Make it work for RT But I have not used this. I'm not sure what the status is And we have five minutes if anyone from here has a question. I'm happy to pass on the mic Yeah, firstly, great presentation. I had a question. Are there any multimedia pipelines that have been tried with? real-time Threads and you know, I was drivers and you know, basically just from here say unlike the RT users mailing list and and IRC channel There's definitely people using it for audio applications and and big like stage kind of sound setups So it's at least for sound. I believe that's true. I believe that's true for video too because There's it's used on cameras and other things Like I said, we're more in the test and measurement space. So I am not Sure how common that is in consumer products, but I believe the answer is yes Probably the later the question is what application do you normally the product? It's really depending on which product. It's a lot of trade-off. Yeah, probably why the reason Yakato don't have this is you once you enable RT you suffer from like a memory Overcommit memory normally So what application do you normally use? Yeah, so if I understood the question correctly is yeah, and you're correct The reason you cannot have like just a standard set of tunings you it's because it's specific to your application in it for us a lot of our applications are Things like basically hardware in the loop test for example, if you have Like an electric vehicle inverter and you're you're trying to simulate the car around it because you don't want to test inside the car Right, you're just testing out a production line So you're simulating everything else the car is doing and and only and and you're hooking up this inverter And the reason real time is important there because you have to feed the information in real time I like as otherwise it can blow up in the most extreme cases If you if you feed it the wrong control signals So so having a real-time response to that is Important another application. I know from from mice companies If you're simulating for example Or emulating a 5g base station for example They 5g has a pretty tight time slots. I believe they're like 250 microseconds or something So if you're trying to emulate the protocol and send all the data you have to do it on those time intervals So it's important to have that real-time behavior There's a lot more like that Production test and stuff like that All right, I think we're out of time out of time and this is the last session of the day And I think the next one and probably a little bit of time is in like one minute If you go back to the showcase where all the sponsors are there is a booth crawl If anyone wants to participate Thank you so much


So, we had calculated in the previous part the denominator of this particular of this expression here. We had this expression which was the probability distribution of the state given the information and we were calculating that probability distribution. We calculated only the denominator of that I will just show you what the actual expression then turns out to be. So, the numerator the numerator here is this given given s this numerator turns out to be in fact 1 3rd times 1 4th times 2 3rd times 3 4th plus 1 3rd times 1 4th. So, if I put these together the probability we get the probability of x 1 equal to p bar given g g s given this information turns out to be just 1 by 1 by 7. So, as a result of this we can now go back and ask what is j 1 of g g s here j 1 of g g s remember was the minimum of these two terms which is the cost of cost of taking action cost from c and the cost from s cost from s was all was one and cost of cost from c is 2 times this probability the probability of x 1 equal to p bar given i 1 cost of s is 1. So, we have computed this probability to be 2 by 7 sorry to be 1 by 7 as a result of that the this here is the minimum of now 2 by 7 and 1 and 2 by 7 is less than 1. So, therefore, this becomes 2 by 7 and the optimal action to take at time 1 when you have information g g s the optimal action is to is to simply continue that means keep running the machine. Now, we can do this again for the information when the information is b g s. So, let me write this as case 2 when i 1 is equal to b g s when i 1 is equal to b g s we can again compute these probabilities it turns out that the probability that the machine is in in state p bar given b g s remember b here is just z 0 g is z 1 s is u 0. So, this probability it turns out this probability is actually equal to the same as before it is equal to all again 1 by 7. So, in this case then we can write j 1 of b g s is also equal to 2 by 7 because a similar expression will will be written out there as well. So, you will again have so so in place of in place of this probability now we have this probability but the numerical value is the same they are both 1 by 7 and so as a result of that j 1 of b g s is turns out to also be equal to 2 by 7 and mu 1 star of b g s is is also c. So, this is when the information is b g and s. Now, let us go to the next next case what are the other cases possible remember that we have remember here that z 0 and z 1 can both be either b and g u 0 can be either c or s. So, in other words i 1 can take 8 possible values. So, we need to really do this for each of the 8 possible values of i 1. So, far we have completed 2. So, now let us go to case 3. So, i 1 we can here i 1 is now g b s. So, in this case we can compute that the probability that this that the machine is is broken in this this can be computed as to be 3 by 5 and then we get from here that j 1 of g b s is unity mu 1 star of g b s is equal to s case 4. So, for case 1 where i 1 is b b s we again get this probability to be the same as the one above we get it to be i 1 b b s this probability also turns out to be 3 by 5 and as a result of that j 1 of b b s is the optimal action in this case is to is to stop. So, in case 3 and case 4 the option optimal action is to stop and inspect and the reason it turns out to be we need to stop and inspect is because because remember this probability is now 3 by 5 this probability is 3 by 5 this times are cost cost is 2. So, the 2 times 3 by 5 will give us 6 by 5 and that is that that is is is greater than that is greater than the cost from stopping. So, we are we are looking at the minimum of these 2. So, that turns out to be turns out to be the minimizing action is to stop and that turns and that gives you a cost of cost of unity. So, now let us go to case 5 and case 6 case 5 would be when the information is g g and c. So, here this turns out to be 1 by 5 then since this is 1 by 5 the cost form continuing is going to be 2 by 5 and that will be less than the cost from stopping. So, as a result in this case the optimal action is to actually continue. So, j 1 of c c g is 2 by 5 and mu 1 star of c g g sorry this is my mistake here this is c g g c is to continue. Then case 6 when the information is b g c when the information is b g c probability of x 1 equal to p bar b g c this is equal to 11 by it turns out to be equal to 11 by 23. And since this is 11 by 23 again the cost form continuing is going to be 22 by 23 which is less than 1 which is the cost from stopping. So, again the optimal action here from b g c is again to continue and this is the optimal cost. Second last case is the case when i 1 is g b c here then the probability of the machine being broken turns out to be 9 by 13 and therefore the cost form continuing is now larger than the cost from stopping and so the optimal cost the cost to go from that in state g b c is unity and the and the optimal action is to stop. In the final case when the information is b b c this probability of x 1 equal to p bar is 33 by 37 and the optimal action in the state b b c is to stop and the cost to go turns out to be turns out to be unity. So, here I have written out all the all these remaining cases case 3 to case 4. Now what you will observe in case 3 and case 4 is that the probability that the machine is broken in either of these 2 cases that means when i 1 is g b s or b b s the probability that the machine is broken at time 1 is 3 by 5. Now 3 this is to be multiplied by the cost remember the cost the cost was 2. So, the cost from continuing is actually for us is 2 times the probability under consideration. So, that is 2 times 3 by 5 and that gives us the cost from continuing as 6 by 5 and that is greater than the cost from stopping which is the cost of stopping is 1. So, we get we are comparing 6 by 5 with 1 and so therefore the optimal action is to actually stop. So, that applies in this in both of these cases in case 5 and case 6 we in case 5 we have the probability of the machine being broken as 1 by 5. So, the cost is 2 by 5 cost of continuing is 2 by 5 and that is less than the cost of stopping. So, the optimal action is to continue the in case 6 the cost of the probability of the machine is broken is 11 by 23 the cost of continuing turns out to be 22 by 23 and the and that the optimal action then is to continue. In case 7 the probability that the machine is broken is 9 by 13 of the cost of continuing then will become twice this which is 18 by 13 and 18 by 13 is greater than 1. So, therefore the optimal action is to stop and the cost of the cost to go is 1 the same holds in case 8 the cost the probability of the machine being broken is 33 by 37 and therefore the optimal action is to stop here the optimal action is to stop. All right. So, in so you can notice here that we there are some very intuitive observations that we can make for example in the last case here if the machine has if your inspection has led to readings has led to readings bad and bad for the last two occasions the probability that the machine is broken is rather high it is 33 by 37 and so therefore the optimal action for you to take at that time is the optimal action that you need to take at that time is to stop and inspect the machine. So, that is that is what that is what we find here. We continue to run the machine until the up until the cost of continuing does not become too much and the cost of for the cost of continuing to become too much the evidence has to be there that the machine is probably broken. So, for that if the machine has so far so far never given you good never given you bad has never been has always given good inspect the inspection has always been good the outcome of the inspection has always been good then we really do not need to bother we just continue. So, it is G and G then you just continue and that you can that that is the optimal action you we continue even when the earlier reading was bad and the current reading is good we continue because this we it is it is a borderline almost borderline case you get that the cost of continuing is almost one it is 22 by 23 but it is still less than one. So, we continue but if but here in this case what is happened is that the bad rating was further in the past and the good rating was more recent maybe that is why the optimal action here is to continue. If when I switch these two when the good rating was in the past and the bad rating is more recent it turns out that the optimal action is to stop and inspect. So, the machine has gone from good state to bad state and you need to now take a look at the machine. So, the same has happened here when the machine has gone from good state to bad state we have we need to take an action S and stop and inspect the machine. So, this here has has told given us what the optimal thing to do at time one is. So, this is this is we got these eight cases because there were eight possible values for the information that we would have at time one. Now, so to summarize our policy is is the policy can be thought of at time one optimal policy. The optimal policy at time one is to continue if the result of the last inspection was G and to stop if the result of the last inspection was B. So, looking back at all of these we find that the optimal policy actually has a very simple and intuitive form as I have summarized as we have gone through above we can be summarized as following as follows the optimal policy at time one is to continue if the result of the last inspection was G and it is to stop if the result of the last inspection is B. So, if we find that the machine is broken as per the last inspection then we stop and we was in a bad state as per the last inspection then we just stop otherwise we continue that is what the optimal policy at time one is. Now, let us go to let us go to stage one the first stage that means K equal to 0. Let us now write this for K equal to 0 here we now need to plug in plug in whatever we have computed for for J 1. So, we have computed J 1 of I 1 for various values of I 1 that needs to be plugged in here now as part of the DP equation. So, here so in our at K equal to 1 this term was 0 whereas in at K equal to 0 this term is not 0 anymore. So, this needs to be we need to plug this in and as part of the as part of the DP equation. So, let us write out what the cost is from continuing and cost from stopping in at stage at K equal to 0. So, the cost from continuing the cost of action C remember this is going to be the sum of the expected stage wise cost and the expected conditional expected stage wise cost and the conditional expectation of the cost to go that we have computed in the previous step. So, the cost of C this remember is so once again we have a similar term we have two times the probability that the machine is broken given the information and given that we have taken an action C but now plus we have also this term which is the expectation of Z 1 given of J 1 of I 0 C and Z 1 given I 0 and C. So, this can be written more explicitly as twice probability that X 0 is equal to P bar I 0 comma C plus probability of Z 1 equal to G given I 0 comma C times J 1 of I 0 G C. So, let me write this in a different order let me write Z 1 first and then let us write the action that is that is the way we have written it so far. So, let us write this and then the action C. So, this is therefore I 0 G C plus probability of Z 1 equals B given I 0 C times J 1 of I 0 B C. So, the cost this is the cost of taking action C now the cost of stopping and inspecting and stopping and repairing this will now have the expected cost of stage wise cost which was 1 as before plus now the 8 an analogous term from the 1 above. So, you have J 1 of now I 0 Z 1 S given I 0 given I 0 S and that is equal to I 1 plus probability that Z 1 is now equal to equal to G given I 0 S times J 1 of I 0 G S plus probability that Z 1 is equal to B given I 0 S times J 1 of I 0 B S. So, this is the expression now J 1 of the J 1 at each of at for all these values can be computed by putting in specific values for I 0. Remember I 0 is just the information at time 0 the information at time 0 is simply the observation at the result of the first inspection. So, that can take only two possible values which is G and G and B. So, we let us put that in here. So, case 1 now is I 0 is equal to G. So, then we can just do a direct calculation from here. So, in this case we can compute each of these probabilities easily. So, we have the we need to compute the probability that Z 1 is G given that given G G C. So, we need to remember compute the following probabilities. So, we need to compute the following probabilities here. We need to compute this probability here. We need to compute this sort of these probabilities 1, 2, 3 and 4. These probabilities the four probabilities of related to Z are the probabilities that of seeing an observation G a certain observation at the next time step given that given the information. So, those probabilities need to be computed and we also need to compute the probability this particular probability here which is the probability that the machine is broken given the information we have so far. So, let us write out these probabilities probability that Z 1 is equal to G given G C is turns out to be 15 by 28 probability that Z 1 is B given G C is 1 minus that so it is 13 by 28 probability that Z 1 is equal to G given G S is 7 by 12 probability that Z 1 is equal to G given G S sorry Z 1 is equal to B given G S is 5 by 12 and the probability that X 0 is equal to P bar given G C is 1 by 7. So, in this in case 1 we now we see that these are the probabilities for the 4 terms that I have for the 5 terms that I have highlighted. So, this is what they turn out to be. So, as a result of this I we can now we can now write that J 0 of G is the minimum of the cost from C and the cost from S. So, the cost from C can be written out as 2 times 1 by 7 plus 15 by 28 J 1 of G G C plus 13 by 28 J 1 of G B C and so this is the cost from taking action C and the cost from stopping is 1 plus 7 by 12 J 1 of G G S plus 5 by 12 J 1 of G B S. So, this here is the cost from C this is your cost from S. We can we now have from our previous calculation we have values of J 1 for each of these that are written out here see notice here we have these, this, this, this, this and similarly this and this these are the 8 cases for which we have written out the values of J 1 and those can be now plugged in into this expression here in place of in place of these but these J 1's here. Once we plug those in and do the remaining calculations it turns out that J 0 of G is just 27 by 28 and the optimal action in when you have information G is to continue. So, the optimal action is when the first reading is good is to simply continue. A similar calculation can be done when the first reading is B in that case it turns out that so case this is for case 2 I 0 equal to B in that case also now J 0 of B is turns out to be 19 by 12 and mu 0 star of B the optimal action in that case is to stop. So, if it says basically that if the machine is in a bad state during the first inspection itself then you stop and inspect the machine and get a complete inspection done and repair the machine. So, the summary in this for this policy is that if we get a bad answer from the inspection then we stop and repair the machine otherwise we just continue. We now need to we have therefore now the J 0 the cost to go computed when the information is G and the information is B. We now need to compute the optimal cost J star we need the probability that the information is G or the information is B. So, the optimal cost J star is given by the probability that Z 0 is equal to G times J 0 is equal to G plus the J 0 of G plus probability that Z 0 is equal to B times J 0 of B. Now probability that Z 0 is equal to G can be computed Z 0 of equal to G this probability can be computed as 7 by 12 the probability of Z 0 equal to B is is 5 by 12. Remember this is here comes from our observation equations here and we need from here we need to compute we need to multiply we need to use we have these observation equations out here and we need to use that the initial the probability of the initial state being P 0 P bar or P that we from so we from there we will get these probabilities the probability of getting an observation good or bad and then we once we substitute this we get that J star is actually can be computed you can check this is equal to 176 by 144. So, this this example showed us a complete demonstration of how one we needs to do the needs to do dynamic programming in when you have you know when we when we actually have the incomplete or imperfect information of the state. So, we need to notice we have taken a long time to solve this particular problem and that is because the information vector is grows with time and therefore can take many number of values and we need to we need to write compute the DP equation at each step for each of the values of the information vector.


from the Palace of Fine Arts in San Francisco. It's theCUBE, covering AT&T Spark. Hello, I'm Maribel Lip as the founder of Lip has Research and I am here today at the AT&T Spark event in San Francisco and I have the great pleasure and honor of interviewing Maison Gilbert, who is the VP of Advanced Technology and Systems at AT&T. We've been talking a lot today and welcome Maison. Thank you Maribel for having us. We've been talking a lot today about 5G. 5G is like the first and foremost topic on a lot of people's mind that came to the event today. But I thought we might step back for those that aren't as familiar with 5G and maybe we could do a little 5G 101 with Maison. What's going on with 5G? Tell us about what 5G is and why it's so important to our future. 5G is not an allergy. It is, it really is a transformational and a revolution and really to not, to what we're doing as a company but to society and humanity in general. It would really free us to be mobile, untethered and to explore new experiences that we've never had before. So is it just, do I think of this as just faster 4G because we had 2G than 3G than 4G? Is 5G something different when you say allows us to be mobile and untethered? Don't we already have that? No, we don't. There are a lot of experiences that are not possible to do today. So imagine that having multiple teenagers experiencing virtual reality, augmented reality, all mobile while they are in the car all in different countries. We can't have that kind of an experience today. Imagine cars as we move towards autonomous cars. We cannot do autonomous cars today without the intelligence, the speed and the latency with 5G so that all cars connect and talk to each other in a split of a second. See I think that's one of the real benefits of this concept of 5G. So when you talk about 5G, 5G is yes, more bandwidth but also lower latency and that's gonna allow the things that you're talking about. I know that you also mentioned things such as telemedicine and first net network. Any other examples that you're seeing that you think are really gonna add a difference to people's lives going forward as they look at 5G? So 5G is a key enabler in terms of how things, how these experiences are going to really be transformed but when you bring in 5G with the edge compute because now today think of compute and storage and security and everything is sitting somewhere and as you are talking something goes to some unknown place in the 5G era with the edge, think of compute and storage as following you. Okay, so you're your own data center. You pretty much your own data centers and wherever you go with every corner there is a data center following you right there and now add to that we're transforming our network to be programmable with our software defined network and add AI into that bringing all of this diamond together, the 5G, the edge, programming the network with software defined and AI and that is what the new experiences is. This is when you'll start seeing really an autonomous world, a world in which that we're able to experience drones flying and repairing cell sites or repairing oil tanks without us really being involved about from being in a living room watching a movie. This is a world that is an extremely fascinating world, a world in which that people can interact and experience family reunion all virtually in the same room, but they're all in different countries. I do think there's this breakthrough power of connectivity. You know, we've talked about it in the next generation of telemedicine. You mentioned some of the dangerous jobs that we'd be able to use drones for not just for sort of hovering over people's gorgeous monuments or other things that we've seen as the initial deployment, something that's really meaningful. Now I know the other topic that has come up quite a bit is this topic of open source and you're in the advanced technology group. And sometimes I think that people don't equate the concept of open source with large established organizations like an AT&T, but yet you made the case that this was foundational and critical for your innovation. Can you tell us a little bit more about that? Yeah, open source is really part of our DNA. If you look at the inventions of the UNIX, you see C++ all originated from AT&T labs and Bell Labs. We've always been part of that open source community, but really in the past five years, I think open sources moved to a completely another level because now we're not talking about just open source. We're talking about open platforms. We're talking about open APIs. What that means is that we're now- A lot of open here. Everything open in here. And what that really means is that we no longer as one company, no one company in the world can make it on their own. The world- Okay, this is a big difference. It's a big difference. The world is getting smaller and companies together for us to really drive this transformational experiences, companies need to collaborate and work with each other. And this is really what open source is, is that think of what we've done with our software-defined network, what we call it own app in the open source. We started as a one company. And there was another, the Chinese, one of the Chinese mobile companies also had a source code in there. In the past one year, we now have over a hundred companies, some of the biggest brand companies or collaborating to building open APIs. Why the open source and open API is important. Enables collaboration, expedite innovation. We've done more in the past one year than what we could have done alone for 10 years. And that's really the power of open source and open platforms. I totally agree with you on this one. One of the things that we've really seen happen is as newer companies, these theoretically innovative companies have come online, cloud native companies. They've been very big on this open proponent, but we're also seeing large established companies move in the same direction. And it's allowing every organization to have that deeply innovative, flexible architecture that allows them to build new services without things breaking. So I think it's very exciting to see the breadth of companies that you had on stage talking about this and the breadth of companies that are now in that. And the other thing that's interesting about it is they're competitors as well, right? So there's that little bit of edgy co-opetition that's happening, but it's interesting to see that everybody feels that there's room for intense innovation in that space as well. So we've talked a little bit about open source. We've talked about 5G. You are in advanced technology, and I think we'd be remiss to not talk about the big two-letter acronym that's in the room that's not 5G, which would be AI. Tell me what's going on with AI. How are you guys thinking about it? What advice do you have for other organizations that are approaching it?  So maybe you could tee up a little bit about how that works. Yeah, AI is transformational and fundamental for AT&T. And we have always developed AI solutions, and we were the first to deploy AI in call centers 20 years ago. 20 years ago, really? 20 years ago. You were doing AI 20 years ago. See, this just goes to show. 20 years ago. I mean, AI really, if you go to the source of AI, it really goes in the 40s and 50s with pioneers like Shannon and others. But the first deployment in a commercial call center, not a pilot, was really by AT&T with a service we called it How May I Help You. And the reason we put that out, because our customers were annoyed with press 1 for this and press 2 for billing, they wanted to speak naturally. And so that we put the system that says, how may I help you? And how may I help you? Allow the customer to really tell us in their own language, their own words, what is it that they want from us, as opposed to really dictating to them what they have to say. Now, today, it's really very hard for you to call any company in the world without getting a service that uses some form of speech recognition or speech understanding. Thankfully. Thankfully. But where we are applying it today and have been for the past two, three years, we're finding some really amazing opportunities that we've never imagined before. AI in its press, in its essence, is nothing more than automation, leveraging data. So using your data as the oil, as the foundation, and driving automation, and that automation could be complete automation of a service or it could be helping a human to doing their job better and faster. It could be helping a doctor in finding information about patients that they couldn't have done by themselves by processing a million records altogether. We're doing the same thing at AT&T. The network is the most complex project ever to be created on the planet. And it's a complex projects that require, that changes every second of the day as people move around and they try different devices. And so that to be able to optimize that experience is really an AI problem. So we apply it today to identify where to build the next cell sites, all the way to what's the right to add to show to the customer, or how do we really make your life easier with our services without you really calling our call center? How do I diagnose and repair your setup box before you're calling us? All of that foundation is really starting to be driven by AI technologies. Very exciting. Well, I'm actually very excited to see where AI takes us and I'm excited to hear about what you're doing in the future. Thanks for taking the time to come here today and be with us on my queue. It's always a pleasure talking to you. Thank you very much. I'm Maribel Lopez, closing out at AT&T Spark. Thank you.


In the fast-paced world of content creation, being efficient is key to success. Video is here to stay and content is being posted everywhere. So how can you work smarter and not harder to keep up with the competition? That is where artificial intelligence comes in. AI can enhance your workflow and provide you with powerful tools to help improve your videos and remove mundane tasks. I haven't talked much about it on my channel, but because of how powerful it can be, I believe that it's important for content creators like you to know about the available AI tools that are out there and how to use it to your advantage. So in this video, I will introduce you to five powerful AI tools that will help you be more efficient in your content creation. First up is ChatGBT3 by OpenAI, which is free to use and can help plan your content. Sometimes it's difficult to come up with new ideas and with ChatGBT, you can find some new inspiration for your video. Think of ChatGBT as your robot friend that can answer questions. If you have a channel about dogs, you can ask ChatGBT to give you 10 content ideas about dogs. And ChatGBT will then generate 10 ideas in seconds, giving you a quick and easy way to get ideas for your content. The first one being the benefits of having a dog discuss the various physical, emotional and mental health benefits of owning a dog. And because ChatGBT remembers your conversation, you can continue asking ChatGBT to come up with 10 titles for the generated content idea. However, the best way to use ChatGBT is to be as specific as possible. The more specific you are, the more accurate results you will get. So I'm gonna pick the first one, I'm gonna copy it and then I'm gonna ask ChatGBT generate 10 title ideas for the topic and I'm gonna paste that title in it, press enter. And from there on, you can continue and start structuring your script and ask ChatGBT to outline five tips and paste the title idea you've chosen. I'm gonna go with the third one, which is how dogs can improve your mental health and reduce stress. So I'm gonna copy this one and then I'm gonna ask ChatGBT to outline five tips on the topic then paste the title in it. And as you can see, based on what I've wrote, it will give me five reasons on how dogs can improve your mental health and reduce stress. So a dog can provide unconditional love and companionship and so on and it structures it really nicely. And you can even ask ChatGBT to write an intro for you based on the title. So I will ask ChatGBT to write me an introduction on the topic and then I will paste in the title. And you will see that it will write it out for you and you now have an engaging intro that you can use to grab your viewers' attention. Now with these five tips or reasons that we have, you can even ask ChatGBT to write a full script about one of these tips mentioned here. Write me a blog on encourage physical activity. So it now generated the script and it did a really good job. It says dogs are known for being active and energetic animals. They love to run, jump and play. But did you know that owning a dog can also encourage physical activity in their owners? And what you can do is copy everything and put it into a Word document so that you can structure it better. And then from there on you can read it through and if you need to make any changes or want ChatGBT to rewrite it, it will do so by just asking for it. Now I highly recommend you test and play around with ChatGBT to get the best result. It does a very good job of writing the script. However, I wouldn't want to use the script immediately for your videos. You want to make sure to check the information so it's correct and adjust the wording to your personality, which is why the scripts are more like sketches and need to be polished by you before you start producing your video. Next up is Adobe AI Speech Enhancement which is currently running in beta and is free to use. Audio is one of the most important aspects of any video and while you might have the best camera and equipment, if your audio doesn't sound good, it will impact the quality of your video. Now Adobe's AI Speech Enhancement is a great tool that can help improve the audio in your video by reducing background noise, enhancing speech, and much more. So I'm here at the construction site and you can probably hear a lot of background noise and for situations like these, the Speech Enhancer by Adobe is a great option and this is how it would sound like once you applied the Speech Enhancer. So this sounds like with the Speech Enhancer applied, a pretty powerful tool, you can't hear any of the background noise and my voice is also more enhanced. So for situations like these where you really want to get rid of the background noise, this is a great option. I usually make sure to record my audio in quiet environments but sometimes that's not possible. Now I do most of the audio processing in Final Cut Pro. However, for someone who isn't advanced in audio editing, Adobe AI is a great tool for speech enhancement. So I can simply upload the audio file in it. So once it has been processed, you can download it to your folder and use it in your videos. So you now have no excuses for bad audio and I recommend using this tool whenever you find yourself struggling with bad audio. By the way, I will leave links to all AI products in the video description below. So the next AI tool is Timebolt, which automatically removes silence in your spoken videos. This is such a powerful tool and even though I'm a fast editor and have optimized my editing workflow over the years, nothing comes closer to having a tool that does this for you and has saved me hours of editing time. This tool automatically removes silence from my A-rolls. All you have to do is upload the video file to Timebolt and let it do its thing. I then export it as an XML file and import it into Final Cut Pro, which gives me the edited A-roll version with the silence cut out. Timebolt is also great for editing podcasts as it can identify moments of silence and remove them with one click. Timebolt does have a trial version with a watermark applied so that you can see the results. They have several pricing options. You could go for a monthly or yearly subscription or if you're like me who hates subscription plans, you can buy it once and own it for a lifetime. I'm not sponsored by them. I just think it saves me a lot of time and increases my workflow speed. Next on the list is auto-generating your captions using CapCut. CapCut is a free editing program by ByteDance, famous for creating the well-known app TikTok. I recently made a video about it. The benefits of using CapCut is that it comes for free. There is also a pro version, but the free version has all the basic tools you need and is available on desktop and mobile. It is packed with tons of features and one powerful tool is being able to auto-generate captions using AI. You see captions being used a lot in short form content. The reason why it's important to have captions as part of your video is because it increases the overall viewing experience and with 5% of the world's population being deaf, captions are extremely important for accessibility. So the way you can add captions inside CapCut is pretty easy. Once you imported your talking head clip into the timeline, you can head over to text, select auto-captions, make sure to select the right language and hit create. It will then automatically detect the speech in your video and transcribe it into captions. From there on, you can look through your captions and make any necessary adjustments in case of spelling mistakes. You can also change the font, text, size, color and more to match your brand. So if you have been manually typing out your captions, you no longer need to as CapCut's auto captions can save you a lot of time and energy. Last, we have Topaz Video AI from Topaz Labs, which in a nutshell makes your footage look better. It's a standalone desktop application where you can import and enhance your videos using AI technology. There are several options that you can choose from. You can upscale your videos from low resolution to high resolution. For example, I took this video that was shot on my iPhone in 1080p and upscaled it to 4K and it does a very good job of filling in those missing details and it looks surprisingly natural and authentic. Another cool thing is that you can turn your video into slow motion. Now the shot that I took was shot in 60 frames per second and I was able to slow it down four times and it still looked buttery smooth. Every time I try this option, I'm amazed at how well it does. It takes your footage and doesn't matter what frame rate you shot it in, it can be 24, 30 frames per second and takes those frames, stretches them out and creates these new frames so that you have more frames per second to be slowed down. Final Cut Pro has something called Optical Flow. However, it doesn't work every time and it's not as good as Toppas Video AI. For someone like me who captures video on my iPhone, having a tool like Toppas Video AI can help make my footage look as best as possible. I don't use it all the time because it does require a lot of computer powering and especially for longer clips, it can take some time. However, there are moments when the footage wasn't captured in the best way so having this tool to clean it up and make it look better is extremely helpful. Now Toppas Video AI has some other useful features which I haven't gone into but I find these two features upscaling and creating slow motion video to be the most beneficial. I highly recommend you test it out. There is a free trial version available that comes with a watermark but if you decide to purchase it, it is a one-time payment which is great. I haven't been paid by them to say this but I've seen other reviews and bought it myself and I really think it's worth the investment. What's great is that once you buy it, you own it for a life. So these are the five AI tools I use to save me a ton of time and help improve my videos. I think it's important to be open-minded and willing to try out new tools that can make your workflow more efficient. Time is one of our most valuable resources and for me as a content creator if I can find ways to save time and be more productive, I'm all for it. So I hope you found this video useful and let me know in the comments if there are any other AI tools you use that help you in your video creation. Until then, thank you so much for watching guys and see you the next time.


Hello, my name is Patty Angelis and I am running for a District 2 Town Council seat. I hope that by taking a moment to view this video you'll find reasons to support my candidacy. I moved to Amherst with my partner Carol and our son Jesse 25 years ago for two very important reasons, the quality of the Amherst public schools and our desire to live in a diverse and collaborative community, the Pomeroy Lane Cooperative, which was conceived by abodes whose members were parents of physically and cognitively challenged adults. Our family along with 24 other families moved into the co-op. We were intentionally diverse. Our households were labeled low-income, section 8 and market rate. We were Asian, European, African and Latinx Americans. We were gay and straight. We had varying degrees of understanding about issues of social, economic and environmental justice. We were ordinary people embarking on an extraordinary adventure and along the way we changed. The assumptions that we held about each other were blown apart and we began to see the strength and creativity inherent in our community. We learned to see and acknowledge our differences and find ways to collaboratively solve the problems that arose. This experience, this community showed me what Amherst could be, provided lessons I've carried forward into my work as a teacher, an activist, a town meeting member and now a council candidate. We face a variety of issues as a town, perhaps the most challenging being the divisions that have occurred over our schools, our form of government and who gets heard, divisions that need to be bridged. As candidates we talk about appropriate development, sustainability, social justice, funding capital projects and preserving the character of our town, but we often mean very different things. We talk, but we need to start listening. We need to find what binds us and build out from there. So how would I look at the challenges we are facing as a town? What values would support my decision-making and ability to collaborate with other counselors as we look at economic and development issues, create policies and procedures that foster sustainability, focus on the quality of our schools while balancing issues of social, economic and environmental justice. One of the most important lessons I learned living in the co-op and also as a classroom teacher was the need for all voices to be heard, for all ideas to be considered, researched, analyzed and reconsidered. I expect to be actively involved in using citizen advisory groups composed of the people who would be most impacted by our decisions as well as other stakeholders. The goal of the committees would be to truly reflect the values and meet the needs of the community we serve. My son Jesse went through the Amherst public schools finding himself challenged, listened to and taken seriously, which was not the case for all of his classmates. As a teacher and learner, I care deeply about the educational, emotional and physical spaces where our children learn. As an educator, I hope to be involved in meetings with the school committee, teachers, administrators and most importantly our students and their parents to get a clear understanding of need and to find ways to provide that need. I feel strongly that we need to continue to develop downtown areas and village centers so that they meet the needs of homeowners and renters, students, young families and the elderly as well as residents who are physically challenged and residents who are homeless or find themselves in need. And I believe that we can work with developers, business people and neighborhood people to both preserve the character of our town and make sustainable changes, changes that are green, creative, long lasting and that benefit all the members of our town community now and in the future. And I will bring to the, what I will bring to the council is curiosity, a willingness to research issues in multiple ways and a belief in collaboration. I am a person of integrity and determination who will foster progressive change and amours, change that rests on the best of what we have already created and the ways we are growing and challenging our community. Thank you.


In memory of W.B. Yates, who died in January 1939, he disappeared in the dead of winter. The brooks were frozen, the airports almost deserted, and snow disfigured the public statues. The mercury sank in the mouth of the dying day. What instruments, we have agreed, the day of his death was a dark, cold day. Far from his illness, the wolves ran on through the evergreen forests. The peasant river was un-tempted by the fashionable quays. By morning, by morning tongues, the death of the poet was kept from his poems. But for him, it was the last afternoon as himself, an afternoon of nurses and rumours. The provinces of his body revolted. The squares of his mind were empty. Silence invaded the suburbs. The current of his feeling failed. He became his admirers. Now he is scattered among a hundred cities and wholly given over to unfamiliar affections to find his happiness in another kind of wood and be punished under a foreign code of conscience. The words of a dead man are modified in the guts of the living. But in the importance and noise of tomorrow, when the broken, when the brokers are roaring like beasts on the floor of the bowers and the poor have the sufferings to which they are fairly accustomed, and each in the cell of himself is almost convinced of his freedom, a few thousand will think of this day as one thinks of a day when one did something slightly unusual. What instruments we have agree. The day of his death was a dark, cold day. Part two.  Your gifts survived it all. The perish of rich women, physical decay yourself, mad Ireland hurt you into poetry. Now Ireland has her madness and her weather still for poetry makes nothing happen. It survives in the valley of its making where executives would never want to tamper, flows on south from ranches of isolation and the busy griefs, raw towns that we believe and die in. It survives a way of happening. A mouth. Part three. Earth. Receive an honored guest. William Yates is laid to rest. Let the Irish vessel lie emptied of its poetry. In the nightmare of the dark, all the dogs of Europe bark and the living nations wait, each sequestered in its hate. Intellectual disgrace stares from every human face and the seas of pity lie locked and frozen in each eye. Follow poet, follow right to the bottom of the night with your unconstraining voice still persuade us to rejoice with the farming of a verse. Make a vineyard of the curse, seeing of human unsuccess in a rapture of distress in the deserts of the heart. Let the healing fountain start in the prison of his days. Teach the free man how to praise February 1939.