Hey y'all, what's up? My name is Sonali and this is my boyfriend Ryan and we are filming our six month update of living here in Austin, Texas. Quick background about us. I graduated from UCF in May. And I graduated from Georgia in May. And we moved here in Austin in June because... Because I got hired at Oracle here in Austin. It's like one of Oracle's big hubs.  Yep, so that's why we're here and I'm currently doing freelance photography and videography and obviously influencer stuff on the side as well. So, all right, let's get into the questions that you guys asked us on my Instagram. The first question is how did you tell your parents you were moving in with Ryan? Did you have to convince them? It's actually kind of funny because I didn't really have to convince them at all. And I was like really shook that I didn't because I feel like it was just too easy. Like everything was just too easy. In high school, I thought my parents were really strict. I think I talked about it in a previous Q&A that we did. So I'll definitely link that down below because I say how I kept him a secret from my Indian parents for like two years. But anyways, yeah, so I thought they were really strict. Obviously when me and my sister got older, I guess they had less rules and kind of just let us live our lives. So they're very supportive of me being here and they just want me to be happy and do good things and be successful. So my mom didn't care if I moved in with her. My brother lives with his girlfriend and lived with her through college. So she wasn't very opinionated on it, I guess. Yeah. Is it hard to balance finances now that you're out of college and independent? I would say yes and no. I obviously had a budget in college too. So it's nothing really that new. Yeah, I mean, I had my mom take care of most of the stuff I was doing in college but now on my own, I have to be conscious of what I'm spending. I mean, it's not super expensive to live here. So it's not like I'm ever worried.  If you guys are interested in our apartment and how much money we pay to live here in Austin, Texas, I will link that video down below and we kind of go through the apartment tour hunting process so you guys can watch that after you guys watch this video. What was the hardest part about moving to Austin? For me, it was like the idea that I hadn't been here. So it was kind of nerve-wracking to move here. That's so true. But- I keep forgetting. I mean, we'd never toured it or anything before. Yeah, we never visited here before we moved here because Ryan got the job and we were like, nothing is gonna change, yeah. So like, there's no point of wasting money to come here for no reason. But I would say like finding an apartment that was stressful. Especially because we had an Airbnb for like, what three, four days when we first got here. And so we were just like stressed out because we wanted to like just go ahead and like pay all the fees for an apartment instead of like paying for an Airbnb. So that was really stressful because we wanted to obviously get a nice place but we didn't have that much time, I guess. Yeah, an understanding where like a nice place to live is versus traffic patterns and all that. Especially because we didn't know anything. We didn't really know people here. So it was definitely hard. And that's why I'm making these videos because I definitely want to help other people who are moving here. And I know a lot of people are moving to Austin. So what's one thing to keep in mind while growing up and getting through college? One thing to keep in mind is like kind of about light at the end of the tunnel. Like it was easy for me in college to kind of lose sight of like why I'm even there in the first place. Like what's gonna happen after? Is it really worth all this hard work? I mean, it depends what you're studying or what profession you're going into. But life after college is a lot better in my opinion. I would say my one thing while going through college would be to get as involved as you can if it's on campus and your community and your sorority because that's just gonna make your college experience and just growing up so much better. How has living with Ryan impacted slash changed your relationship? I feel like it's just so easy. I feel like it just comes so easy and I feel like it feels so right. Does that make sense? Sometimes she's a little too much. I have to have my alone time. She doesn't like that because I'm like at work all day around people and talk to people all day. Yeah, and I'm alone all day.  Yeah, that's when I come home and I crawl into a hole and I sleep. I guess we just have to learn each other's boundaries a little bit more because we are together like almost 24 seven. So that's kind of different, but I don't know. I personally think like it feels just so normal. I feel like it just nothing was ever like different. It doesn't even feel like we did long distance, you know? Yeah. I mean, you kind of lived in my house during like Christmas breaks, summer breaks. Yeah, yeah. You had test trials. Yeah, but now we have our own place and I actually have my own closet and like my own space. So it's nice. What is your favorite part about living in Austin? I would say that Austin is such like a compact city. So you can really get anywhere within like 20 minutes without traffic, I would say. And I personally just like hate driving. This is in comparison to Atlanta, which is extremely spread out. Yes. So I don't know about other cities and maybe similar to Austin. I mean, even Orlando where I went to college, the school itself was in Orlando, but like Disney World, which is, you know, known to be in Orlando is actually in Kissimmee. So that's like 45 minutes away. The beach is like an hour away. Downtown's 30 minutes away. Everything's pretty much really spread out and same goes for Atlanta. Like our family lives in like all these different cities that are like 45 minutes away from each other. So it's just like very spread out. So that's what I really, really like about Austin. And there's so much to do here. I feel like we never get bored. He has different thoughts. I don't know what's up to do here. I don't know. I feel like there is so much to do here. It's a big college town. I don't think so. The best part is by far the food. Oh my God. The food here is so good. So good. That when I go back to Atlanta, I'm like, I don't know what to even eat. I didn't even know what like real barbecue was before we had it here. But yeah, it's very different here. So freaking good though. I love the food here. The Tex-Mex is so good here too.  Like I don't really crave it like Ryan does and it's everywhere here. Like tacos is like the big thing here in Texas. So well, at least in Austin. Would you recommend moving to Austin? Yes. Yeah. I think so. It's definitely grown on me a lot. And I think it's kind of getting like the experience of being in a city that's not that big but not small either. It's nice. Like when we were planning on moving here, I was trying to get just like advice from everyone else that have had visited. And everyone always said it was like a young city to be in. So I think it's a really cool like transition city, especially if you just graduated from college and are like starting your like post-grad life. I think there are just so many young people here to just make friends with. Like everyone's moving here for like jobs and tech is blowing up here. So I would definitely recommend it to like recent grads and young people. But I mean also old people because Austin is just a cool city in general. Do you guys plan on settling in Austin and buying a house? Or do you plan on settling somewhere else? I don't know if I could see myself buying a house here. I feel like I'm just wanting to like see other cities if possible. But I really do like it here in Austin. Like I don't know if another city could really top this, but like I don't know. I mean I could see myself buying a house here as an investment. Yeah. And then like I could see myself coming back here as like a 30 or four year old. And I have like, I don't know there's like a study that says you need to make like $136,000 in Austin to be happy. And if I made that much money and lived here if I could be a lot more fun. I really like Atlanta. And that's like where all my family is. Yeah, that's where we're both from. Yeah. It just kind of makes more sense for me to be there. But the tech industry there is pretty advanced in terms of like entry level roles. So I'd have to be older to kind of get the job that I want there. Gotcha. But yeah, I could see myself selling down in Atlanta. Yeah. In Austin, it's pretty similar to Atlanta. It's a little more like. It does have a lot of similarities. But I feel like it's way more similar to California than Atlanta. I would say it's way more similar to Athens in Atlanta. They're really big Athens. Okay, okay. I'll do that. Yeah, you know what I mean? Yeah, for sure. Do you like it here? Yes. What are some things you don't like about Austin? I don't like the traffic. The traffic's just as bad as Atlanta. It's actually really bad. The road's really bad. The only good part about the traffic is you're not going more than 10 miles. Yeah. In Atlanta, you drive 30 miles to work. You're in there traffic two hours. Yeah. And then the roads are really like not paved well. Oh my gosh. Especially if you're in an Uber and you're not sober, it does not feel good. But even when you're sober and you're driving, it also does not feel good. Yeah. Yeah, the roads, I don't know. They need to do something with that. And there are toll roads here. So like, why aren't they better? I never know when a toll starts, when it ends, when I know anything. It's weird because the tolls here, like you have to have the Texas tag. You can't just pay with coins like Florida. You have the option to pay with coins or you can like get it mailed to you. But here they just mail it to you if you don't have the Texas tag.  Can you pay it? So yeah, that's annoying. Okay, so when we first moved here, I thought it was dirty and not the nicest looking place. But it really just has to grow on you. Like give it two weeks. No, I think it's more during summer. Everything's really overgrown. Oh my God, it's hot. That's like the biggest thing. It's super freaking hot here. It's overgrown. And then like- What do you mean overgrown? Like the trees, there's like ivy everywhere. Like it just like doesn't look very like, that's what looks unkept. And then when I first moved here, I'd never seen that many homeless people before. Oh yeah. So like when I saw my first came here, I was like, wow, this place is very dirty and like homeless people everywhere. Especially, I will say in the part of Austin that we do live in, it's not the nicest, but like it's still safe. It's like up and coming. Yeah, it's up and coming. So. Yeah. That's what- That's what we say. That's what we like to say to make us feel better, right? Yeah. Yeah, but I mean there are so many like pros that outweigh the cons of Austin. So what's the best thing about moving to a new city? I think just exploring everything. I think it's so fun.  You know, you get to spread it out. You get to do things when the weather is perfect for that activity and like, like you really just get to enjoy the city. Yeah. I think that's the best part of moving to a city. What do you think? I think it's kind of like a clean slate. You can like make new favorite restaurants, meet new people. I was gonna say like it's definitely hard meeting new people but it's also like fun and exciting to do that. Next question is, how did you know you were ready to live together? It's my metaphor to do it. Yeah.  But since he got a job in Austin, I was like, you know what, I'm gonna come too. So obviously we just, you know. You didn't want me to come. I wouldn't even apply for this job. I know. I kind of, I kind of not pushed you but just definitely persuaded him because I wanted to go to a new city. So I think it was just like the convenience of it. But I was always wanting to live with you. So that worked out. Would you move back to Orlando or even back to Florida? Yeah, we're gonna retire in the seaside. Yeah. At 30. I've said this like a million times but like our high school would always go to like the Panhandle of Florida for spring breaks, like seaside, deaths in area. So we are like obsessed with that area and you know, that'd be nice to get a vacation home there one of these days, but probably not to Orlando. I don't know about that. I like Orlando. Honestly, I've lived there. You like it? Seriously? Like downtown Orlando? I mean like it's cool, but like, I don't know. I feel like it's like small and there's not much to do. No, but there's like not much to do. There's like- Because you did it for four years. Oh, wait a minute.  I guess so. You got a point. I don't know. But I feel like Austin is just so picturesque and like Orlando is just not, I don't know. That's my thoughts on that. How do you stay confident during a post-grad job search? I couldn't help you with this one because I'm not very confident in it myself, honestly. Yeah, I mean, when I was searching for jobs, I had to like, I probably applied for over a hundred jobs or more. And then I only did about 10 interviews maybe. Yeah, so it's like not, I mean, there's so many people graduating and so many jobs out there. It's like discouraged if you don't get the one view, the one you think you want. Easier said than done, huh? Yeah. Last question is where is the best place to eat in Austin? There's too many places to eat in Austin, first of all. Yeah. If you ate somewhere every day, you would never even like reach half of it. I don't think so. There's no way. And you would also be very broke. But I think, I don't know, where are some good places? My personal favorite, which I keep top secret. Okay. Is this taco truck on our house that is so good and it's never really that packed. Yeah. But it's only open from like 6 p.m. to 3 a.m. So you can like never really go there at a good time. Yeah. But they're like incredible. I would say for sushi, we just ate all you can eat at Sushi Janai and it was so good. I liked Matty's brunch, Matty's, so good and so beautiful like the restaurant area is so beautiful. Damn, I'm like blinking out. Terry Blacks. Terry Blacks is a good, good one.  And just, we like regret it so hard afterwards, but it's so yummy. It's barbecue. So very good. Gordo's public house has the best fried chicken here. Yes. Gordo's big fat donuts. I don't even know why that didn't come to my mind, which is a good thing because those are like my weakness. So good. Get the Cinnabomb. I don't know. Do you guys want me to do a blog post on Austin restaurants? Let me know in the comments down below and I'll write that up. So if we're going to rank one out of five, how we like Austin after six months, my personal grade would probably be like a four, I guess. Eight. Mine would be a five for sure. I like it here a lot. And I never want to leave. You just said you did. I know, I know. I want to explore more cities, but I really, really love Austin. Like truly, truly, I really like it here. So. I don't want to move somewhere where there's not a view, but there's no but really around here that has a view. Wait, you mean a city view? Yeah. Oh. I mean, let's move downtown. Yeah. What an elevator though. Yeah. We're on the third floor right now and it gets real hot in the summer. So we're always winded, but I hope you guys enjoyed this video. I will be linking all my Austin videos and my moving blogs down below so you guys can binge watch them. If you guys are moving to Austin and have any questions for us, feel free to DM me on Instagram or comment on this video. And I will be sure to get back to you guys. So I will see y'all in my next video. Bye. Bye.


new stuff that we introduced in the past few quarters. One of it is hardening, which is yeah, based on Openscape. Basically, how this works. So Openscape is a tool, you give it a profile and it scans your whole system and applies some remediations and these profiles are basically based on certifications. So if you need certification ABC, you apply a profile ABC and Openscape will try to do as much as possible so your system is certifiable by the ABC certification. And ImageBuilder can apply such remediations on the build time. So once again, the system is hardened from the very beginning, from the very first boot, which helps to establish a secure pipeline. Also, a lot of the certifications require a specific partitioning and it kinda ties together, right? You can use ImageBuilder to do the custom partitioning and also immediately harden the files like the system. So these two features are very often used together. We can inject extra files into the image and that's also very useful because you install a web server, you can enable it and you can also configure it so you can have an image that immediately starts to behave as a reverse proxy, for example. And the feature that I like maybe most from this list is embed containers. What this means, when you tell ImageBuilder to embed a container during build time, it will download the container from the container registry and put it into the right directories. So when you run Portman or MicroShift, it doesn't need to pull that container but it immediately has it on the system. And this is great if you have a disconnected environment because then of course you don't have any container registries or for the IoT slash Edge use case because your device might be on top of a hail and there is no good internet connection and you don't want to wait 10 minutes for a container to download for a registry. So that's quite useful for these kinds of scenarios. And yeah, I just wanted to mention that OpenScape, by the way, and so you can, so you know what it does. So for example, it can do some small changes to the configuration like changing, configuration auditing, SSH configs, OpenSSL configs. I mean, base distributions are nobody's pretty secure but these certifications require more and we don't want to put this config in the base system because sometimes the system might behave weird because users want to use Kaina, you know, older hashes and things like this. So it's always about trade-offs. Anyway, there is one more thing that ImageBuilder can do and it can immediately take care of your uploads to the clouds. Which is pretty cool because because if you've ever tried to do multi-cloud image uploads, it's a pain. Like I uploaded an image to all of these and IBM cloud actually also and OpenStack. And all of these CLIs are different. The process is different. You need different compartments. That's my favorite word from Oracle Cloud. You need different storage groups, storage accounts, storage containers, whatever. It's crazy. And of course you need to have all of these CLIs installed on your machine and it's annoying. But ImageBuilder abstracted this away. You just give it a config and it will build an image and immediately uploads it to a cloud. Which is very cool because to be honest, we believe that this should be more, like more tools should do this because building an Azure image locally is fine. But what are you going to do with it? Yeah, you can boot it with QM, but it's an image meant for Azure. So it really should end up in Azure. So we are trying to tie these two processes together. Because it really makes sense. Anyway, yeah. So the ImageBuilder that you can install locally, it's just both GUI and CLI. And at least my understanding or my vision is that you can play with the GUI and see what ImageBuilder can do and kind of click around and add customizations, define everything nice and visually. And then when you are done, you can just export the blueprint. And then you can just use the CLI and set up an automated CI pipeline. So yeah, I consider this as a two-step process every time. Yeah, and if you want to build an image or start building images, it's very simple to install, just DNF install two packages. One is the CLI, one is the GUI, and then enable the service that runs on the background and builds images. And you can start building. It's that easy. Yeah, good. Now I promise that I will talk also about other options how to consume ImageBuilder. And it can be also run as a web service. And currently the web service has two clients, one, well, three. Yeah. One builds golden rel images and one builds a golden currently federal IoT images. So that means that ImageBuilder can be integrated with Koji. And it is, it builds images every day. The second client is our ImageBuilding service in Red Hat Hybrid Cloud Console, where you can just grab the free rel subscription. It's fine, go to console, there's a link here, console.redhat.com slash inside slash ImageBuilder. And Red Hat will build the image for you and you don't need to maintain an infra, you will just send you a link or share the image into your account and it will just work. So my message here is that ImageBuilder is run in a production environment and it's suitable for everyday use. So just use it, it's stable, it's fine. Good. Is that that? Yeah, and I should also mention that that the service has also an API so you can also automate your CI pipeline using the hosted service. Which is pretty neat because you can just call it from a container in GitLab CI and it will just get up actions and it will build you an image. There should have been an image of the service, but yeah. Anyway, and just two items for the future. We would like to collaborate more on using ImageBuilder to build more federal artifacts. Currently the project is with the installer team. Hello, because we are talking together to building Federa installers because we would like to modernize the ImageBuilding stack. And also our big dream is to enable the community to easily build and share customized Federa images because we think that the process currently, you know there's Koji, Banji, Relang and it's like, it's pretty, it's pretty a big process. So we want to make something that would simplify this process by a lot. I also forgot to mention at this slide, the ImageBuilding service, I just like how everything ties together in this conference like with Kubeware, I mentioned, I mentioned, what did I mention, REL4H and Federa IoT and the service, it's actually tied to the keynote because it's an open source service. And yeah, it's one of the, we try to follow as much open service principles as possible and Simon here is going to have a talk right after this talk about open services. So you can go there. Anyway, that's a bit of promo. It's demo time. Good, so this will be the part which can go wrong at any time. So I have this small shell script. I will show you the script in a bit, but I will build a config called demo for this demo and it will be a QCOW too. So we will boot it in a bit with KVM with just QMO. So I will start it and now I have about five minutes to explain you what's going on because the image will be ready in five minutes. No uploads will be done because of the Wi-Fi here. Yeah, I don't want to risk it. So it will be local, but whatever. So the image is building. By the way, I will quickly show you the script. Yeah, it's pretty short. I have some debug stuff in there, but basically I'm just calling Composer CLI with a bunch of commands and then parsing it. It's pretty simple. I made this script so I don't need to go over all the steps because they can go wrong. So I automated it a bit. My script, by the way, will be available at the archive of my talk after the talk, probably. Anyway, let's go to the configuration. We call this the blueprint and this is what ImageBuilder consumes. It's a Tomo file and you can put a lot of stuff in it. So let's go over it and see what our image, when it's done being built, what it will contain. So a nice feature of ImageBuilder is that it can do cross distribution builds quite easily. So my system is actually running Fedora 38, but for some reason I buy more different distribution. So the image will actually be Fedora 37. So it's completely possible to do that with ImageBuilder. You can also, for example, build CentOStreme 8 or CentOStreme 9 and yeah, that's it. The next thing that I want to have in my image is Portman because I want to demo the embedded container thing and I need a container, how do you call it? Container engine. So I install Portman and some optional dependencies. I'm not really sure why they are actually needed, but that's a story for another day. Not sure if that's a Portman bug or I don't know. So yeah, this will install Portman inside of the image. Then I want to embed a container. So I can just tell ImageBuilder, hey download me this container and yeah, I give it the source which is my repository at gitlab.com. It actually contains the slides of this talk in web server and yeah, it will just save it locally as a talk container, just talk. It will be named talk and that's it. But you know, download the container, that's boring. So let's run it. And for this I am using the new Portman quad-let generator. So let's explain it a bit. It's very cool. Yeah, it's cool, right? So basically, it looks like a SystemD unit and SystemD can consume it via its generator thingy thing that Portman implemented. And so it is a SystemD unit, right? It has a unit thing and it apparently runs after network target. So after the networking is up and it can be even installed. So like when I install it, it will run or it will be required by the multi-user or default target. So it's a SystemD unit, but there is no service but there is a weird container section which basically the generator will translate it to a service and it will translate this to a Portman running Portman container. So I can just tell it, hey, take this talk image which I downloaded in the previous step and you can do the usual stuff that Portman can do. So I can just tell it, hey, publish me these three ports because it's an HTTP server. So I want 80 and 4443. And of course, the container is embedded. So I can just tell Portman, hey, don't ever pull it. You already have it in the store. Yeah, so that will save me some bandwidth potentially. So I also expect maybe with my image that I will need to store more data. So I put there an extra bar partition with 10 gigs of size and yeah, it will be there. So this is the example of custom partitioning. Yeah, there is just a quick note, but we have a support not yet in our image builder, but we are getting there slowly. So this will be just X4, but I mean for the cloud, I don't really care. So it's fine, I think. Then let's harden the image, which is this section. Basically, this tells the OpenScape tool that's run during the build time to apply from this file. This is like a real file that you can install on your system. This profile and then OpenScape will run anything was defined in this file and just scanning. As I told you previously, it can, I don't know, configure SSH to not allow password keys. Well, that's disabled by default in Fedora, but I don't know. Some other stuff like enable USB guard and yeah, whatever is needed in terms of security. And there are different profiles and on our website there are some examples of other profiles that you can use and you can also see the OpenScape documentation for more details. Anyway, let's go to the next customization we can add at firewall. So just install firewall D and immediately configure it. So we want to allow HTTP and SSH and that means that all the other ports should be closed. And the last thing is adding a user. As I promised you before, image builder can add users. And yeah, I can just create an admin user. I can add my key, this is my public SSH key and I can add it to the real group. But the issue is that, so the real group is for accessing sudo. And the issue is that Fedora by default doesn't allow passwordless sudo, but that's fixable by the first customization because I can just add a drop-in and real will be passwordless. So that's possible. Good, let me just quickly open the front end. This is actually the front end. So you can see that the demo blueprint that I put into image builder via the script it's there. So it's the same thing as I did on the CLI just visualize here. And you can see that everything that we talked about is here. So there is the openscape thing, openscape remediation, there are the file system options, firewall users, if I go to packages, there are some extra packages there. They are also with versions, which is pretty nice. Custom files are currently not there. We still need to add them, but otherwise everything is there. And you can just click create image here, select that you want QCOW2 and it will build an image for you and you didn't need to touch the CLI, but you like CLI, right? Anyway, so yeah, while I was talking, it took about five minutes to build the image and I downloaded it as an image, as an file and I will just use a simple script that I have to deploy it. It's just calls QEMU and yeah, forward some ports, it's nothing super amazing. You can also use LibVirt if you like LibVirt and when you install it, it doesn't really matter, it's there's no difference. And the image is now booting and I should be very quickly able to SSH into it. This is the scary part of the demo because it worked like 20 times in a row. Good, it worked. So this is the image and we can go over the customization. So Fedora 37, let's quickly verify it. It is Fedora 37, let's see if the container is running. So the container is there, yeah, it's the image that I had in the blueprint. You can even see that the image is installed. I can even curl to there. Yeah, there is something. I can even show you in the browser that if I go to the localhost 8080, it's there. I just forwarded the part. So that works, so everything is good. And what's there, what's next? Next was, this was shown. Oh, let's show the partitioning. So if I do df-h, you can see that var is there. It has 10 gigs or 9.8. Oh no, it has 10 gigs, right. So the custom partition is there. And yeah, what else? Firewall, so firewall. Let's see if I can do it on the first attempt, list all. No. Oh, good. So by default Fedora enables the HTTP v6 client and MDNS and we edit HTTP and SSH. So yeah, this also worked and the firewall is actually enabled. And I think, oh, I edit the user, yeah. And the user, yeah, sure, I am logged in as admin and it didn't ask for a password. So apparently my key is there and I used to do without any password. And yeah, that's my demo. And I think that I am almost out of slides. Yeah, I am totally out of slides. So yeah, image builder. I think that it's a great tool and not only because I work on it, but also because I like it. And I think that with these new options it can get you pretty far if you need a customized image. There is a lot that you can do to basically build your image, push it into a cloud, launch an instance and it will run its workload as intended. So yeah, this is image builder and I'm glad that you were here for this talk. And if you have any questions, please I'm here to answer them. That's a great question. So I will repeat it. Thank you. So the question was how does the environments I showed KUKAW or QEMU, but there were also Azure, AWS and other ones and how do they differ? So there are some slight differences. For example, on Azure, you want to install their Azure client. It can do some more integration. Agent, ah, agent. On Google that's the same thing. Google has their own agent. So you want to install some extra packages. And yeah, also sometimes the configuration is a bit different, for example. It's not done actually for Fedora, but maybe we should do it by the way, Cloudsync. EC2 really recommends that all the distributions as the default user of Fedora has Fedora. So this would be something to configure just for EC2 images, which would be, I don't know, quite cool. Yeah, and the package that's my, kernel arguments can be different. I think that for CentOS 3, we are disabling some drivers because they are causing some issues on some instant types. So the differences aren't huge, but I think that making like images pair, you know, suit or tailor to the target environment, it makes sense. Oh, sometimes you need to take care also about like bootloaders a bit. Yeah, hope that answers the question. No, yeah, all right. So the question was whether we always do QCOW 2. No, it depends. For example, for AWS, we will build a raw image because they accept only raw images. For OpenStack, it's QCOW. And for Azure, they need stream optimized VHD images. So we will provide that. And of course, then for the nonplots, you know, ISOs are produced as ISOs and yeah, that's it. Yeah. Neil? Yeah. So the question was why we don't have a customization that would simply allow you to alter the Sudoverse drop in to allow a password as configuration. Yeah, that's a great suggestion. And actually a funny part story when we introduced the Etsy customization. Yeah, I got a report from a colleague that it doesn't work. It's grew up his environment and yeah, he forgot like something in the file, white space. So I think that's a great suggestion. And our team is trying to actively seek for input so we can integrate more stuff that would help. And this is a great input. Thank you. Indeed. Next question, there was, yeah. Well, redhead pays me. And we use only RPMs, but if a community member comes and does some stuff, for example, the guy next to you did some work for Arch Linux, then we are not opposed. We want to build everything. Especially Arch. No. David, sorry, you mean, okay, we are out of time. I can answer this after the talk. Okay, good. Good, okay, we are done. Thank you. Very good. All right, hello, everyone. My name is Davide. I'm a Production Engineer at Meta on the Linux User Space Team. This talk is about what we're doing with ELN and how we're using tools to help us use our tools. ELN and how we're using to test upgrades on the production fleet at Meta. The agenda for today, we'll start with a quick introduction about what ELN is and how it's made. We'll talk about what we're specifically doing with it at Meta, what we got out of it, and how you cannot get involved and maybe get some value out of it as well. So without further ado, ELN stands kind of our enterprise Linux next. There is an explanation of the puns in the name on the website that I will not try to repeat. It is a continuously variable of raw height using the centos, macros, and toolchain. So the idea is that every day we take the raw height as it is today. We rebuild it using the same macros that would be used as if we were cutting a new centos train train today. And we put the compose out. So you can take it and use it. And you get a preview of what centos train would look like if we were starting today. It is part of the development process for stream and for rel. And it is used for this ongoing process. That's the link to the documentation. I'll add the slides on the website later so you don't need to worry about links. So here's how the sausage is made, more or less. Development work happens in raw height all the time. Raw height is continuously rebuilt into ELN. ELN is used to test how these changes would look like in the future centos train release at some point next month-ish, I believe. The first branching for stream will occur. And at that point, we will have a new set of branches and a new set of repos and everything. And that will be stream 10. And then stream 10 will be stabilized. And then eventually, we'll have rel 10 and so on. By the way, if you're wondering about the colors, this slide comes from an older presentation. I gave the orange boxes, because those are all things you can actually contribute to and work on if you would like. You can't contribute to rel unless you actually work at rel.at, which I do not. So how this actually works is that the composers are made using something called the ODCS, which is the on-demand compose service, which is the thing that is able to make distribution composes and put them online so people can consume them. The package itself is defined in the content resolver, which was an offshoot of the phenomenonization project that is still developed. If you go on tiny distro builders, you can see the actual package set and play with it and slice and dice it. That's a picture I took of yesterday that gives you an idea. You can also see kind of the absent flows in this. As we get closer to the branch point, you can notice this is getting lower, because I guess folks are realizing they don't really want to maintain stuff long-term for 10 years, so they're chopping it out. Which seems reasonable to me. Now, this is just for ELN itself, which is what will end up in CentroStream NRL. Most people, I would say, do not run just that. They need additional packages that they generally get from Apple. To test those, we came up with something called the ELN extras. The idea behind the ELN extras is an additional set of packages that are composed together with ELN, but are not going to be part of ELN itself. It's still kind of influx how we're going to do this, but the general idea is that we will use this set of packages to bootstrap Apple 10 so we can get a head start there and try to make life a bit easier, both for users and for packages. This is also defined in Contra Resolver. You can look it up there. You can actually contribute your definitions here, so if you maintain packages in Apple and would like to get started or have them ready for Apple 10, you can put up a PR against that GitHub project. There's a little YAML file you have to fill in, and your packages will end up getting continuously rebuilt in ELN. Now, of course, this means you sign up for maintaining them, so if they break, you will need to fix them. But you'll need to fix them eventually anyway. And again, that's the text set. This work is coordinated by the ELN SIG, which is a Fedora SIG. We hang out in Matrix and in IRC. While ELN is primarily a thing that is happening on Fedora, I read that infrastructure uses a lot of internal components. Work on the project is by no means restricted to folks that work at Red Hat. You're welcome to join and hang out there. There's regular meetings on Friday, where it's discussed. These meetings are also good to hang out if you're curious on what's going to happen in Strain 10 down the road. I'll talk about this later, but this has been very useful, for example, to get ideas, features that might be coming down the road that you'll want to know about if you actually run this in production. And here's a link on details about the SIG itself. So much for the intro. Let's talk about what we're doing with this in Meta. Before I do that, a quick primer on the infrared Meta. As you might have heard, Meta has a lot of machines. We have millions of servers. These are all physical machines running in data centers across the world. They all run CentOS Stream right now. We always run CentOS. We started with CentOS Linux. Well, I started in 2012, and we were running CentOS Linux 5 back then. We went through several major release transitions. So we went from 5 to 6, 6 to 7. Linux 7 to Strain 8. We just about finished in now Strain 8 to Strain 9. I was very happy to discover last month that we got rid, finally, of the last CentOS Linux 7 host. So we don't have those anymore in production. So that's nice. So I could put 2022 on that line. I expect we'll probably have a long tail of 8 stuff at the end of the year, but we should be mostly done with it by the end of the year. Right now, we're at 86% on 9. This is some containers on 8. Actually, 8 is the majority on containers, but containers are comparatively easier, I would say. And for contests, whenever we do this migration, we already provision the whole fleet. We don't do in-place updates for production systems. So this is wiping the machinery, installing, which is fairly expensive, as you might imagine. Now, our customers don't love these upgrades, because they're kind of expensive, because you're wiping their machines. They have to spend time qualifying the distribution. Usually, the way this has worked in the past is that we get the new CentOS 3 release drops, we start working on it, we put it out, we will play with it, we find some major issue that delays everything by six months, because that's how it works. Like, it's absolutely normal. When you have a large deployment, it's pretty common to end up having to deal with these things. With 9 in particular, the SHA-1 deprecation ended up being a fairly sizable annoyance, because we had a lot of packages that predated that, that had to be rebuilt. We also had a lot of packages coming from external vendors that, shall we say, questionable packaging practices that required a lot of work. So in general, we had been looking at ways to make this process better and more streamlined. When we did 9, I actually started working on these, started working from the very first beta that came out, and that helped quite a bit, because we could get a head start, since the, basically, since branching time. But you still get kind of surprises, because while if you follow Fedora, you kind of know what will probably end up, because you know where it's branching from, it's hard to tell until it drops. The way we do this better is if we could start this earlier, and that's kind of where ELN comes in for us. The idea with ELN is using it so that we can streamline bring up of major rest releases. Turn this from a thing that we have to do at qualification time when a new major release drops to a thing that we do all the time. So we can continuously validate what's gonna come down the pipe, find issues that they come up. When something comes up, we can either figure out internally if it's we that we fucked up and fix it, or if it's something that we have to discuss with the community, because maybe there's a better solution that can help everyone. It also allows us to identify policy changes early on. Things that will come in the distribution that might impact us, since there have been deprecated, new things that are coming up. And because this is a continuous effort, and because we started long before, it's a much more pleasant experience for customers, because instead of having to deal with, oh, all my packages don't install anymore, and I have a month to fix it, now all my packages don't install anymore, but this isn't a production system. Maybe I have a year to fix it. It's much nicer. People are a lot happier when you tell them they have a year to fix it, and they can sort of sort it out to their own leisure. And long at home we like to have an actual CI platform, where we continuously roll out ELN on production systems for some value of production, so that every customer with different workloads can get a sense of what's gonna come in down the pipe and do this validation at their own pace. Now, to do this, we started actually bringing up ELN in our infrastructure, and bringing up ELN is about the same effort as bringing up a new center stream release. So this was broken down into getting the repos, hooking them up into our update pipeline, adding them to our configuration system, which is Chef, actually provisioning and building provisioning, medium provisioning machine, and then deploying it and qualifying. And pretty much every step here involved fixing bugs, mostly on our end. So starting from repos, the repos come from all DCS, as I mentioned. There's the daily-ish snapshot or weekly-ish, depending on when it's got the shows up there. We didn't particularly want to scrape that with WGAD-R, because that seemed kind of bad. So we worked with Infra, folks at Fedora, to get an Arsync endpoint exposed from there, so we could mirror it using Arsync. And now we have these exposed on our public mirror, so if you want to access that and you don't want to hammer the Fedora servers, you can use the mirror on the face with the net. I believe we see that daily-ish, so it should be reasonably up to date. But we don't consume that directly internally. We actually snapshot it via another process that I talked about in a minute called runningOS updates. Now, this is only for the base repos, but of course there's also a Pell. So for a Pell, we use the ELN extras that I mentioned before. So we started populating two workloads in ELN extras, one for packages that we maintain as part of the CentOS Hyperscale SIG, which is where we do most of our upstream work within the CentOS project. And another workload for packages that are really specific to Meta that we felt it would be easy to have all in one place to keep track of. You can look this up. I put the screenshot here, so you can see roughly the volume. The giant jump was when I started testing development systems, which have a lot of random packages on them. So I ended up adding a few dozens there. Oh, that's what this looks like for. And these are all packages that we'll be maintaining then in FL-10, or at least assisting to maintain, because we're not the primary maintainers on all of these, obviously. Now, once we have the repos, we have to hook them up into our base pipeline. I won't talk about rolling-glass updates in detail here. I talked about this in other talks, and it's not terribly interesting, because it's fairly internal to Meta. But the idea is that for CentOS Trim, we snap short the production repos of stream every two weeks. We roll them out on the fleet across two weeks. We do this via Chef. And effectively, Chef will update the DNF.com from the machine around DNF upgrade. And it just works, more or less. When it doesn't work, people get tickets and fix it. In the case of ELN, this process is much easier to some extent, because we just made a really strange called ELN, which is going to be continuously updated. We snap short it every day initially, because that made it easy to eat very quickly and later every week. We don't really care about in-place updates for ELN, because what we really care about testing is the provisioning and the initial bring up. We don't particularly care about testing updates from one ELN from yesterday to ELN from today. So we just stopped out of the test for that and automatically promote all the snapshots. And this basically works. The only caveat here is that ELN composites can be broken, because we publish all composites regardless of state. So we need to filter for only composites that are tagged as finished. Otherwise, you end up trying to roll out something that is missing three quarters of the distribution. Also, ELN extras workflows can fail. They can fail, for example, if you add a package that doesn't build. When the workflow failed, the entire workflow is excised from the compose, which is obviously bad. So right now, we just keep an eye on it, but I need to actually write monitoring. So we do the same thing and validate them before pulling them in. Okay, now we have very close. We need to do the config management side. The config management side is fairly straightforward with one caveat. ELN is odd in that it really is Fedora. It identifies itself as FedoraOI. If you cat, et cetera, what's really success FedoraOI. The only way you can tell it to ELN is because it has variants set to ELN. However, everything else about it is sent us and we really want to validate it as if it were sent us. So what we ended up doing was something horrible, which was adding logic to detect ELN based on the variant ID and then monkey patching it so that it actually looked like sent us. So in our Chef code, we have methods that would like, if not sent us and those we return through on ELN. Do not do this. This is a terrible idea. And like, we did this because it worked for what we were trying to do, but not something I would recommend in general. After that, it was just the usual loop of like testing it on a machine iterating, finding there's all these packages missing and adding it to ELN extras, or all this logic is hard coding sent nine, but we can actually just flip the convert the gate and make it hard code eight because it's just a negative thing, things like that. Our open source code books are there. There isn't much that's ELN specific in there, but I put the link in case you're interested. If you use Puppet or Ansible or anything else, I expect it would be roughly similar. Frankly, I expected this to be a major pain and it turned out to be very straightforward. Okay, now we have conflict management that we want to do provisioning. Provisioning and meta is complicated. And to size that the issue, and because there's a circular dependency that you need Chef to have provisioning working and vice versa because the provisioning image is built with Chef, but you need machines to be able to test that Chef is working. We did another horrible thing which was converting in place existing CentOS M9 host to ELN with ENF Easter sync. To my surprise, this actually works. Not only it works, but it produces systems that still boot. Initially I expected to do this and then throw the machines away, but they still booted afterwards with minor, minor caveat. And I would not do this in production. It is a terrible idea, but it works very well for development because I was able to take development systems, convert them in place, iterate quickly on the Chef stuff, and then later when the Chef stuff was sorted out, get the provisioning images going, and then we could provision machines from scratch. The provisioning system uses an internal thing that is in particular useful outside of meta, but basically they're just images that get dropped onto machines, as you might expect. The images are continuously delivered and tested. So the provisioning system will build the image, try to install a bunch of machines with it and then tag it if it's good or not good. Right now we're at the point where we're effectively in feature parity with nine. We actually were able to refector quite a bit of this logic to make it saner and better support future distributions. After that is just a matter of testing. So we started using, we have a small set of machines we call kernel test that we use for doing kernel and hardware enablement work, which is very handy for these things because it doesn't run anything. So you just have to get the base OS working. So we got that going and that took care of most of the basic Chef stuff. Then we switched to doing the development servers. The development servers are great because nobody cares if your own development server is broken. There's no services running on it except your editor. But they have a ton of packages on it because they have to support all possible work cases from building the Android operating system to doing God knows what. So these are very useful finding packages that were missing. We didn't need it to add to Elenxtras. We didn't quite know. Also, we were able to work with the folks that maintain these systems to discuss, oh, these things you're doing here, maybe you should rethink it because it will probably break in the future. So this is kind of where we're at now. Right now, I'm working to expanding this to the container platform systems, which is the one that I was actually into the entire thing since the start because that's where most of production actually runs. And those will give us a good measure of whether this will actually work long term and also the container system has extensive unit tests that should be able to tell us when things break very quickly. So let's talk about what we got out of this. Took about four months to go from nothing to having provisioning working. This isn't four months of head down work. This is four months of spending time on this but also doing other work. I suspect if you were speed running these, you could probably do it in half time. We now have some dozens of systems running on Elen. I can't say exact numbers for reasons, but it's not a huge amount. I expect this will expand in probably low thousands at some point. I don't think we'll ever have a large deployment of this because it doesn't really make sense. And because these machines will effectively not be doing useful production work. But we have started conversational wider deployments because at this stage what I would like to see is different product owners at Facebook to start trying this out and evaluating, so say the database folks can use this to help the qualification for the databases in the future and so on and so forth. We've also began preparing for sensor in 10 because the work we were able to do with Elen made us aware of changes that might be coming in 10 that we should start thinking about. We are also able to start testing things like the NF5 because Elen is built from Fedora, it has the NF5 so we can actually install it and play with it and test it and start integrating it. Changes that have been deployed and discussed as part of Elen can also be integrated and discussed internally. There's discussions ongoing if you look at the meeting meetings for previous Elen meetings. There were discussions about for example dropping 32 bit packages. That's something that I was very happy to know I had time because I can start talking to the people that use them. And then of course we found bugs and we fixed bugs and I will give you a few examples.  So you might know that RPM for ATN ships with Sequoia which swapped the entire OpenGPG implementation that we want that actually works. Notably these validates packages and make sure the packages are actually compliant. It turns out our internal signing service was using a Golang library and the Golang library was generating invalid signatures by design. Literally if you read through the comments it says we know this is broken, we don't care. So that wasn't ideal. Luckily we found out that the Protomail folks at a fork of this library that was actually compliant. So we swapped it in. This fixed our signing service but of course we had all of the existing packages that now had to be fixed. So we had to wait for a few mass rebuild cycles. We still have a long tail of this because not everything is auto rebuild but it's at the point that it's manageable at least. If you have your own signing implementation I highly recommend making sure this is not happening. Also maybe don't use Golang. Now this isn't all the fun we had with Sequoia. After we had the packages we installed a system for scratch and which our key wasn't importing. Now you might remember I mentioned in stream nine the policy disallows sha1. Turns out RPM never validated the key at all. So when you imported the signing key on nine it imported just fine even if it was signed with sha1. Probably even if it was signed with MD5 I suspect. So this is a problem. We turns out the key was not actually valid. We were using a very old, well it was finally actually wasn't an old key. It was a key that was kept updated but they kept signing with sha1 for reasons. So we fixed that and then it was fine. There's actually a very handy tool that I got packaged into a Pell for this. That was already in Fedora that's part of Sequoia and I branched for a Pell because as you hear in Linter you can feed it a key and it will tell you in human readable form what is wrong with it. It also has a fixed option where if you have the private key for the key and it's an actual private key not an HSM or something you can just fix it and spit out a fixed key. That can be handy if you're your own key. If you have an HSM you will need to work with your security folks on how to fix this. You may want to talk to your vendors about this. I audited hours. We, I would say at least 50% of repos that we were consuming were invalid. And most of these folks when I talked to them they had no idea what I was talking about and it took quite a bit of effort to make people aware that this was actually an issue and they should do something about it. Also if you're re-signing your key make sure you don't actually invalidate your key because otherwise upgrades will be very painful for users. Which actually happened with at least one vendor from what I can tell. Okay, enough about RPM. The other fun thing was Utilinus. So ROI ships with the very latest Utilinus 239 which is a lot of fun features. It also had a couple of regressions that we managed to hit. One was easy, they changed the option parsing for NS Center. NS Center has some bespoke logic for option parsing. That changed lightly where you had to pass equal something instead of just doing option value. Okay, we fixed it. That wasn't a big deal, I would say. That was also easy to work around. The more fun one was the mount API. There's the blog post that explains the mount API. We discovered this when I started provisioning machines that they never came up and I would go on console and see a slew of error coming from system D. And then when I finally managed to find where the console password is, I would get in and see that the root file system was read only. And if you passed RW on boot, it would find but it would, for the like of me, I couldn't get it to remount and if you try to remount manually, you'll get inbound back. Turns out there is a bug in Utilinus that if you're running an old kernel, it doesn't quite detect that the mount API is fucked so it should use the old one. Finally enough, I was talking to Adan the other day and this was actually independently discovered in Rohite thanks to OpenQA. But we don't run OpenQA for ELN so on ELN this lived through. This is fixed in Utilinus by the way. I think I put up the RF2 back ported in Fedora. I don't know if that got merged but I'm sure it will get sorted out quickly there anyway. But this is a good example of something that we were able to catch quickly and get sorted out in a way that is hopefully beneficial for everyone. There's also a couple of other things I wanted to mention. By the nature of how ELN is built, it is like 98% like CentOS but not exactly like CentOS.  In this case, we found that ODD was installed using the FedoraRollSet and not the RallyRollSet because there just wasn't any logic in the package for this. I guess when it was branched for nine, the maintainer probably branched it and then just updated the CentOS side of it and never back ported the changes. So that was easy. Put a PR to the conditional. We only noticed it because in Chef we disable ODD and the way you disable it is different depending on whether it's the Fedora or the CentOS RollSet. The more interesting one was SystemD. So you might know the SystemD SD thing called presets that defines what services will start up on boot. On first boot when you install the package. So it will tell you when you're installing this package should the service be enabled or disabled by default. The presets from Fedora and the presets from CentOS are very different because the use cases for the distributions are different. ELN uses the Fedora preset. So you might wanna keep an eye on this if you deploy this. The way we found this out is because Fedora is explicitly disabled systemD networkD, which we want to explicitly enable instead and we never actually did that. We just, because there was no preset for it, it would get enabled by default in our case. So that's something that caught up by surprise. I don't think there's, we've discussed this with the SIG, there's not quite clarity right now on how this is gonna be handled on the CentOS claim side after branching. I expect this will probably keep deviating because that's not really a good way to manage them. But that's something else you should probably be aware of. All right, so what can you do with this? Well as I said, CentOS will branch in soon. There is a thread on the belt that I would encourage you to read that talks about the plan, how the branching is gonna happen, when it's gonna happen, how this will work out. You also have about two weeks to get a system by changing Fedora if you would like and getting a system by changing Fedora now is a higher chance of it maybe ending up in CentOS stream. Definitely a higher chance of having to do less paperwork than later because once this branch is a lot harder to get changes into stream for obvious reasons. But also if you maintain anything that runs on stream or on rel and you would like to get a head start, this is a good time to start playing with ELM and trying it out. It's also a good time because once stream 10 is fully branched, ELM will transition to 11. And we'll be able to start doing this all over again. As I said, you can join the SIG, the SIG and Gausson matrix and on IRC. The channels are bridged. If you maintain packages in a Pell, you can contribute them to extras. That's the link to the ELM website in general that has information and details about the project. The documentation itself is on GitHub, as is most of the code that controls the build itself. And the Fedora minimization work and counter-resolver is what actually controls the set of packages that you can find there. And I would be happy to answer any questions. Yes? Yes, that would be very helpful. Oh, and the question was, we found this out because we didn't run OpenQlN ELM and whether that would be helpful. Yes, it would be very helpful. So I think in general testing composites would be useful. Which set of tests I think is something we would need to, yeah, we would need to sit down and figure out. I don't know if doing the same test that we do for ROI is useful. I expect a lot would fail just because it's a very different system. So maybe looking at what kind of tests are done as part of qualifying, stream, and rel on the other side of the house and seeing if he makes sense to pour those to OpenQa, maybe, that could be interesting to look at. Any other questions? Yes? Oh, yes, I meant to add this but I didn't know where to slot it in. Yes, there is support. There are shows for ELM. There is also support for Packet to build against the ELM. Yeah, so what we actually do is for our open source projects we use Packet to get continuous builds and signals so that when folks internally land changes they won't break the open source builds and we build this in copper for all the releases including ELM for packages to have help. That works really well. Neil. So it depends. There are ISOs. How do you install ELM with the question? So there are ISOs that you could theoretically use. I have never tried them. What I would do is the DNF install route process that you use normally to build images or you can use your favorite image builder. Then who is there, landed support for building ELM images in MKOSI the other week. So that's an option. I believe they should work with any other image builder. It should work with you, we probably do. Yeah. But yeah, in general I think this will depend a lot on what you want to do with it. Any other questions? So that was the question I actually had and that's where the discussion started. The question is whether we could fix the preset issue by providing a Fedora ELM release package with the REL preset. The problem is that we don't know what the REL presets will be because you can't just blank use the ones from nine because it's different enough. And my understanding is that the way these presets have developed as part of stream is that they start from the right ones and then over time they get more into the new ones. Yeah, I'll go to see if you have any questions. Yep. I don't think that the quality of the new ones is just start with the ones that are nine and then they get a job coming over. I would be in favor of having something because it would make my life easier. It's definitely, I think it's definitely something we can revisit. This would be a good argument to discuss in one of the next sick meetings. Yeah. I think that we have a proposal with Martin where he said we can look at what can't be changed. But perhaps you could have somebody that or a board that every time that we set change in the Fedora or I expected to open an issue on the ELM package so that people actually take a look at the changes and decide what to do. I think that's a great idea. Yeah, we, and to repeat for the stream, the proposal was that we have a process where whenever there's changes to the package in Roight an issue is filed to see if the corresponding changes meaningful for ELM and for the future stream as well. I think that's a great idea and it's probably something we should look at. Yes, Neil says that's probably meaningful only if the package is fourth. I don't think so, because you might have packages where you want a service to be enabled in Fedora but not in stream because it's not supported because it's not part of the default set, you know. You had a question back there? So, I haven't seen breakage from this just yet. I expect I will at some point. I knew about this change because it came up in the last meeting and to repeat for the stream there's a change in ELM where it will use X86 64 V3 as the baseline for X86 and this will cause breakage because anything other than as well I believe, don't quote me on that, we'll stop working. Anything, any hardware that is older than a certain GPU generation which is probably as well we'll stop working. So, we haven't seen breakage from this in production yet because the systems I was testing this on were all fairly new. I did start inventoring stuff. I am, I don't have hard data on this but I am fairly sure we would see some breakage from this and this is something we will need to deal with. Yeah, there's, I think there's a long enough window of between now and when stream 10 and real time would be released that I expected it should be less painful than it looks like now, hopefully. Yes. It might be changing. That should be. It should be changing sooner or later. Certainly it will change in ELM. I hope it will also change in Elmite. Yeah. But I'm pretty sure it will change in ELM and also it will change that even if you by default until you have the calendar two, there wasn't any support or there was no way to do it with the time generator. But now it's the manager. So, right now, if you don't use KVM, you can, or you cannot do it for KVM. You can actually run ELM because it supports ELM. Yep. And to summarize for the stream, one area that's impacted by these is VMs because KVM right now, by default, uses Kimu uses an unsupported asset that's older than B3 but that is getting addressed and should get backported to our right as well, hopefully. Yeah, but I mean, we should have passed for a lot of things. But yes, I don't disagree there. I haven't done much testing on the RecreationStack so I don't have signal on my end on this. I was going to talk to the Recreation folks on my side to see if they could put this up and whatever they do. Sure, we'll do, thank you. Anything else? Going once, going twice? Alrighty, thank you very much.


Your progressive web app needs to load instantly and reliably, never showing the downer saw, even under uncertain network conditions. Let's take a look at how service workers allow you to do that. Hello, I'm Mariko. I want to start with a user scenario for a typical app. Let's say I'm sitting at a park and I'm humming to myself and suddenly thinks that's a great lift, I could totally use that in a song. So I open up my voice memo app to record myself. Unfortunately, my phone doesn't work in this park and my web-based app fails to load. Users have learned not to expect the same reliability from the web platform as they do others. Users expect it to work instantly, regardless of where they are, what they're doing or what the local connectivity is like. And unfortunately, this isn't just about no connection. Slow and intermittent connection affect users just as much. Because of the way web pages are typically constructed, a slow connection can increase the time that a page takes to load. On top of that, mobile phones sometimes are not online. Even when your phone says you have four bars, it might not have a live connection. In order to become reliable, we need instant loading and offline to be part of the web platform experience. Because remember, as we said in previous video, after three seconds, 53% of users abandon sites. Users are not willing to wait. Globally, over 60% of mobile connections use 2G. We need to do something. We need to be able to build reliable sites and apps for users when connectivity doesn't exist or when it's slow. It's clear that we need to stop relying on the network. The key to building instant loading and offline-capable web app is a relatively new technology called service walkers. In an earlier video, we talked about how a service walker is a programmable proxy sitting between the network and the browser, but it's more than that. When a page registered a service walker, it has a set of event handler to respond to things like network requests, push messages, and updates to the service walker. Because it's event-based, it doesn't have to be listed in memory unless it's handling one of those events. I'll say it again because I think it's important. It's not consuming any system resources until it's been walking up to handle one of those events. In fact, even if the browser is closed, the OS can wake up a service walker. We are no longer limited to the model where web app must be loaded in an active tab to be useful. This is important because it introduces an app-like lifecycle for websites and web apps. When a service walker is first registered, it fires an install event, which we can then use to prefetch the resources we need for our app and cache them locally. This is like install as you go. In effect, we are learning a fully scriptable install much like native app. No bulky zip files. We have file level control over what is fetched, how it's cached, and how it's updated. Once the service walker has been activated, we now have a full control over how the system serve your app's resources. We can inspect network requests, rewrite them, get a response from the network, or respond with something from the cache. We can even synthesize our own response. All of this is completely transparent to the user. In fact, well-constructed service walker can act as a very smart caching system and be transparent to the rest of your app. It is just a progressive enhancement of the network and the cache, choosing smarter ways to respond to the requests. It gives us the ability to make our app reliable even when the network isn't because we have full control over the response. Service walkers are progressive enhancement. It's important to understand that the first time you load the page, it's still going to load from the network. The initial page has already been retrieved from the network and the page resources will be loaded normally. The service walker will come into play only on the second request, though there is one caveat to that. I'll cover that in a moment. Let's dive in and take a look at lifecycle of a service walker. When the page is first loaded, it registers a service walker for a specific scope. For now, think of the scope as the set of page that service walker will handle. An install event is then fired and the code in your install handler is executed. Then the service walker goes to idle, essentially sitting there quietly. When the page is reloaded or you hit another page covered by the service walker, the service walker is activated, ready to handle all of its events. At the same time, the browser also makes an asynchronous request to the server to see if the service walker has changed since it was last retrieved. Once the service walker has handled all of the events, it goes to idle and eventually it's terminated, ready to spin up again when any new events come in. Let's keep going with our previous example and see how a service walker gets updated. Since we've already been to the page, the service walker is already installed and immediately activated. Then, just like before, the browser checks with the server to see if the service walker has changed and this time it has. The new service walker is downloaded and installed, but goes to idle and the original service walker is activated and still handling requests. In fact, our original service walker will remain active until all the clients using it has been closed or until it used skip waiting and clients.claim. We'll dive into those more in a second. This is important to remember as you are working on your own service walkers. This isn't the problem for users, but as a developer, you may find yourself with multiple service walkers installed and waiting because the original one hasn't been terminated yet. Enough theory. Let's take a look at how to implement our own service walker. The first step is registering a service walker in the web page.  First, we check if the service walker API is available in the browser. Then, if it is, we call navigator.servicewalker.register to register the service walker with a reference to the location of the service walker JavaScript file. Next, we need to handle the install event, which is fired after the service walker JavaScript file has been downloaded and parsed. Let's start with something basic. Remember, earlier I mentioned that when a service walker is installed the first time, it doesn't take immediate control of the page. You have to wait for the page to be reloaded. Well, there is a way around that. Calling self.skipwaiting tells the service walker you wanted to skip waiting and go right to work. We'll use the install event to pre-fit the resources we need and cache them in the local cache, ensuring that the files can be loaded instantly and reliably. In effect, this install event handler is your opportunity to have a fully scriptable install process, just like native app. You have absolute control over what resources are prefetched and how they are cached. First, let's open up a cache object referenced by cache name. Then, we'll use cache.atall to retrieve the resources we've listed in file to cache and store them in a browser cache. In a little implementation, you should make sure that you take care of any exceptions here. If one of the files were to fail while downloading, the entire event fails and the service walker wouldn't be activated. And finally, once everything has been added, we want to call self.skipwaiting so that our service walker takes control immediately. There's one other thing I want to point out. We wrap that whole code block in event.waituntil. This makes sure that the service walker isn't terminated or go to idle while we're waiting for the embedded promise to complete. Once everything has been cached and our service walker has been activated, it's a good idea to clean up the cache. For example, remove any outdated resources from the cache. There are plenty of ways to do this, but in this case, we iterate over the list of keys from the cache, then delete any cache that don't match the current cache name. And there's one more thing we need to do. By calling self.clients.claim, we tell the browser that we want this service walker to take control of all the clients. That means we don't have to wait until other service walkers are terminated. We are ready to go immediately. But at this point, if you were to go flying, then you'd still get that dreaded offline dinosaur. But why? We haven't added a fetch event handler yet, which allows us to respond to the network requests that are made by our page. Let's assume that all of the required resources for our app are already in the cache. But if it's not, we'll just go out to the network to fetch it. When a network request is made from our webpage, our service walker wakes up and the fetch event gets triggered. Next, we check to see if the requested resources is in the cache. If it is, we return the version from the cache. Otherwise, we request it from the network and return that. We now have a reliable experience that loads almost instantly. Our app will work whether we are online or offline and it will load extremely quickly because we've eliminated any potential network latency. Our last step is to let the page know that the service walker is now registered and ready to go. This is helpful if you want to indicate to the user that the page is offline capable. Earlier, I mentioned the scopes. One of the cool thing about service walker is that you don't need to register it on every page you want to use it. You only need to register it once and any page within the scope will be handled by the service walker. The easiest way to define a scope is simply based on where the service walker is served from. I recommend that you serve your service walker from the server loot. That means every resource requested will be handled by the service walker. If you put it in your script directly, only resources served from there will be handled by the service walker. In this case, unless the page registering the service walker is also in script directly, it won't be handled by the service walker. Putting the service walker in your server loot gives you the first chance at responding to any network requests and helps to ensure your app is reliable and loads almost instantly. The magic a service walker provide is that you are in control. You get to decide what gets cached, how it's cached and how it should be returned to the user. I've shown you the cache first with network fallback strategy. It provides a fast, reliable experience for users. There are several different caching strategies and depending on your app scenario, you might want to use different one or even multiple strategies for different kinds of requests. Let's take a look at some of the different strategies. The cache first with network fallback strategy we laid out, it's pretty straightforward. Always go to the cache first and if the request matches the cache entry, response with that. Otherwise try to fetch the resources from the network. This option is good for resources that don't change often. It will give you a very fast response since response came out of the cache directly but it doesn't update. On the negative side, if an item is not in the cache, it is not added after it's retrieved from the network. Network first and falling back to cache means you give online users most up to date content but offline users get an older cached version. If the network request succeeds, you'll most likely want to update the cache entry. However, this method has flaws. If the user has an intermittent or slow connection, they'll have to wait for the network to fail before they get the content that is already on their device. This can take an extremely long time and it's a frustrating user experience. Cache first then network requires the page to make two requests, one to the cache and one to the network. The idea is to show the cached data first, then update the page when or if the network data arrives. Sometimes you can just replace the current data when new data arrives, things like game leaderboard. But that can be disruptive with larger piece of content. Don't make something that user may be leading or interacting with disappear. With the generic fallback strategy, a request is made first to the cache, then the network, and if both of them fail, a second request is made to the cache for a generic page. This is ideal for secondly imagery, such as avatars failed post-request and unavailable while offline page. One of my favorite use for this is a newspaper that gives you a close word puzzle to play while you're offline. And of course, there are more. Cache only, network only, service worker templating, just to name a few. But the core point here is that when it comes to caching strategies, you are in complete control over how to respond to requests. You can manage the network and the cache so that you can build a consistently reliable and performant solution. Because you have so much control over the network, it can seem quite daunting to get started and to debug experience. Luckily, there are a number of tools that can help you. The Chrome DevTools have added new set of tools to help you work with service workers. In the DevTools, you will find application tab. In application tab, you can inspect the service worker and register it, false it to update, simulate offline behavior, and more. We'll talk more about these in the tooling video. And of course, Firefox, Opera, and other browsers have similar tools for debugging service workers. DevTools let you inspect not only application strollage, like IndexedDB, but also inspect cache strollage. In the past, we used to recommend people use incognito windows when developing service workers because it made it easy to start fresh when you need to. But the new clear strollage pane simplifies things tremendously. I use this one a lot. Next, we have a tool that can speed up your service worker development. Walkbox is a collection of libraries to help you develop PWA. If you remember the code for a cache first service worker from earlier, it was a bit cumbersome to write. With Walkbox SW, this turns into simple declarations for caching strategies on multiple resources. But you are still in control of your site. It's just a set of packaged loudings and strategies. So I hope this has given you a good understanding of service workers and how they provide the core of instant and reliable experience for progressive web apps. Thank you.


Hello everybody! In today's video, I'm going to tell you what to do if your phone can't see a memory card. These days, all phones in almost every other gadget have a special slot for a memory card. One day or another, everyone may face the situation when a phone can't recognize a memory card or it displays a message saying the card is out of order, damaged or not supported, to say nothing about other possible issues. From today's video, you will learn why such things happen and how they can be fixed in an Android device. In our channel and blog you will find solutions to any problem from installing an operating system or configuring it to fix impossible bugs and errors or optimizing mobile gadgets. Our specialists will answer any questions you ask in your comments under the videos or articles. The first thing to check is the contacts of your memory card. There are cases when those contacts get corroded or covered with dust, and it makes the contact too poor for the phone to recognize the card. Start with taking the card out of its slot and then try inserting it back again. If it doesn't help, clean the card contacts. Use a pencil eraser to rub the contacts very carefully. If you can access the contacts in your phone's memory card slot, clean them too.  For example, contacts inside the phone are damaged or the memory card is out of order. To exclude one of these, try using another card with this phone. If it's the contacts inside the phone, it will not see any card at all, no matter which one you use. If it's the card to blame, no matter what device you put the card into, the faulty card will never be recognized. Another possible cause is using a memory card which is not supported. For each gadget, there is a limitation for capacity of memory cards it can handle. If you see such an error, your phone might be unable to work with cards of a larger capacity. Know every time that a phone or tablet PC fails to recognize a memory card, check the specifications and make sure that your gadget supports the capacity and type of the memory card you are trying to use with it. One more thing to get in the way could be a bad block. To find out if this assumption is true, insert the card into a card reader. If the card is recognized, go on to repair it. Open this PC and select the memory card. Right-click on it and select Properties. In the window that opens, find Attempt Tools and select Checking the disk for errors. When the check is over, take the memory card out of the card reader and put it back into your phone to see if it can be recognized now. If this method doesn't help, the last option is to format the card. If the card reader does recognize it, save all important files to your computer because, as you may know, formatting will destroy all data. A widespread situation when your gadget cannot recognize a memory card looks like this. When you insert the card, the phone shows you the message that the SD card doesn't work, is damaged or not supported. When you tap on the message, the phone will suggest formatting the card and configuring it as portable media or internal memory. However, if you see a message like that, it doesn't actually mean the card is damaged. Especially when you know it worked fine just a day before with your laptop or desktop computer. In such a situation, the actual cause behind the alarming message is either the card having a file system, which is not supported by Android, for example, NTFS, or the card without any file system, which can happen after certain operations with a memory card on a computer. Well, there are a few methods to solve the problem. Method 1. If there is any important data on the memory card, transfer it to the computer using the card reader. And then, use the computer to format the card into FAT32 or XFAT. Otherwise, insert the card into the phone and format it as portable media or internal memory by tapping on the message about unsupported or damaged SD card. Alternatively, you can do it by opening Settings, Storage, SD card, Menu, Format. Method 2. If you don't have any important data on the card, just tap on the message saying that the SD card doesn't work properly or open Settings, Storage, and in the SD card section, tap on the card marked as damaged, tap to configure, and select how you want to format the memory card. If the phone on tablet PC can't format the memory card, try doing it with your computer. Connect the card to the computer with the help of a card reader. Method 3. Right-click on the disk, Format – FAT32. Also, the message about card being damaged or unsupported may appear if such card was used as internal memory on another device, or the current device which was then reset to default settings. Finally, the last option is formatting, but it means losing old data. So remember about it. In our channel, there is a good video about restoring data from a memory card. Just check the link below. If none of the messages in this video helps you and your Android device still can't see the memory card, try the following. If the card is not displayed in the File Explorer when connected to the computer, but it is shown in Disk Management, find your memory card there, right-click on one of the partitions and choose Delete Volume. Then, right-click on it and select New Simple Volume. Then, connect the card to an Android device. If you fail to format the card with your phone or computer, try using a specialized utility SD card formator designed as a desktop tool. It can detect and format a memory card even when it's not displayed in the File Explorer at all. By way of goodbye, here is one more thing to keep in mind. There are fake memory cards. Some people tend to buy them dirt cheap from Chinese online stores. And these are advertised to have a certain capacity which is displayed properly on a computer, though the actual capacity is less than expected. The simple trick is based on the card using special firmware. But when you stick such a card into an Android device, it's merely a question of luck. It may or may not work properly. And that is all for now. Hopefully, this video was useful. Remember to click the Like button and subscribe to our channel. Hit the bell button to receive notifications and never miss new videos. Leave comments to ask questions. Thank you for watching. Good luck.


Balloon flower root, BFR, has been shown to contain bioactive compounds that exhibit anti-inflammatory, proliferative, and antioxidant properties. These compounds are contained within BFR-derived extracellular vesicles, BFR versus, which were found to be effective at reducing inflammation, promoting cell proliferation, and combating oxidative stress. This suggests that BFR versus may be used as a natural treatment option for chronic skin wounds caused by excessive reactive oxygen species, ROS. This article was authored by Mano Kim, Hyjin Jang, and Ju Hyun Park.


Senior Navy and Marine Corps officials conducted an evolution of the human machine team panel as part of the Sea, Air and Space Exposition near Washington, D.C. May 17th. So I think these panels might give people out in the audience ideas of the way we're thinking of things. During the panel, leaders discussed how to efficiently integrate unmanned systems into the team. Right now, by and large, machines add force protection, so I can lose a machine instead of losing a marine. We need it to be reliable. We need it to not interfere with other systems and endanger either our weapons systems or our marines on the battlefield. We want to get to the point where the machines are forced multipliers and not just a change in the task. The Sea, Air and Space Exposition provides a venue for leaders across the Department of Defense to display the most current information and technology relevant to maritime policy. Reporting from Washington, D.C., I'm Sergeant Eric Heenan.