Okay. Hello. So welcome. My name is Lubomir, this is Mohan and we will try and tell you something about what happens with Fedora Composes, how they are done, what issues there are, and what might be possible to do about fixing that. Like in the keynotes you heard a lot about what's the bright future and what are the high-level goals. This is nothing like that. We will basically just tell you what we can do to keep on tracking the way we are right now, just slightly faster so that we eliminate some of the problems. So first we'll sort of try and describe what problems we're facing, why it's bad, then try and describe, say, what's actually the cause of those slowness issues that we are facing, and then we're trying brainstorm some solutions. And this is actually where you come in because we are looking for suggestions. So the main problem is that Fedora Composes are taking way too long. So for Rohite on a good day it's eight and a half hours, but there were cases where it took maybe 18, 20 hours as well. That's not really doable, especially given that there's basically a single person who usually runs those composes. So especially before release, we need to make it possible to test changes in Installer and all the important packages. And if it takes eight hours, that's sort of not really fast iteration. So that's what we would like to fix. And let's actually try and describe what's happening in the Compose. So as you might know, the composes are done with a tool named Pungy, which does a whole lot of work and it tries and does do it in a parallel way so that it's not completely stupidly slow, but still it's not that perfect. So the actual work is split into multiple phases. I'll try not to ruin everything here. So we are not at C, okay, I just hit the screen. It will calm down. So this is basically the overview of what's happening in there and I will go into more detail about each part of that. Okay, next please. So first some of the housekeeping, those are things that are fairly necessary for the Compose, but are relatively quick. I mean, I will have these average times. Those are averages taken from last five row height composes and it's an average. So this sort of housekeeping takes almost 40 minutes. So in the init phase, we start by preparing the Compose files that includes checking out the Git repo, including translations in there. And some housekeeping at the end of the Compose is computing checksums for all the images that are generated and there's actually quite a lot of them. So it's not just the same server installer. We have Netinst images. There are live media. There's a bunch of spins and laps. For all of these, we need to make sure that we have a check. Some people might want to check those at some point. And the test phase at the end is relatively simple. Essentially all it does is we run repo closure on all the repos that we created. Usually in row height, there are problems for actual released versions of Fedora like on GA date. We probably shouldn't have many of those, but to be honest, there still are some of them. And also we run some tests on the images themselves. Like if something claims to be an ISO file, we actually check the headers if it is an ISO file. If it claims to be bootable, we check some flags and magic bits in the file to verify that it can actually be booted at least in some way. Because in the past, we have been bitten by this. The process changed a little bit and suddenly we screwed ISOs in a VM, I think. It worked if you burned it on a physical media, but who does that nowadays? So next slide please. So the first real slow part is the cold package set. And this is essentially about talking to Koji and figuring out what packages were built and should be included in the compose. Historically, we have started with a single Koji tag. You find signed copies on the file system because you can't really get that nicely from the API. And once we have all the signed copies, we created temporary RPM MD file repo on the file system that's used in the following phases. So this just includes really everything that's in the tag. And there's one repo for each architecture. There's no filtering at this point. And as you can see, it takes over an hour. So it's not ideal. Next please. So the first part that's using this repo is called build install for historic reasons, because originally it used to call the script called build install. Nowadays, here we just run Lorax to create install tree. So there's a boot ISO, there's configuration for grab. And on average, it takes maybe 43 minutes, but it varies quite wildly. Like if it, on a good day, it can be done in 10 minutes. On a bad day, it can be two hours. It also, it depends on, for example, how busy Koji is, because all this happens on Koji builders. That's the individual instance, actually. That's the actual individual instance, because if you check the logs from the compose for how long the phase took, due to how it's implemented, like it will only report that it's finished after these two things are finished as well. So it's not really doable from that. So I just looked at, I think it was 90 tasks in Koji from those five composes. And I ran the average over that. And that was the result. They might be in the, sorry, the comment was that the builders in Koji that are used for these tasks are in their own channel. So it shouldn't vary based on the load. So in that case, I have no idea why it's very so much. It can be affected by the overall IOL load on Mount Koji. And it quite possibly can be. The variation is most likely caused by bad things in the Mount Koji, but I never actually had Anaconda run quickly through an image build. So like, there's an Anaconda slump. The other way, if you download all the packages locally and rerun war apps five times, it should have about the same time. No, it'll be consistent if they're all downloaded. Except it doesn't catch anything. So there was a very long comment that Lorax is just slow. And that's why it might take this time. So the next part is called gather. And this is basically when we when Panji decides what packages should go into each part of the compose. Because technically, it's not just one big compose. There are separate parts. There's everything, which as the name suggests, includes pretty much everything. There's server, workstation, cloud. And for each of these, we just include subset of packages that are in the overall tag. This is configured mostly by comps file that each of those variants basically say, I want these comps groups plus all the dependencies. So that's how we decide what goes in there. And once we know what goes in there, we create hard links to every single RPM, which is also in this phase. And overall, this takes over two hours. So there's definitely a room for improvement here as well. Because it probably shouldn't be taking this long. Why are we creating trees for every variant when nothing uses the outside of this process? So question, why are we creating trees for every single variant and architecture combination if there is nothing other than the compose process using them? Well, the answer to that, if there is something in the compose process that's using that, we sort of need the files and the repos there. That's a different question. Why are we doing the entire tree every time that way instead of... So we know what got built since the last time we did a compose, why aren't we just removing the things that aren't there anymore and just having the new ones back? Suggestion, which is what we are exactly looking for. Why are we rerunning this whole process every time and not just including the changes that happened since the last compose? And wouldn't we know things like the build IDs that are the source of this and as build IDs are changed based on that you could just do mercury flows and change all the... Yes, and when we don't need to rerun the whole create repo everything, we could just include the new builds. And that's a great suggestion. The reason why this is not done is that no one has actually implemented it yet. In addition to that, it would still have to handle this case, right? Because you would still have an initial condition or if you mess something up you need to do a full... Yeah, we need to make sure that we can actually run the process as a whole. So the part that would be doing the incremental changes would still be sort of like an add-on to that. We still need to make sure that we can say for GA we need to be able to create the whole thing and run it at the same time but not just collect random pieces from Rohit and say like, hey, this is Fedora 13 now. So I have another suggestion here. I noticed that it does hardlinks in a separate base so it creates all the repos and then runs hardlink over it, essentially. Here we don't yet create the repos that's over here. This is just figuring out what packages will be in that repo and hardlinks them into place. Okay, so it is hardlinking them as it's... So they can make the create repo and then we will run it for the directory then. So we're doing this so that you can run a create repo against the directory, right? Not necessarily. It's not just... Sorry, the question is, are we linking the packages just so that we can run create repo over the directory? And the answer to that is no. We could run create repo against packages in random locations. That's not a problem. We already did that in package set phase. We need to copy the packages into some locations so that we can r-sync them to mirrors so that people can consume that. So when you validate the tree, you can just go and say the clutch and you're done. Like you're pushing to the error master. Yeah, yeah. I mean, for pushing the mirrors, we need to make sure that we have something that is r-syncable. Like we don't want to r-sync random stuff from Mantcoji. I mean, technically it's possible, but I don't want to be the one who has to implement that. So one of the questions was related to why do we generate install trees for things when we don't ship them? So one, we stop doing that for cloud because we don't ship an ISO for cloud. Right. Well, I'm just saying we don't do it anymore for cloud. And the reason that we have done that in the past is because we actually ship an ISO with an embedded repo on it for server, right? So we need a separate, you know, repo for server specifically because we need a repo that will fit on a 4GB DVD image, right? For Workstation, we ship an ISO, but it's a live install, right? So there's a long thread on a Pagger relinch issue that says, do we really need to do this for Workstation? And so there's a long conversation there, and I don't think the answer is no. We just haven't explored everything yet. Also, there is some value in having the repos even if we are not shipping them for creating, say, the Workstation live media in that we know what set of packages was there. Like in Fiori, you could have a package that's not listed in comps get on to the media if it was done from some other repo, not just this subset. It's not like a big deal that's in Fedora, but for other use cases where this process is used, it is sort of like a requirement. We need to make sure that the images consume only the stuff that should be there. How long was two hours? What is the slow part? The slow part here is mostly figuring out the dependencies for the packages because from comps groups, we know what... Do we know the time? It's not really the question of reading it. That's still relatively quick. We read the repo data that we prepared here. The thing is that we actually have to run a transaction for every architecture and variant combination and figure out all the dependencies, what should go in there, apply rules for multi-layer plus a few other magic rules that create it over time because someone needed to fix some problem. Is this effectively then slow because RPM doesn't do something like this? Yes, using Libsoft could possibly speed this up quite significantly. There is some work in progress on implementing a backend that would use Libsoft. Right now, it's basically Python code using LibDnf to parse the repo data and ask for dependencies like what provides this thing. So that's a big reason for the slowdown. Does it similarly? They use PerlBSSol and it's just a pin wrapper around Libsoft itself to process that information and just export it. The comment was that OBS is using Libsoft to solve similar problems and it's very fast. I can actually confirm it from the tests on the new backend that I've been running. It's a lot faster if we just run Libsoft. Technically, we have to run it multiple times to add the multi-layer packages. But once it's moved in to C, it's a lot faster than the Python implementation we have now. Maybe we should move questions to the end. We'll let you guys get there. That's fine. Next slide, please. As I said, this is just creating the repos for the files that are already in place. That's fairly fast. It could be faster because right now it all happens on the machine that's running the Compose, which is usually there's one Compose machine for Rohite, one for releases, so on and so on. And it could run anywhere. We could ship it to Koji builders and run one create repo on every single on a separate builder. It might help us a little bit, but it's not a big pain point. It's just five minutes. We run multiple processes at the same time. I've actually bundled two OS3 parts here. That's creating the OS3 commit itself with the new updated packages and then creating a bootable image that includes the OS3. And this does take a fair bit of time. The thing that we can do here and that will happen fairly soon, hopefully, is to move this and run it in parallel with these other bits and pieces. So to be clear, this is the OS3 and the OS3 installer phase. Yes. So that includes a run-of-the-war app. That includes a run-of-the-war app. In the OS3 installer phase. Quick question. The times you're using are our current times. So this has the S390 stuff in it? I think so. The times are from five Rohite composers and I took five last successful, which is like two weeks ago. Yes, some bits and pieces are missing from that, but... So that's another thing that could be slow, although I don't have a solution there. The comment here is that a slow part here might be that it's over all architectures and some of them are relatively slowish. So the solution here is basically move this and run this part earlier than we do now. The reason why it's not done is that there was a bug somewhere between RPM, OS3 and libDNF and it got confused when it saw a repo with binary and source packages. This should be hopefully resolved now, but we need to make sure that it's tested. So question then. If this is the time for the OS3 creation and then the process for creating an installable image with OS3 embedded in it, what is the time for actually creating an OS3 commit by itself? Does anyone actually know how long that takes? I've never actually made an OS3 commit from our PM ministry, so I don't know. Because I'm wondering how much of that time actually is one thing... The question is how does this time split between creating the OS3 commit itself and creating the image? And my guess would be that the installer creation should be roughly the same as for the regular installer, so about 45 minutes. So that would leave about 20 minutes for the OS3 commit itself. The comment is a lot of that is copying images, sorry, RPMs, not doing actual work. So that seems to be a thing that happens in a bunch of phases, so a question that I've got is why don't we actually fetch the stuff the first time? Since this is being carried through through the same machine, why don't we just hold the cache and pass it forward to the rest of the phases so that we don't keep redownloading every time? The question is why do we download the files over and over again? And the answer to that is that we are basically not. I mean, it's all on the same NFS volume. It's all running in the same data center as far as I know. Okay, yeah. What I'm saying is hold the cache directory and just keep filling it up and that way you don't have to keep redownloading. That skips like half of the process. Like a DNF install transaction is a simple one. It's about three quarters of it is fetching metadata and then downloading the packages to process the transaction. The rest of it is relatively quick. Okay, so the suggestion here, and it's a good one, is to create a cache with the packages locally so that we don't have to actually check that again and again for multiple things. And then the follow-up to that then would be, can we also do like all the Lorex runs we're doing? Can we do it on HEPFS? Oh yeah, that's fast. Yeah. Moving Lorex to TMPFS is also a good suggestion. That one might be slightly problematic because we can't run all the Lorex tasks on a single machine. That needs to be, like for each architecture, it needs to run on that particular architecture. Like you can't create the installer for S390 or something else. Every machine, like theoretically an entire run group, you should just be able to run it in a random app. Why potentially do cross-arch stuff with creating routes and for images? The question was, why can't we create installers for different architecture? And the answer to that is, I have no idea. So take it down as a node and possibly run things in RAM. Yep. This brings us basically to the last part and that's actually the biggest, slowest part and that's creating all the live images and spins and whatnot. There are technically, all that stuff is split into four different phases because there are four different kinds of things that we generate. But overall, it's four or not five. I can't count today. Basically, it's all the same thing. We just run a time for it to finish. Once it finishes, we copy the file, the actual image into the Compose directory and we get done with it. Most of the time here is really just spent sitting and waiting until the task finishes. And overall, it takes about three hours on average. So there might be some possible improvements here but it would have to happen in the actual tools that create those files. This is a live media task. I know runs Lorax yet again. It should not run Lorax because it's consuming the output for the Lorax run we ran before. LMC is Lorax. So it executes Lorax to produce and runs in Anaconda again to install back into the ISO environment to create the live media. That's how that works. So you have at least there each piece of live media probably taking about 30 minutes by itself to factory download because it doesn't understand caching. And then we'll actually install and produce the tree and then take about five minutes to make it. So the comment here is that at least live media creator is running Lorax again. And we might be able to save some time there. Good thing to look into. Thank you. We might want to look at because all these things are in different channels or many of them are in different channels. We might want to create some logging or do something to indicate when we're filling those channels up because we can re-outtake where the system goes. So it might be that we have eight images we're trying to build and only seven builders in there. If we have one we actually save ourselves a lot of time waiting for that one to... Another comment is that we might get some speed ups here by reallocating builders in Koji so that it matches the actual demand. It's kind of not obvious from the actual process like what takes how long. And we don't actually run everything on the same architecture. There are more images for X8664 than for other different architectures. So I think that's it for my part and I'll hand it over to Mohan for his more ambitious ideas on how to fix our problems. Hello. Sorry, I was just taking some notes with all the solutions. Don't mind me tapping on the mobile. But anyway, so you heard of the pain points why it is slow and thanks for all the solutions. Now I want to talk about a couple of solutions that I have in mind and these are the four solutions that I'm thinking of right now. Well, fourth one is not my preferred choice but I'm just putting it out there. First one is like we are running depth solving as part of the compose process. I want to move it away from the compose and run it on demand. Let's take the example of Rahari. Whenever it gets built, it gets stuck into F29 pending tags, gets signed and moved into F29, 29 being raw height right now. So I want it to run depth solved once it gets signed and stored by variant, by arms. So that it just also finds the, puts the location of these all the builds. So whenever it runs, we can eliminate the packet set and as well as gather phase. Well, part of gather phase because gather phase also has some hard linking but packet set and part of gather phase can be removed which essentially saves about two to three hours because it's taking more than three hours right now for both of those phases and they're not run paralleling. So that being said, there are a couple of things we need to fix especially Koji or some service that Koji will call to do that absorbing and punchy changes because obviously we don't need all these gathering phase and gather phases so some changes over there. And that's the first option and my favorite option. And the second one is also my favorite. Basically, tracking all the changes in the images, what goes into the images. So currently the images are generated from looking at the kickstart and kickstart might include some comms and as well as some build-up changes and everything. If you can track all those changes and identify if anything has changed from yesterday to today, composed basically. And we can eliminate the creation of the same image over and over and just like hardling the previous image from previous composes. Essentially, what happens is we have to run everything because everything changes every day. So any image creation related to everything we have to run it. But all the other labs and spins and workstations and even servers sometimes we don't have to run them. So that will save a lot of time and when we looked it's about three hours plus. So I'm guessing it will come down to less than half of it. Anyway, guessing because we don't know how much of the things are changing every day. But definitely spins and labs won't change much. Then to quantify, can you just look at the previous images over here and just see how many are there? Because my guess would be that actually almost always something changes, right? Yes, but not in all the variants and all the arches, right? So like for example, our maybe changing a little bit more than say, ICT 16. Or S390X or PPC 64. The S390X is special. Yes. The S390X is what reduced the average time but not reduced to the maximum time. Because like one day you get locked in and changes, the next day the chronological scene changes and every world is broken. Yes. And it takes eight hours or ten hours. Yes, there is a possibility of that. You repeat the question. Oh, sorry. So if we implement this change, so the question is if we implement the change, I am going to reduce the average time of these phases but not the total time. So I would like to say that any change or any time that we can say is really helpful, especially during the freeze time. That's what is actually driving me to make these changes happen. Because during freeze times we actually get about one or two days to create the RC Compose and sometimes we'll find more bugs and every time we'll be stuck with like postponing the release by a week because we just found a bug. Yeah, just before we go on Nogo meeting. So I would like to implement as much as I can to get that number down. And some of the things we cannot do, basically as I said, asking for a new strength TX mission, it's never going to happen. Or adding more builders probably, but still. Why would that never happen? We're buying a mainframe? Sure. I didn't say buy one, but like why would there not be more builders or things like that? No, no, it's not about that.  We have access to extremely builders. Yes. Just in Boston. Yes. The physical location difference is the problem actually. Yeah. And all of our builders... Sorry. All of our builders and Mount Pudge is located in Phoenix location whereas us three net X is in Boston. So... So that means we're in network? Yes. Yes. Well, it's a slow lead. One response to helping during pre-signed, I don't know if this particular change would help us that much in pre-signed because I feel like when we're doing an REC, we wouldn't want to actually... We want to validate everything. Do everything, a total compose as part of an REC and not a partial compose that's optimized based on whether or not the previous... What? Compose... Because the actual... version would be different, right? Between the two. Compose basically has the... The dates and stuff like that. We would probably want to all come from... Another case there... Another case though where this would be very helpful is with raw hide. So a compose goes along, it fails. Right. Oh, I think that might be DNF, but I'm not sure... This is great. Until I rebuild everything and then you have to wait another eight hours. This is great for generating. Yes. Yes. Well, even in the freeze time as well, definitely the first RC compose going to be the everything. Right. I'm not going because it's going to be the different directory and... We haven't composed... Any RC compose before that. Right. For that release. Right. So first RC compose will have everything, but let's say we found issues. Second one. Right. And probably we add one or two bills. QF found something... They want to add one or two bills into the compose. Right, yeah. Then it will save a lot. Yeah, I would say the first RC... Yeah, first RC, yes. Definitely not. And the last one that we actually release is a full compose, but we can do optimized releases in between there. Yes. And like get on the QA immediately. Yeah. And say yes it works and then pick off the full compose. Yes. So yeah. And in my experience, I have done one RC compose for one release. Every time else, it's like multiple RC composers for one release. Yeah. So out of all the like six releases or seven releases I've worked on. Cool. And definitely, that needs some changes to punchy as well as we need to have some smartness to identify changes in these things that we talked about. And district was probably you guys might have heard about it. And it's kind of implemented, but we're still not using it because there is a small issue over there. But anyway, with district was what we can do is a packet surface where basically in punchy it runs on all tags combined together on one machine. With this, we can split that run multiple like district on different tags on different builders basically calling Koji and then split them again on based upon arch and sending it to different builders based upon their architecture as well. So it will save some time. I'm not sure how much, but definitely packet set being long phase, about takes about two hours. So this will definitely save some time over there. And the idea is basically call the create district task which is part of Koji whenever a new gets, a new build gets type and use that report. So, it definitely needs some changes to punchy and they have to fix that bug in Koji. So, the bug is multi arch? No, it's the debug packages and source packages are going into the system. So that's actually fixed in Koji 116. It's just never been pulled into an oriental structure. Yeah, like I was working with Mike, and a couple of other people on like finalizing the structure for this and that has actually been fixed in the latest Koji release. It's just never been packaged for Fedora. I actually built it in my last week. Oh, you did? Okay, that's awesome. And I'm holding off on it for the rest of the branches because 161 is probably due for 14 long years. Yeah, it is, yeah. And does the Koji still create the bug from scratch or it catches the previous? It does murdering those internally. It catches the previous. I'm not sure. It does, it catches the previous. I opened that. Okay. The design of it is that it uses the previous repo's input and the same as it does for the builder and repos. Yeah, it's very efficient. It's supposed to be. If it's not, then it is a bug. But I've not seen that reported or seen that. I reported it, but... I haven't checked it for some time so I will check it now. The last one, which is not my favorite one, creating these images on-demand or on-interview phasers. On-demand, obviously, I'm obviously involved in some of those images. Definitely it will be slow. Let's say the package gets built and it goes into one of the images. Probably we are creating an image and waiting for 40 minutes to complete, which is not optimal, hence not my best option. And also, it's very complicated to implement because we have to create the repos and then create the images out of it. And then we carefully have to copy all these repos into the punchy compost later when we run the actual punchy. Not ideal. You're still not saving time on depth solving because, well, you're not really saving time on depth solving because that task is still going to get run because of each image build process. And on top of that, you're still downloading the package to the vain again? Yes, yes. And so, like, probably an easier quote-unquote when that would be to first get it so that it doesn't force you every time that we download everything, that there's a way to tell it there's a permanent capture. Yes, yes. And then the second would be making it so that if you've got a purchase tree already in place, you can use that as a source to feed in to produce it so that you can get some steps in the sense. Yeah. Thanks for the session. And nothing much. It saves time in image building and, like, need changes for Koji, Panji, and some smartness as, like, not my favorite. These are the four options that I was thinking about and I have taken sessions from you guys. That's really helpful. But this is the goal of my previous release engineer and now became my goal. Panji needs to be a gatherer, not a creator. So everything should be ready. Panji just goes there, collects them, puts them in place. That's the goal. I mentioned you. Why is it not from the bush? You're having rhythm to Koji. I didn't work here anymore. I'm a manager. Huh? It's your job. Any questions for this moment? This is just... So, I mean, the work here is going ahead and looking at how you're going to take takes. I'm wondering if it would make sense to basically create a root chart for every kind of broadness. You can just look at this and say, wait, we're waiting for this for 2 hours or just, you know, having a graph with you why it took that long on every compose. The comment was, would it make sense to have an overview of where time is spent in the compose process so that we can actually see what was the slope bar? And you can sort of get this information in part from the logs because there are timestamps for everything. The only downside is that due to some implementation details, especially for the image building, you don't actually get run times for the tasks themselves. You would have to query Koji about that. And if there are multiple phases running in parallel, like if you remember the diagram, they all start in given order and finish in given order. So it might be waiting just for another phase to finish. Even if we got the duration on each task and sorted it by what's the slowest task here, that would be useful. It would be useful if we get the duration for every single task. That's a fair point. And also I just wanted to add one more point over there. Yes, we can do that, but just act differently. Yeah, but so that you're seeing it act differently, but you if you can see it, glance how they act differently, maybe that would give you a better idea of everything that's going on here. Over the time. I'm not saying that one single one would be a serious problem, but yeah, definitely. Just to repeat the comment, it's that if we have this information, we can actually tell that these weird outliers are happening and where what suddenly starts taking longer time than it did before. Actually, I was doing some profiling on the package set phase like a few weeks ago and I found out that the second function which was called most of the time, and took most of the time was the interpreter log for threads. Like Python interpreter log. Over the time of the whole package set duration it's like blocking acquired threads Okay, so the comment is that from some profiling might be that the actual slowdown is ineffective implementation in Python that we're using threads in a non-optimal way. Again, this might actually be solved by the suggestion number two of moving this logic into Koji and not actually preparing the package set in Python. That part is mostly an improvement like enabler for further optimization. There was an item like the first step was checksum in checking checksums or something like that. Is this actually necessary or is it possible to track checksums that are on this or cash checks somehow ? The question is if we actually need to compute the checksum for the images or if we could cash them somehow and I'm not actually sure if it's possible because in each compost you get a different image. Even if the packages are same I believe that you will get a bit twice a different file. It will be different. I was just looking at it because I was wondering the same thing. It's just not that hard to make gen isoFS just spit out the songs. We do not always run that command. The comment was that when we run gen isoFS whatever the command is it can spit out the checksum as well. The downside with that is we do not have direct access to that for creating, say, live media. It's somewhere behind . So it needs to be changed in four different places. We also check some things that aren't . You're checking the checksum in two files and we can make it so you don't have to check some one file. The fact that we don't check some of the other files in the other files is not the right you can crawl with everyone. Even if you don't want to work on that. Yeah, we should just take the checksum data that's output by the image creation tool. The image creation tool either old gen isoEvent or the resoFS, they're going to output that information no reason to do it twice because it takes five minutes to wait. The comment is use checksums from the tools that create the image. The reproducible build guys are doing work on making iso creation and stuff to be reproducible. It's currently not totally possible because of date and time stamps and stuff like that. There's people working to try and make that reproducible so they could do things like that. The reproducible guys actually do have some of this work already in place since the resoFS which is one of the reasons why I'm working on moving to it in live scene creator which you guys don't use anymore but there's one of the reasons I have a branch where I'm working on actually changing over to that but the other thing is gen isoEvent is basically dead. There's some of that going on in a while. Yeah, it's good to see. The key we guys have switched over and I'm in the process of switching over for live scene tools. I know there's apparently a couple of ports with how morax actually produces bootable media that is blocking the resoFS from being used instead of gen isoEvent but... Well, so the problem is that morax assumes that the isoEvent tool cannot actually create the hybrid isoEvent one step. So resoFS can and so it mangles the way it does it and manually mangles the image once it and it screws it up in the process because the padding is slightly different between how gen isoEvent does and what does resoEvent does Yeah, it is easy enough to fix this just you know, you see how it's getting around here and I found that in my own code it was like, this is really weird but yeah, there's a couple of ways to like improve that. Okay, so this was a very long comment about possibilities of improving creating images with different tools that might be more efficient. So I got a question which is not a question because there was a discussion There's another slide that says special so let's read them that they want to show them right now. I can change this with one more thing. So this one word, you know, image short thing coming out, one more file coming out of it in the end. The good thing is it takes roughly 5 seconds to make so it's not going to make the timeline any worse but I don't really understand how to fit it into that because I would love to have a discussion with you guys perhaps. Listen, should I want to make a base snap after cooling for it so I don't plug it into that so it fits out of the whole structure. So I don't think I'm just going to get to these and then you know, you're going to have to have the lovely titles having to produce them to do a run time flat time. So that'll be a whole other set of things. That goes most likely for making images spark. I think currently I don't have a plan to create a composed process that I would consider to be just as we don't have to create the core parameters out of a composed process to get it to be done separately. Wait, what's the parameters? The base images are the applications that are built out of those on there. It could definitely be something that was, you know, you could even put it into there or it's another approach to that to run. I think we're not. We just want to figure out where people are going to be happy with something. We just want to give you guys five lines of show. I mean, I've run a Python program that will basically do it and have support for caching and stuff. We don't know where to plug it in and that's one of the reasons we're here. The simplest thing is just make it a run root task in Koji and then plug that into in. Basically, Cunji doesn't have a generic run root and it doesn't have. It basically knows how to call Cunji with a run root which is basically a generic do these things container essentially which is not exactly a container but yeah, so that's what I would do. Because that's what we do for war axe and other things now, just run them in a run root. Okay. Absolutely. Yeah, okay. I'll explain you in post. Okay. The buzzword of project makes it fantastic. Except for Koji. Well, I like Koji. So, this will probably be productive or not. So, a lot of us have mentioned caching things so that we get away from hitting Mount Koji for everything, every time. We have any good ideas on how to implement something like that. And because there's a lot of ways to do caching wrong and basically confuse the hell out of yourselves so that it makes it impossible to debug or you thought something was happening but it really wasn't because of caching, right? So, do you guys have any ideas on the best way to do something like that? So, the question is caching is easy to get wrong. Do we have some ideas on how to actually implement that? And at least myself, I do not really. The issue with the caching is that if we actually want to reuse the same cache in multiple places it gets a lot more complicated and we would need to make it so that the I mean, if we create a new cache for every single step in the process it makes no sense. So, we would need to make sure that the cache is used by the dnf code that's figuratively necessary we need to reuse the same in a different do as well. So, that part is also mythical. I have no idea how that works right now. That's another question though. It seems like some of the local fruit we have by the index of the dnf code you guys can build it can write to more than one place and then just put in a copy of Mount Cody sitting next to the S390 in Westburg. Well, oh boy. My calls are basically that the cache would be per builder and so for example the cache is per builder and we fill up over time. If this task went on this builder this time it would create a cache if it goes to a different builder next time it creates a separate cache but that's okay because the cache is an optimization if it's not there it will basically go get the cache but the idea here we could do an optimization where you say you went to this builder last time and that builder is currently free so go ahead and go there but the idea here is that essentially you make it generic enough so that all of these different tasks including pulling dnf including pulling other things could possibly reuse you know this cache that is per builder so if you're going to build like if you're going to create images using Anaconda and you need to pull down stuff from the build install that was created we can also pull those down not just rpms, right? That's Cody That's already in the berth So that it's actually per post that they're posted on Yeah, because then you just have a shared disk You can pull that in So let me summarize this Are you talking about writing cache for all these things so that it includes not just the tools that would consume the cache but also a bunch of the infrastructure around it that would maintain the cache which makes it even more complicated Right The question more is can we filter it through something like currently nfs is basically something that you can't create like a proxy for it, right? However, HDBP is something you can create It is already old cache The cache packages when all the downloads via HDB happen because none of the dnf processes are using nfs directly, they're all using it via HDB which has a huge big cache on coge packages which caches all the rpms so they're not hitting disk they're being served out of memory 90% of the time It's very rare that they're actually hitting disk The only time you hit the disk is when you're doing things like running free repo to actually generate the repo though because then you got a and then coge does optimizations like skip start and stuff like that where it assumes the file name is exactly the same nothing's changed and it shuts all existing and pulls the metadata from the existing metadata and just updates it rather than creating it So what you're saying is it is cached so we're not hitting mount coge but we are hitting a proxy that is outside of the build It's cached at the wrong point basically because the tools are still fetching every time and revolving everything and having to go through that work every single time but they're just not hitting mount coge The reason that's intentionally done that way is to ensure that every time that you make the new image, new whatever that you can recreate that so any caching that you put in there to make it some optimization elsewhere you then also need to take extra steps to make sure that however you go about doing that that you can then reproduce that exact same thing regardless of the state in which you start Right I think what a lot of us are thinking right now is that those extra steps are smaller than what we're doing There should be anyway Yeah It's literally just what's the build idea of the build that we're using at the list That was the question that was essentially that started this conversation How would we do it? Right And I think what you're saying is if you're interested in what we're already doing then it gets harder to do it right But is there a happy medium in there that helps Is our assumption that mount coge is the bottleneck not valid Probably not valid So things like the anaconda runtime we don't actually know what's inside of that so this is a very enlightening discussion of your out of time so maybe you can go over this Thank you Thank you very much


How do you increase your status? Men gain a lot of status through intelligence is a simple way. People underestimate that. Cutting edge in elect, which is not something that you have to be Albert Einstein with. I'll tell you this. Read, man. I've seen this happen 50 times in my life. When you're reading something, you'll be able to offer people insight, even people smarter than you, and they'll see that as kind of reciprocal altruism.


During the past few years, we have covered many aspects of Mankuri, Kafra, and Kufu, the three great pyramids of Giza. We have explored numerous amazing facts regarding these structures which have remained secret for many years. As the interest has grown regarding these three amazing structures, more people with suspicions, hypothesis, and technical and intellectual talents are fortunately beginning to approach these mysterious and wonderful structures in more explorative ways. We are experiencing the beginning of an ancient Egyptian renaissance thanks to the gift of modern technology. At the beginning of this year, an international team of researchers began investigating the buildings from afar, gazing at them with unusual cameras. Using state-of-the-art infrared heat detection technology, they have discovered some surprising anomalies regarding the heat signatures visible on their faces. But these thermal anomalies reveal are undiscovered shafts, more than likely leading to additional and undiscovered secret tombs deep within these amazing pyramids. The thermal scanning that they have successfully completed has revealed that there are many of these temperature fluctuations in many areas undocumented as containing anomalies. Thus, what the team has done is pinpoint unexplored shafts dotted across the pyramids. The team also found a particularly impressive anomalous signature located on the eastern side of the Khufu pyramid, very close to ground level. From the beginning, the team had always maintained that they would publicly disclose their findings. All of the staggering finds were made public by Antiquities Minister Mamdu El-Damati. During a press briefing, quote, there is something like a small passage in the ground that you can see leading up to the pyramid's ground, reaching an area with a different temperature. What will be behind it? said El-Damati. The scanning was done throughout a 24-hour period, allowing the researchers to monitor subtle temperature changes as the pyramids heated up and then cooled down during the day and night. Though the huge granite and limestone blocks which make up most of the pyramid, this technology was capable of recognizing the slight differentials in their temperature. By monitoring the speed of this heating and cooling, thanks to these miraculous cameras, the researchers were able to isolate several persistent anomalies. Thus, they may have just unlocked more of the pyramid's secrets in one day, using state-of-the-art technology than Egyptian antiquities or archaeologists worldwide have in more than 100 years. While the difference in temperature between most adjacent limestone blocks was between 0.1 to 0.5 degrees Celsius, the largest of heat anomalies discovered on and within the Great Pyramid was an impressive 6 degrees warmer than the surrounding bricks. So far, there are plenty of theories being put forward as to what these heat anomalies might indicate. Not surprisingly, with the leading assumptions being that of just empty areas, a hypothesis I'm sure some would like to make a reality. The good news is that the study, which is called Operation Scan Pyramids, will continue. Next, the researchers intend to use cosmic particles called radiographic muons to create a 3D reconstruction of the Pyramids of Giza in an attempt to map all the secret chambers and passageways within the pyramids.  Who built the Great Pyramids? How? Why? Questions many have attempted but seemingly failed to answer, although claimed as tombs with the different internal chambers within the largest, Khufu, named in representation of this purpose. Interestingly, Khufu, or Cheops, is the only one of the three pyramids with internal chambers. The other smaller two merely have tunnels beneath. An enigmatic box whose lid has long been lost to history lay within this enormous structure, long claimed to have been the sarcophagus of Khufu. However, although suspiciously small, no one seems to be able to explain how they got it into the chamber in the first place. It is as if the pyramid was built around, as it doesn't fit through any of the known entranceways. Since the 19th century, when these chambers were first rediscovered, a tremendous amount of research, though it must be noted, always supervised by official Egyptian antiquity academics, nonetheless remarkable discoveries have at least been partially shared with the world. Notably, Gontenbrink's door, yet the tomb of Osiris, where this once inaccessible tunnel led, was, once the media was permitted back into the location, found empty, claimed by officials as being found conveniently vacant. A room only discovered thanks to 21st century technology, according to mainstream Egyptologists, was somehow looted. However, there still lay many mysteries within this most intriguing of structures, and we would expect at least one, or possibly many more, which no matter how long it takes us to rediscover them, will be too big to hide. For example, although we once thought the tomb Gontenbrink discovered was inaccessible, the chamber at the top of the structure, one of considerable size, estimated at 30 square meters, is so inaccessible. It was only found with technology used to register cosmic rays, a technology usually utilized in high-energy particle physics, capable of tracking particles called muons, produced when cosmic rays strike atoms in the upper atmosphere. These incredibly sensitive detectors were first developed for use in particle accelerators, but they have also been used to gaze into the inner bowels of many geological and ancient artificial features. In December 2015, physicist Kunahiro Morishima of Nagoya University, Japan, placed detectors inside the queen's chamber to detect muons passing through the pyramid. Thus, any large chamber still hidden within the pyramid would be detected due to a higher register of muons than expected from denser angles. The chamber's discovery was cooperated by two other teams of physicists. All three teams observed a large void in the same location above the grand gallery. It was a big surprise, says Teyubi. We're really excited, he continued. The researchers say it might even be made up of two or more smaller spaces. Teyubi suggests that it could be, quote, a second grand gallery. It is a discovery which we are finding highly compelling. There are many mysteries to be found within ancient Egypt. Unexplained, seemingly impossible mysteries which litter the caverns, tunnels, flooded underground layers, and indeed the once inaccessible passageways only recently explored using advanced modern technology. However, some of the most perplexing mysteries lay in plain sight. Not only the great pyramids themselves, an obvious enigma for academia to explain the construction of, but many anomalous features which can be found within objects often leaving academics baffled as to an explanation. The Chiops are cofegous, being one such anomaly. Although these pyramids are entered and explored by millions of people every year, and indeed this mysterious sarcophagus shown to many of these inquisitive explorers, what many of the funded academic tour guide often leaves absent from their explanation of this supposed tomb is how exactly it arrived at its current location. As we have explored and exposed previously, the casing stones that can be found on many of the pyramids are to us not only indicative of another phase of construction work, once having been undertaken upon these structures, but due to the erosion present and the different styles featured are in fact indicative of more than one attempt to conserve these marvelous structures for future generations. Thus one must conclude by more than one now extinct advanced civilization. As such, the age of the sarcophagus of Chiops could be immense, so it is not surprising that it has encountered not only grave robbers, but has been vandalized also at points within the distant past. Furthermore, and perhaps most intriguing and frustrating, is that the sarcophagus lid is missing, a lid that could have explained the past contents of this mysterious box, or like the Tomb of Pachyl, exposed extremely controversial illustrations of possible past technologies. Unfortunately, however, or rather most conveniently for academics, this lid has never been discovered. Yet what is most perplexing regarding this diorite box, notably one of the hardest workable stones on earth, is that no one seems to know how the original builders managed to transport the box to its current location, deep within the bowels of Chiops. The diameter of this supposed tomb, being too large to have traveled down any of the known tunnels, which have so far been discovered within the ancient pyramid. This leaves us with two likely possibilities. One, that the diorite box was placed there and the pyramid built around it, which is a mysterious and confusing hypothesis, mostly due to the lack of markings of significance found upon the sarcophagus, or indeed the lack of any dedicated markings found anywhere else surrounding it. It is as though the box was placed there without much effort to indicate any importance to its existence. Yet to cut such a box, which has since been discovered to have been cast from one single block of diorite, would have taken tremendous effort, a feat that modern man would only accomplish with the use of diamond-edged power tools, not to mention the effort that would have been involved in moving this multi-ton stone into its found location. The second hypothesis regarding how this sarcophagus found its way into its current location is that the box itself was transported to its found location through tunnels and passageways we are yet to discover, possibly hinting at the fact that within this great pyramid there are indeed many more hidden layers and cavities we are yet to explore or discover. Maybe the placement of this seemingly inanimate box was placed there to suggest exactly this. Furthermore, what was on the lid of this supposed sarcophagus? Why is it known as the sarcophagus of Kufu when Kufu was not discovered within it? In fact, nothing was discovered within it. And why is the lid mysteriously absent? Where did the lid to this sarcophagus go? Why, if destroyed by grave robbers, was it not left where it lay? Did this lid contain controversial information, possibly pertaining to the original contents or indeed purpose of the great pyramids? We find the diorite sarcophagus of Kufu and indeed its unexplainable journey into the center of the pyramid highly compelling. Since the rediscovery of what is unquestionably the most puzzling, astounding and enigmatic side on Earth, the great pyramids and the Sphinx of Giza, we have been led to believe that what could be described as an astonishingly accurate, yet somewhat vandalistic later edition was once put there by a caliph named Al-Mama, now popularly known by its coin title, Al-Maman's Forced Entrance. This title, although argued as his work, has a pretty compelling tale attached to its possible original purpose. When one actually looks into what an incredible achievement this tunnel once was, it becomes apparent that it was cleverly bored by an ancient people, far more advanced than a ninth century caliph. Additionally, possible hypotheses have been put forth as to its origins by individuals who may have known of entrances into the pyramid. We in the modern world have either lost knowledge of or have been prevented from knowing about their existence. Hinged doorways made of stone perfectly counterbalanced to allow an average sized man to open and close them. Doorways along the structure's north face that when closed becomes seemingly indistinguishable from its surroundings. Are there still secret entrances along the pyramid's northern side? Quote, the Great Pyramid, a little way up on one side, has a stone that may be taken out, which being raised up, there is a sloping passage to the foundations. End quote, written by Strabo in Pyramids and Temples of Giza, Flinders Petrie. Yet regardless of these additional, highly compelling investigative leads, put forward in addition to an explanation for the tunnel's existence, its remarkable accuracy remains a tough thing for supporters of academia's tale of events to explain. As author Ralph Ellis puts it, quote, the main problem with the classical explanation was that Maman's tunnel is rather too accurate for comfort. It tracks into the pyramid in a direct line for the all important junction between the descending and ascending passageways. It is often cited that Maman had to turn the tunnel sharp left to discover the original passageways, a fact that Ralph had in the back of his mind when they first visited the Great Pyramid. But he ambled down the forced tunnel, rather mystified, because the left turn, cited in the literature, could not be found. Having backtracked the tunnel and to try again, that left turn seemed to be no more than a slight widening of the tunnel. In fact, the digging was almost right on target, end quote. For how does one know where one is when deep within the passages of such an incredibly huge ancient structure? Secondly, if instead argued as having been started from without, the same problem has to be solved. For how did one know how to create the initial angle? Although it is now the most used entrance, and although it has been drawn upon countless plans, to draw an existing tunnel's precise line of descent is far more easier than to have created said precise angle in the first place. And within the Great Pyramid is the remaining half of what has often been used to create a compelling, possible explanation for this tunnel's original purpose. Known as the sarcophagus of Khufu, an anomalous object found within the pyramid, an artifact we have covered in the past. No one can explain how this giant stone object came to be within the pyramid. It would not have fit through the existing entrance tunnels. However, at some time in the pyramid's life, someone smashed into this stone box, took its past contents, and the sarcophagus lid, an object that would also have not fit round the turns of the existing tunnel system. Yet would have fit through the forced tunnel, and due to the vandalistic nature of the tunnel itself, could be argued that this damage to the sarcophagi was inflicted by the same group of individuals who built the tunnel, one used to extract the so-called sarcophagus's lid. Is this the real past purpose of the tunnel? And if created by a caliph in the ninth century, how did he tunnel so accurately on target? And additionally, where is this lid now? Was this tunnel like the many different layers of casing stones indicate built by a later, yet also lost civilization, one who flourished far before even the ancient Egyptians? We find the evidence to suggest such highly compelling. What's behind the Sphinx's ear? Hidden in plain sight for many a millennia, there is clearly a blocking stone still in place, a blocking stone we would never have noticed if it weren't for a rather unusual source of information. Recently, we covered the amazing story of Kipri Yanovich Boriska, the extraordinarily intelligent boy from Russia, that from a young age has supposedly been able to remember a past life, a life as a pilot upon the once flourishing planet Mars, destroyed during a catastrophic war. What is extraordinary regarding this claim, however, is the remarkable information that Boriska has somehow been able to share from a very young age, information which has taken astronomers many years to realize. According to Boriska, life on Earth will change irrevocably when the Great Sphinx is unlocked using a mysterious mechanism behind one of its ears. Unfortunately, he has not given any further details about what exactly the opening of the Sphinx will do, though this was enough for us to notice the anomaly resting upon this very ancient monument. Boriska has unfortunately since disappeared. However, while in the public eye, he claimed that he was a reincarnated soldier placed here upon Earth to avert the same destructive fate as Mars, claiming that many of his kind exist upon the Earth, calling them indigo children, often stating all of this while in a trance, which is highly compelling, as he is not the only one who once prophesized very similar astonishing developments that would, one day, arise surrounding the Great Sphinx. Edgar Cayce, an American Christian mystic, would often answer questions on subjects such as healing, reincarnation, wars, Atlantis, and future events, also while in a trance. Was Cayce also an indigo child? Graham Hancock is another figure who has publicly claimed that there are many mysteries still left to be unraveled surrounding the Sphinx, specifically that a time capsule is hidden within the Sphinx, a capsule that we will only discover as a species once we are intellectually capable of absorbing its message. When it was shared within the mainstream media recently, that an enormous cavity had indeed been quietly discovered within the Great Pyramid of Khufu. The largest of the Great Pyramids, the only one with tunnels constructed within its inside, and additionally, the only one which is, in fact, eight-sided. The reaction by the Egyptian antiquities authorities was very revealing of their attitude towards secrets being revealed to the public. Not only were the claims made from reliable sources, but they are also backed up by extensive research projects, and indeed, it's resulting evidential data. However, this has not deterred Professor Zawi Hawass from publicly denying any such cavities very existence, shrugging off all claims and accompanying research as, quote, lies and hearsay. With such enormous hurdles in place, prepared to stifle any such discoveries from going public, it is inevitably going to be an uphill battle to expose the truth regarding the Great Sphinx of Giza. The Queen's Chamber, which lays within the Great Pyramid of Khufu, more commonly known as Chiops, has astonished, shocked, and mystified Egyptologists since its mysterious existence was discovered. The intrigue into this elusive chamber, along with its mysterious adjacent shafts, comes as no surprise once one understands the anomalous characteristics of their construction. As we have already covered before, massive cover-ups have been suspected as taking place surrounding this mysterious chamber since its discovery. Strange shaft tunnels, said at a 45-degree incline, no larger than 20 centimeters in diameter, run away from this room, and no one seems to know why. Not only would these ancient shafts require being hermetically sealed during the pyramid's construction to stop them from becoming blocked, but the excruciating effort that would have gone into making them becomes all the more of a confusing undertaking. Once you realize they were not even connected to the chamber, but hidden 40 centimeters away from entering the tomb within the walls, completely invisible from the inside of the burial room located deep within the structure. Chiops, noticeably being the only pyramid to have ever been constructed with such shafts, making their addition a popular mystery within Egyptian history. One leads out from the subterranean chamber, two lead out from a termination point some 40 centimeters from the wall of the so-called Queen's Chamber, or now popularly suspected to be that of an alien tomb among ancient alien specialists, and two from the King's Chamber above. Here is where our story becomes interesting. Rudolf Gantenbrich, famous for actually discovering the blocking door within one of the Queen's Chamber shafts, which could lead to an as-yet undisclosed tomb, has also made other curious discoveries within the Great Pyramid. Discoveries which could only be explained by modern covert explorations of tunnels that were supposedly to that point unexplored. Gantenbrich's cache being but one example of these mysterious finds, deep within the tunnel systems in the Royal Chamber, at a 90-degree turn going vertically upwards, a pile of papers, possibly wrapped artifacts, weighed down with a small piece of timber or stone, possibly another artifact, was discovered by Gantenbrich's robot. Also, during initial location attempts to find access tunnels leading to the Queen's Chamber, several blocking stones required removal. After the removal of the seventh block, a modern-era hexagonal steel rods were discovered discarded upon the tunnel's floor. Each section of the hexagonal steel rods measures 2.7 meters in length and was fitted with a round socket, which allowed them to be joined to the next section. In one of the lower shafts in 1872, Wainman-Dixon found three more objects which could be considered proof of prior covert exploration of the mysterious northern shafts. A copper grappling hook about five centimeters in length, accompanied by a small gray-green-stone ball and a broken-off piece of a square wooden slat or rod about 13 centimeters long, the woodwood today be the most intriguing of his finds. These artifacts expected to be remnants of the grave robbers' tools could have been carbon-dated, yet this fragment is the only one of the three to now be missing out of the London Museum's collection. Unfortunately, in his writings, Dixon doesn't say in which of the two lower shafts he actually found the objects, but he mentions them in connection with a northern one. Not only did these obviously highly intelligent people leave evidence of how they must have gotten in, but also traces upon the previous untouched ancient walls of the shafts within geops, clearly left by their previous robotic technologies. Other square metal rods have been recovered, along with other artifacts discarded within some tunnel systems deep within the ancient structures, meaning these guys got to the treasures way before we did. Interestingly, reported evidence of covert excavations continues to this day. Heavy-duty electrical supplies discreetly running into and trailing deep into the pyramids have been noticed and photographed by some of the more astute tourists. Witnesses to the sounds of heavy machinery being used beneath the site is also frequently reported, yet rarely followed up. It seems it's not a question of whether brilliant minds have achieved the seemingly impossible in penetrating these secret layers, but more a question of how and what astonishing finds have possibly been kept concealed. During a previous video titled Secret Missions into the Great Pyramid, in which we covered the most bizarre of artifacts once found in a seemingly inaccessible shaft, eventually discovered to be an entry shaft into the now-named Queen's Chamber. Just how this bronze ball, hook, and several bizarre fragments of wood found their way into the pyramids is unknown. We shared the fact that the wood had become conveniently lost, thus preventing any future dating of the artifacts or indeed this possible attempt to have once penetrated the pyramid, far before the Spanish invasion of Egypt, their modern rediscovery, or indeed before the entrance to the pyramid was located. However, in a rather strange, yet fortunate twist of fate, sitting within a collection of ancient Asian relics within Scotland, an Egyptian archaeologist was shocked to rediscover these cedar fragments, once mislabeled and thus never classified, lost for almost 70 years, yet re-found within an old cigar box. One has to wonder with our prior hypothesis, and indeed the convenience of the wood somehow becoming lost, was this a deliberate act by someone, possibly someone who realized the controversy attached to this artifact. What we find most compelling, however, and a possible motive to hide such an artifact are the now realized result of modern carbon dating, showing that the wood dates to somewhere between 3341 and 3094 BC, long before the claimed construction of the pyramid. Furthermore, although many have claimed that counterweights and timber structures were utilized in the construction of the pyramids, this wood not only predates the claimed date of their creation, but does so by some one to 2,000 years. So any mainstream explanation for this dating anomaly is severely lacking. However, it fits perfectly with our original hypothesis and is indicative not only of a far earlier date of construction, but could indeed have been a possible successful attempt at penetrating the pyramid's deepest inner chambers, simply due to the mysterious yet impressive location in which these enigmatic artifacts were found and subsequently retrieved from. Curatorial assistant Abir Aladani found the fragments of wood as she perused the Asia section of the archives of the University of Aberdeen. Quote, once I looked into the numbers of our Egypt records, I instantly knew what it was and that it had effectively been hidden in plain sight in the wrong collection. I'm an archeologist and have worked on digs in Egypt, but I never imagined it would be here in northeast Scotland that I'd find something so important to the heritage of my own country. End quote, as you can imagine, we find the wooden artifacts highly compelling.


Let's get started. I just want to say we've just come through as you everybody knows a historic snowstorm There are parts of the state For whom this is the biggest snowstorm in living memory Harrisburg, New York County. I think I think it's the biggest in living memory in the Lehigh Valley Philadelphia is one of the four biggest snowstorms. They've ever had and it was not only in terms of it's not just the magnitude but The rate of snowfall at different points of time yesterday was just Remarkable in some case some places two to three inches per hour And we were also surprised by the weather it started earlier and farther west than we thought yesterday And so we have been Continue to try to respond as as quickly and as effectively as we as we can so we have done everything I think we can to keep the the major roads open the state highways are open We'll continue to to work to widen the state highways Through Monday and Tuesday But I think the Penn Dodd is making good progress there We'll take questions and folks in the turnpike and answer questions about Where we are with the bed for summer set, but at this point We have I think 20 tractor trailers that are still waiting to move, but the drivers have voluntarily stayed with their trucks They're all safe and ready to get get going so We have come through this And I just want to say that that one of the reasons we were able to make it through and recover so quickly Is because the people of Pennsylvania exercise real restraint and and stayed off the roads when they could And it made all the difference in the world and allowed the emergency crews the local levels the state levels Throughout Pennsylvania all the affected areas and allowed them to do their jobs and getting the roads open As quickly as they as they could so I want to thank the people of Pennsylvania for doing what they did And for making this process go as smoothly as it did so Rick do you want to say a few words? Sure just very quickly and I just to re-emphasize that there's been a tremendous support Across the Commonwealth Neighbors helping neighbors and and you see that in every disaster and and that's actually fantastic I think the other message is remaining vigilant there from the standpoint A lot of areas are still in that response mode a lot of people are still digging out You still need plow trucks to escort ambulances and such if you do not have to be on the roads please stay off the roads, but Just overview real quick. We have currently Introduce yourself. I'm sorry. I'm Rick. That's okay. Thank you I'm Rick Flynn the director of the Pennsylvania Emergency Management Agency the county emergency operations centers there's 18 of them We're starting to see them starting to be demobilized or and their operations 20 emergency declarations from the counties and over 211 municipal declarations of disaster emergencies You know we talked a little bit our primary focus yesterday was the the center of gravity was focused in on the turnpike and as the governor pointed tremendous success of Taking care of everybody and that was our primary focus and with the small number of tractor trailers Left they're being taken care of and moved out. We had two shelters open And they were both in Bedford County associated with the turnpike incident 216 personnel were Individuals were stayed in that shelter. They're all gone They left for the day and then at 200 of the individuals from the turnpike state in local hotels and again Chairman Brown's here with the public utility commission, but statewide there are only 689 customers out of service that do not have power and that's primarily in the Pico area inside in the southeast Pennsylvania area with Philadelphia having 284 and I will tell you and as we say in a blue sky day, that's normal So for this to be able to have a very significant event With the pyro just tremendous tremendous from that standpoint. We're still activated in state emergency operations center We're going to continue to do so we're going to contact each county that's been affected to determine what their needs are we still have a National Guard troops on state active duty and We're working with again About 326 Pennsylvania National Guard members Aaron army are collectively working and ready to respond They've received over 19 mission assignments meaning that we gave them 19 missions to perform yesterday We're going to work with the counties do an assessment county by county to determine what their potential needs are and anticipated Guard resources or any other resources to do what we have the primary offenses that they have right now is Getting plowed out in their streets getting ready for schools. We can make an assumption schools Some schools will be closed some a lot of schools appear on delay But we're coordinating and working any kind of support we have can to provide to the counties. Thank you very much Okay questions Well as I say there were a number of surprises the magnitude of the storm this is the biggest storm this part of the country has ever had and So that was one second thing was the speed of the rate of snow fall Oh, so we were having a tough time keeping up with the snow as it was coming down yesterday Crews work throughout the night and and so expressways are open secondary roads The state roads are open to one and a half two lanes still some Requires some some work on the shoulders and we'll keep doing doing that But now that the snow has stopped we can do that But during the while the snow was coming down that was that was tough and then the third thing was as I said We we didn't expect it to be as far west as it was and we and it started I think about six hours earlier than we expected it to start so There were just a lot of surprises. It was an emergency and and our folks, you know kept Up with the that emergency as it unfolded as well as we could and we will have learned some things we're going to get together this coming week and sit down and Talk about lessons that we've learned things we can do to improve on what we've done this time around Yeah, I mean exercise restraint this is the weekend if you can stay home because it really helps the the crews do their work Unencumbered by by traffic So that made a big difference yesterday the people of Pennsylvania really contributed mightily to to what we were able to do and To the extent they can all of us can continue to do that today That'll make the recovery even even quicker So it allows the plows to to plow it allows them to to late to continue to lay down the brine, which is why Expressways like 83 are down to the the pavement So if Pennsylvanians can continue to do what they did that would be great. And as I was coming here today A lot of people out digging out their own driveways and sidewalks Anything else I think to the extent there were people who experienced hardship We we need to figure out how to do a better job there and and that's one of the things we're we're going to do I think statewide We did a good job, but there's no question that for the people trapped between Somerset and Bedford on the turnpike That was something that that we're sorry that happened So let me let me turn over to someone I can work I think at the point where we can say things are a lot better. Thank you I'm more confident the CEO the Pennsylvania Turnpike Commission and we will do an extensive After-action review to take a look at how this started some of the calls that were made out of my office some of the things that Perhaps we could have done a little differently and we'll also take a look at our response But I must tell you with over 300 people that responded to this incident both turnpike and our partners I'm not sure I could ask them to do another thing I'm tremendously proud of each and every one of those folks who worked for us And there's an old adage that says don't ask someone to do something. You don't think you could do yourself Was we watched those folks clear? Estimate of over 500 vehicles. I'm not sure I could have done what those folks did so as it relates to an after-action review Yeah, we're gonna take a look at all those things, but as it relates to the folks who responded I'm not sure I can ask them to do a to do another thing We are going to have the road open some time by mid-afternoon So we're gonna we're in the process of clearing those remaining 20 vehicles I know there's a lot of folks who are looking to get to some of our wonderful ski resorts And we're gonna we're gonna get the road open so folks folks can do that And we should have an estimate here by by noon as to when exactly we'll be able to open But I sure you will be some time sometime today That's what we're gonna take a look at through the after-action review when you're in a section of the roadway That's very mountainous. You're in a section That's what we call cattle shoots because of the fact that it's in a construction area and on a good day That's very narrow because of that setup So we're gonna take a hard look at how these tractor tractor trailers got together What was the the word on if they should or shouldn't have been there and the timing of that of that message And if it's in a construction zone should they've only been in one lane versus the two lanes because of the water lane in the left Side is is wider so we're gonna take a look at all that and we'll have a full report It'll be exhaustive and we'll spend some we'll spend a lot of time on it and make sure if there's anything We could and should do differently. We'll make that part of the next the next process A lot more snow Yeah, that's gonna bear where the snow is going to go is going to vary from municipality to municipality On the state roads. We've been plowing it to the side of the road in cities each of the cities has Their own place to take the snow obviously becomes a real challenge in a street in a city street or a tab borough street They have to sometimes use front loaders to get the snow onto a Dump truck to get it out because you can't just push it onto the sidewalk then people can't work So we can't walk so each each municipality has their own plan for doing that They've been they face this before and they will continue to do that some of them will take longer than than others to get out We we have the state has has offered full support When cities other municipalities have problems With the equipment that has broken down. They need extra help. We stand ready to to assist Any other questions? Okay. Thank you very much