There's stuff going into the kernel every release in terms of drivers and SOC support. I'm not going to talk much about that. But I'll also cover a little bit of development stats. These are the kernel versions that we've had over the last year, starting with 5.14 in August of last year. You can see the cadence, the duration of each release is about nine or 10 weeks. You can practically set your watch by it. The kernel is always released on a Sunday evening and we're three weeks in to the 5.19 release candidate cycle. So we expect 5.19 at the end of July. So this is boring. This is exactly what Lena said in Esquino yesterday that we like the Linux process to be boring. Just a couple of things in each of these kernel versions, I'll just point out. Something called MemFD secret was a new system call that was added. I'll talk about that. Some new tracers have to do with real-time analysis. A fair amount of Qualcomm and MediaTek driver code went into this release, clocks, pin controls and sound drivers. And there's a new driver called the simple DRM driver for direct rendering interface for simple frame buffer devices, which is good for embedded. So the MemFD secret system call, this just creates a region of memory that even the kernel can't access very easily. The purpose of it is to create a region of memory that you can't unintentionally overwrite and that's even difficult for this kernel to access. This is a type of thing that you'd use to store cryptographic keys or something. Of course, it's possible for the kernel to access everything with enough effort, but this kind of hides the keys a little bit better. In terms of 5.15, really, really big news is we finally got sleeping spin locks and I've got a whole slide about this I'm going to talk about later for the real-time preemption patch. The core scheduler has support for asymmetric systems. So it turns out that there's chips that have, particularly there's some ARM chips that have cores that can run 64-bit code and some cores that can run 32-bit code on the same core. So the scheduler has to be kind of aware of that so it knows which process is to run on which cores. And then if you happen to be doing something like a NAS server, you may be interested to know that there's a kernel version of the SMB server that's actually in kernel. It's not a full replacement for Samba, but pretty interesting. Something called printkindexing. So there's a bunch of tools that are used for parsing logs from the Linux kernel. And what happens is as developers change the printk messages, those parsing tools break. They usually write regular expressions to say, oh, if I see this event from the kernel message stream, that's something I need to report. But if the kernel messages change, then the log parsing breaks. But unknown, right? It's like silent. So now you can generate a list of all of the possible kernel messages for each release and see if the messages change so you can update your tools. So that's pretty good. There's a new data access monitor tool and I'll talk about that. And the kernel, OK, so I know in embedded, a lot of us are using old kernels that were not at top of tree, the kernel we got from our vendor or whatever. When you switch to 5.15 or above, you're going to notice that your builds break probably because the kernel now uses the dash W error flag, which means that any kernel warnings are turned into an error that will break the build. And that's intentional to kind of encourage people, nudge them to fix all those warning messages so that we can get clean builds. The data access monitoring system I thought was kind of interesting. So this provides tools to record data access and show visualizations. And there's a couple of different visualizations that are available. So it's a diagnostics tool, like a lot of the other tracers we've got in the system. But is it actually more? And in 5.16, we see that, yes, it's a sneaky way to get some new functionality in. So let me talk about these other ones. So the enhanced read-only file system continued to get some new features, multi-device support. So there's a lot of work been going on with something called IOUring, which is the new asynchronous API for file IO. This is for high-performance file IO. If you're not using this in your product, this is something maybe you want to look at. The thing in this release was they added support for SE Linux and SMAC security controls. And then data monitoring, operation schemes was added. So now the data monitoring, you've got a diagnostics tool that can monitor usage of memory, but it now has a way to actually do something with that in terms of hinting the kernel to reclaim memory. So this could help with low-monitor situations. So if you have a product that is really low on memory, this is something you might want to look at in terms of handling low-memory conditions. In 5.17, a couple of changes. The kernel can now decompress kernel modules within itself. It doesn't have to rely on user space to do that. And it turns out that that's important when you're doing a type of secure boot where you're looking at the hashes and having the kernel analyze the hashes of the kernel modules. So the load pin security module, for example, uses that. The real-time Linux analysis tools were added. And they used those tracers that we mentioned before. And then there were some changes to the flags field. So if you're using Fuse, I'm not sure if that's the right way to pronounce that, but a user space file system, there were some changes that might cause incompatibility. So I need to watch for that. And then 5.18, this is interesting. The support for some older architectures, ARMv4 and ARMv5 MMULA systems was removed. So it's not always true that the kernel just keeps getting bigger and bigger. Sometimes stuff gets removed. So there is still support for MMULA systems, but about ARMv7. There's lots of RISC stuff was added in this release. The tracing system now supports user events. So if you need to trace user space applications, you can use kernel tracing for that. And then the kernel compiles against the C11 language standard. I'm going to talk about that later. And then the final thing is that in 5.19, we're still in the merge window. This is not a released kernel yet, but we've got a lot more SoC support and device drivers. The big news in this one I think from an embedded standpoint is that the ARM multi-platform work is almost complete. So if you're not familiar, ARM, all the ARM kernels used to build separately and were different from each other. And now you can build a single kernel that you can feed at a device tree. It's called the generic kernel image that you can feed at a device tree and it can adapt and boot. Just like on Intel, you never had to build like an Intel for each individual board. Same thing on ARM. You no longer have to build a kernel for each individual board. You can build a generic kernel. So that has been work going on for about eight years. And it's really nice to see all that consolidated. There's initial support for a new architecture by Loomson, which is a Chinese chip manufacturer. So new architectures are going in. And then there's a new hardware timestamp engine, which I'll talk about that again later also. So in terms of developer stats, in case you've not noticed, if you've been looking at my URLs, I really owe LWN.net a debt of gratitude. And I see Jake is here and I'll buy him a drink or something after the session. But we get all this data from LWN.net. If you're not a subscriber, please go subscribe. They do excellent work. And if you want to stay on top of stuff in the Linux community, go do this. Pay them a little bit of money. So there's about 15,000 changes in this release, which is pretty average. You can see the areas where some of this is happening, device tree updates, folio patches, refactoring of the block and FS. Kristoff Hellwig is always in the top five because he's always refactoring something. If you see by lines of code, then you see that the GPU, the AMD driver, almost 400,000 lines of code in one release. So that's pretty heroic there. But you see, you can get into the top stats by removing stuff, too. And then in terms of commit log entries, the stats for this are just a little bit, numbers are different because I use my own tools for this, but it's about 13,000 to 16,000 somewhere in their range of changes every release. So that's every 10 weeks. That is pretty amazing when you think about it. Sometimes you've got to take a step back. Oh, the other thing, I want to back up and just point out one thing real quick. So Linus said that he thought there were about 1,000 developers on Linux every release. It's actually closer to 2,000. It's pretty consistently around 2,000 developers. And if you have never contributed to the Linux kernel, note that about 2 to 300 every release is a new developer. So if you've never contributed, don't be shy about saying, well, the Linux kernel, it's done by all these experts. No, there's new developers every single release. So go ahead and jump in and contribute your stuff. Just a real quick, I didn't finish the slide on this one, but about 40% of all bug reports are now coming from automated test robots. So the reported bylines, we're getting a lot of reports from test robots. OK, so technology areas. These are the areas I'm going to go over real fast. And again, I apologize. These slides will be made available after the talk. And actually, I have a detailed version of these slides that is the full hour and a half that I gave just a couple of weeks ago for Japan that I'll also include a reference to. But so in the area of audio. So PipeWire is kind of the new shizzle. You ought to be looking at that for your new products. Pulse Audio is not taking it lying down, though. They've just had a new release, Pulse Audio 16. I told you I was going to go fast. In the core kernel, we have these things I already talked about, MMFD secret, printkindexing. So the kernel compiles against C11. So what is that about? It turns out that they found out that because of our adherence to the C11 spec, where variables need to be declared inside certain kernel macros, according to that spec, it was leaking information that you could detect with speculative execution attacks. And so they made the decision to go ahead and move to a newer version of the spec that allows you to declare variables right in the for loop. And so this unscopes the variable better. And so that was actually the justification for this move. And I think it's a good move. I mean, we can come out of the 90s or the 80s, actually. It was the C89 was the last spec we were adherent to. The other big news in the core of the kernel is this Rust. This was talked a lot about by Linus and Dirk. So there's this effort going on to support Rust code in the kernel. The patches are still considered experimental. And the support for these patches was recently moved to a recent version of Rust. There's some grousing about how fast that changes. It's about 35,000 lines of code. It's out of tree. And there's ongoing debate. There's a debate raging this week on one of the kernel mailing lists about whether we should let it in or what are the ramifications. But most kernel developers are on a wait and see. And we're willing to see what happens with it. In terms of file systems and IO, so like I said, IOU ring, which is the new asynchronous IO that continues to mature. And there's a bunch of new stuff happening, zero copy networking. There's some patches flying around for that. In terms of enhanced read-only file system and F2FS, flash friendly file system, those continue to mature with better compression support, better X attribute. And then there's those changes to Fuse that you need to be watching. In terms of graphics, a couple of small things and a couple of big things. So simple DRM, driver was merged. The frame preferred FVdev subsystem was kind of orphaned, but it got a new maintainer. So that's really good for us in the embedded space. We have these old devices with 2D things. And it wasn't being supported really well by DRM. There was some conflict. The DRM maintainers didn't like some of the things that the new maintainer did, resurrecting some code that they thought should die. But that friction was all eased over. And hopefully, I'm hoping the new maintainer does a good job. So one of the things though, Mali GPUs. Mali is a GPU architecture that ARM has been supporting. That now has a fully conformant open GL driver, fully open source. And we have a lot of color board, I think, for that. Also really big news, NVIDIA has announced that they are going to open source a lot of their code. So they've actually released some of the code already. It's not upstream, but this represents a big shift for NVIDIA. And I happen to have an NVIDIA card on my desktop, and it's a pain. It's a hassle to deal with the proprietary drivers. So sometime in the future, maybe I'll be able to use an open source driver that's all upstream. That would be awesome. Not sure the effect of this on the existing open source project, which is the Nuvo project, but this is definitely something to watch and kind of big news. Networking, there's just kind of a steady stream of weird networking stuff going on. Not a lot of new protocols. And some of the esoteric protocols that get in are not really relevant to embedded. But there is this new thing with there's an internal function that allows driver authors to instrument, to provide the reason that they're dropping a packet. And so that will help with some diagnostics. OK, so real time. So this is really, really big news. Well, the small news is there's some new tracers, real time Linux analysis tools. The big news is RT preempt, right? So sleeping spin locks went in in 5.15. And then patches have been going in continuously through 5.19. So the sleeping spin locks, if you're unfamiliar with that, or this is what allows a process to switch while a lock is being held, and this is what allows you to get low latencies guarantees through the kernel. And this is a core, core feature of preempt RT. Possibly, I think this is probably the core feature of preempt RT patch set for Linux real time. So you have to turn on a config option to get this. But this has been extensively tested and verified. It's been 17 years in the making. It's taken 17 years to get this feature into the kernel. So a natural question is what's left. So if you go look at the actual patch set, it's not big. It's about 1,300 lines of code in only about 50 patches. And that may sound like a lot, but compared to where this was before, it's unbelievable. And they've upstreamed about 1,700 lines of code just since February. So I mean, if you were to extrapolate that, the whole thing would be upstream by August. But of course, there'll be details. Like the print K code, they just got up.  So if you want to hear more about that, there is a Birds of a Feather session with Steve Rosedet on Friday, so make sure you go to that. In terms of security, we basically have a whole bunch of kernel hardening. There's something called control flow integrity to make sure that you're jumping to the place you thought you were jumping to. And then mem copy bounds checking. And there's always a stream of specter mitigations. Specter as an exploit is the gift that keeps on giving. And so one thing I thought was interesting is there's actually been a removal of some specter mitigation code that was found to not be very effective and kind of bad for performance. I'm running behind, but I'm going to try to squeeze these out quick. So Zero Day is doing an awesome job of helping reporting bugs at the time of patch submission.  I just want to point out, if you haven't seen this other one, Compass CI, I think that's a new automated testing framework to watch. It's by Huawei. It's by the same developer who did Zero Day. So he went from Intel over to Huawei. So this is one, I think, to pay attention to, because if you look at it theoretically, he really knows what he's doing in terms of testing. There's a whole bunch of suites that have new releases and new tests, new coverage. Don't need to talk about those. GCC versions, GCC 12.1 was released in May, so point release for them. And the new focus seems to be one of the things that comes out is a lot of really good stuff to handle uninitialized variables. There's some runtime stuff and also some static analysis stuff that's in this version. So that's pretty good. So more catching of bugs. LLVM also had a point release in May. People are using this for whole distributions, not just for the kernel. And if you're interested in that, using it either with Yachto or some other way, there's a good talk by Bernard Rosencrantz around that. OK. Tracing. Steve's not here, so I can just skip over this quick. Oh, the only thing here, the hardware timestamp engine. This is a pretty neat system for actually causing the hardware to grab a timestamp based on a hardware event. So without any software interrupt required, that is really good for low overhead tracing of some events. OK, so I'm sorry how fast that was. I know it was a quick run-through. The slides will be online. The main reason for that type of quick run-through is just so you have in your mind a search term that you can go look up and go find it on LWM.net and read the full article. So in terms of industry news, I want to talk about just a couple of things. So we have the Open Source Security Foundation, and in particularly the Alpha Omega Project and a couple of miscellaneous things. So if you haven't heard of the Open Source Security Foundation, you've got to be living under a rock or you missed the keynotes yesterday. But this is a big, comprehensive project to analyze security throughout the Open Source ecosystem. And they're going to do a whole bunch of things, vulnerability disclosures, security tooling, best practices. They are well-funded. They have over $70 million of funding. And they're well-connected. So the people from the Open Source Security Foundation represented the Open Source ecosystem at the White House, at a special cybersecurity summit, and also at some recent congressional hearings. So this is good. We have a well-funded organization that's looking at security. Here's some of the activities they're going to be doing. They're going to be doing scorecards and reviews of existing code bases, a metrics dashboard, package feeds, and package analysis. And then doing a best practices badging program so that you can see if you're dependent on certain components, you can see if they have a certain level of security badging. They have some standards for vulnerability schema and something called Salsa, the supply chain levels for software artifacts. So they're doing the SBOT software build materials work. And then also guides and training. So they'll be a vulnerability guide and help if you want to work on security. One of the big projects they recently announced was called the Alpha Omega Project. And this is an effort to systematically search for vulnerabilities in open source code. The alpha part of it is they're going to identify the top 100, I think, or maybe 1,000, but the top critical code that is in widest use. And then actually go fund developers, go in and find problems and fix problems. But the problem with open source, or I don't know if it's a problem, is the benefit is there's this huge long tail. There's over 10,000 packages or libraries that need to be analyzed, and a lot of them have very few maintainers. There's like one maintainer or two maintainers. And the project doesn't have enough resources, even as well funded it is, to go after all of those. So they're going to do things like provide training and provide automated tools to help people analyze their own. And hopefully as a community, we can raise the level of security of all of the open source that we use. So that, I think, is a really good project. In terms of miscellaneous, I have to note, this one was interesting, that Intel has acquired Linutronix. So Linutronix is the company that Thomas Gleichsner. I don't know if you know who that is, but he's the guy behind preempt RT. So Intel has acquired them. They're now, I guess, Intel employees. But I think that represents a good sign that Intel is going to keep funding Thomas to work on security stuff. And then I've been hearing a lot more seeing stuff about Oniro. I had never heard of this before. Maybe it's been around, and I'm just out of the loop. But it's a project by the Eclipse Foundation. It's an IoT operating system. So it sounds like a user space stack, or if you don't have user space, just a stack. It's just driven IoT OS. I don't know that much about it. I saw a bunch of, I'm on the program committee for ELC Europe, and there were a bunch of talks submitted on this. So it has blueprints to build end user-ready products, according to information I've seen. And it's Yachto-based, builds the entire system at one time. So this is something maybe to watch and look into. I'll see more at ELC Europe. I got to talk some about, I guess I'm doing OK on time. So I may have rushed too fast. I got to talk a little bit about Mars Ingenuity Helicopter. So for years, I've been asking, have we got Linux on Mars yet? And it turns out we finally did. So we got the Ingenuity Helicopter. It's a little helicopter. And in there is some custom off-the-shelf hardware that is running Linux on Mars. It's just amazing. So it landed last February, well, not this last one, but February 2021, and performed its first five flights. And it was supposed to be a technology demonstration. But it did really well. And it is still flying. It has performed 29 flights so far. Just a couple of recent news. They had an opportunity to go back and look at the back shell as the rover and the helicopter as they come down the descent. There's a part of it called the back shell. It has the parachutes that flies off and crashes separately from the rover itself. And so they were close by, so they went over and took some pictures. The first time ever, they've been able to get detailed pictures of what some of the crash site looks like. It's not just a super cool picture to see a picture from this on another planet. But it actually has scientific value because they can analyze the debris field and see what the effect was of the landing, how fast it landed, and how the debris spread. So they can use that information for future. In terms of updates, it's the little helicopter that could. It's, bless its little heart, it's trying to survive the Martian winter. We are entering Martian winter. And it's getting, there's colder weather and less sunlight. And there's dust on the solar panels. And so it has been having problems maintaining enough charge. So the helicopter right now, it's supposed to charge enough to heat the CPU core, keep its core cold overnight, or warm overnight. Now it doesn't have enough power to do that.  So Linux shuts off overnight. And the core temperature gets down to about negative 80. These are off-the-shelf parts. In fact, some of the parts for the helicopter, they ordered from Sparkfun. And they're not rated for negative 80 Celsius. So anyway, so because of this shutdown, they had problems with the clock synchronization. And they missed one of their communication windows. They were very worried. Also, I think because of the cold temperatures, the inclinometer is now broken. But NASA figured out how to way to compensate for that by using other sensors. And of course, NASA being NASA, they had a patch ready. I don't think they have yet to submit it. But they're going to do a live update. They're going to update in the field on Mars. That's pretty cool. And just so NASA knew this was going to happen, or they figured it would happen. And so they had prepared and tested a patch months ahead of time. So the patch that they're sending is actually several months old. But that just shows you how cool NASA guys are. Guys gals. So it's keeping up with the rover as best it can. And it actually goes in scouts locations for the rover. Totally cool. I think it's one of the most interesting uses of embedded Linux in the solar system. If you want to get more technical details, Tim Canham is the JPL lead on this stuff. And he gave a great keynote last year's embedded Linux conference. And so you can get a lot of the technical details about what version of Linux and diagrams of their CPU utilization while they're in flight and stuff like that. So scorecards. So I have to move through my scorecard stuff. I took too long on the helical. I knew I would. So are we done yet? Well, it depends. Where are we?  I'm going to talk about three different scorecards, technology, development, and markets. So the original focus areas that we decided in 2003, how are we doing on those? Well, if you look at the contributions to the Linux kernel, we don't see very many on system size or boot time or really power management. Real time, I just said, was done. So we're done, right? Well, so congratulations, everyone. We did it. It's like, well, OK, let's back up a step there. So system size. Yes, we're done. We don't see a lot of system size patches going into the kernel. That's because we kind of gave up. The lower limit is about 16 meg. Well, I won't say we gave up. Moore's Law made it so that we could care less. Well, and that sounds bad if you say could care less. But alas, Linux is never going to run on 1 cent processors. And believe me, OK, there are such things as 1 cent processors, and they're not capable of running Linux. So those 10 trillion IoT nodes that we were hoping to run on, we're not going to be running on those. It'll be something smaller like our FreeRTOS or Zephyr. But we're in a pretty good space. So boot time, we're done. We don't see any boot time things. Well, cold boot time reduction. We don't see any new patches for that. Most products these days are just kind of cheating, right? They're going into suspend-resume, or they're going into a low power state. Like my phone, you think your phone is off, it's not off.  OK, so cold boot, again, we kind of gave up. But we've worked around the problems. Power management, is that done? This one is harder to call. So the stuff we need to do power management, ooh, three minutes. Ooh, I'm off on my timing. Oh, I better. OK, I'm going to have to really hustle. So we have the stuff we need to upstream, but it requires SoC and board support. So you really rely on your architecture or your SoC vendor to write the drivers that utilize those features upstream. In terms of real time, done? Yes, preempt RT code is almost all upstream. But again, it's going to require ongoing maintenance. And security, of course, is an in-progress thing. So the technology scorecard looks much more like this. I'm going to go through the development scorecard. These are the areas that make it good or easy to develop for Linux. And I'm going to say that actually we're doing really well. We have tons of build systems and distributions. We have good training and consulting. Lots of companies are available to help you with your projects. Our tool chains are really in good shape. And SoC vendors take care of writing a lot of the code for the tool chains. Debugging capabilities are good. I would say test systems are still in progress. I think there's still a lot of work, a lot of gaps in testing that we can fill with automated testing, particularly in the area of hardware testing. And then hardware support I think also is still in progress. So we've come a really long ways. SoC vendors now very customarily provide support right in drivers. But the issue is that oftentimes it's not upstream. And so it's very, and that requires that as product makers, if you're making an embedded Linux product, you're having to do a lot of extra work and carry a lot of extra technical debt because of that. But things are improving. And so I'm going to give our overall development scorecard mostly good in a couple areas in progress. And then the market scorecard. And I know this is not a comprehensive list of the markets, but just really quickly. Drones, we're doing great. Walmart is about to roll out drone delivery in 34 states. So get ready to be attacked from the skies. Amazon has also announced trials in California. Most of those drones are running Linux. Robots, robot operating system is really kind of taking over the world. It's already a majority in academia. And it's also projected to be 55% in commercial robots. Cars, AGL just released a new release with instrument cluster and infotainment. And it's used in a lot of telematics and self-driving. And then in space systems, surprisingly, I thought this would be the last frontier. But Starlink is running Linux, Space Rockets, the Mars helicopter, there's a lot of CubeSats running Linux. I think as we get more commodity hardware up into space, which will happen as launch costs come down, we'll see Linux even more in space. Routers, anyone can make a router. You're running Linux. Mobile phones, we have 70% market share with Android. And for consumer electronics, we're just killing it. I think it's about 100% of TVs, about 100% of DVRs, most cameras, a lot of audio. So we are doing really well in these verticals that I chose for this session. So conclusions. Overalls, we're doing really well. Linux is widely deployed and embedded. I would say we're at world domination. We're not going to be in the low-end devices. That's kind of disappointing. But you know, one-meg RAM system that's running on just harvested energy, that's going to be tough. And I don't think we'll get there. We're not going to see Linux on a serial box. That's a callback for any of you who are here 10 years ago. Core kernel systems, though, are in place to handle everything we need to do and embedded. New hardware keeps being made, so of course we've got to keep writing new drivers. Core code can always be improved. But the bottom line is we're not done yet, but that's OK. We've got job security for the foreseeable future. Hopefully you can use Linux in whatever embedded product that you want to build. And I want to encourage you to go forth and use embedded Linux, and I hope that, especially here at the conference, you find out the bits of information that will help you make a better product. And help you also become an active and productive member of our ecosystem and our community. And so with that, I will say thank you for your time. And I don't know, do we have time for questions, or are we? One question, I guess.  OK. Oh. Oh, I will get the slides up. There's supposed to be, well, keynotes, they give you an exception, and I'm calling this a keynote for that reason. But I'll get these up by the end of the day. So they should be on the site. The question was, when will the slides be up? And they'll be up soon. OK, I think that's it. Thanks.


So welcome everybody. I'm Mustafa Saeed, a security engineer at Intel Internet of Things Group, and I'm joined today with Stefan Schultz, a security researcher at Intel Apps. Today we're presenting our topic, a journey into fuzzing and hardening edge hypervisors. So to give a bit of perspective, what's the intention of this talk? So the ACOM security team really focuses on the security validation and offensive security validation to make sure that the product comes out in the best shape. And the idea today is to walk through some of our experiments and efforts and activities that is related to negative security testing or known as fuzzing, in order to conduct a comprehensive fuzzing campaign for different security components in the ACRON software stack. So agenda for today will be as follows. So first we will start with introducing ACRON and what are edge hypervisors and highlight some of the use cases related to edge and IoT. And then we go into fuzzing for product assurance and how does fuzzing fit into the product security validation. Later we start talking about fuzzing road trip and walk through some of the tools and the techniques that we followed in order to fuzz ACRON and any components related to ACRON. And finally we'll summarize some of the lessons learned and what are the future steps for this kind of activity. So to start, hypervisors for the edge. So first of all, what is the edge? The edge we can see the edge now in almost every single industry. It can be in transportation, it can be in smart factories, things like industry 4.0, it can be in smart cities. It even touches on some of the industry that we're not used to like using, like in retail, in financial services, in hospitality and so on. And the role of virtualization in such industries is to provide a solution for different use cases in order to have operation efficiency and a way in order to manage different workloads or what we call heterogeneous workload and consolidate them in a safe way. So the question is why don't we just use the cloud? What's the problem with that? So on the edge, we face a different set of problems and those problems are mostly related to bottlenecks in the transportation bandwidth, things related to storage cost and the limited amount of data that we can move from those devices on the edge back to the cloud. Even the nature of those application running on the cloud is a bit different. So we have real-time application, closed-loop application that require high responsiveness, low latency and in sometimes local authority and time-drifted decisions. And for that, we have ACRON. So ACRON is a flexible open-source lightweight hypervisor that is intended to come and support many of the use cases related to the edge and the IoT. So from architecture perspective, ACRON is a type-on hypervisor that is running directly on top of the hardware resources. It's a bare metal hypervisor. And on top of that, we have different sets and types of VMs. We have mainly three types of VMs. We have a service VM. We have user VMs and we have what's called pre-launch VMs. So the service VM is kind of like the control VM that has the native drivers that communicate with the hardware resources. In addition, it possesses the device model or some kind of emulation back end that would assist the user VMs. As we can see here, like Windows or real-time VMs in order to do device sharing or configure some of the devices directly to those VMs. Regarding the pre-launch VM, it's as the name describes, a pre-launch VM is not controlled by the service VM, but it boots in a separate boot path.  The goal of ACRON is really to address the gap between data center hypervisors and how partitioning hypervisors. And it does that in two ways. First, it tries to tackle the key challenges that we face on the edge. Things like workloads with mixed criticality, real-time versus non-real-time workloads, having safety and non-safety applications and so on. Even for the real-time application, some of them has real-time requirements, some of them have hard requirements, some of them has soft real-time requirements and so on. So these applications have a very diverse nature. In addition, ACRON focuses also on functional safety and certifying the hypervisor components, things like for automotive or industrial use. That sounds good. So the second part is being very flexible and very highly configurable. So as we can see now, ACRON can work for a different set of use cases. Take for example, the automotive sector. So for in-vehicle infotainment, we would require things like hardware resource sharing. You can think of running your instrument cluster in the service VM and using other VMs to run a real-time entertainment application, things like virtual offices and so on. And here the bottleneck would be that VMs are trying to access and share different hardware resources like ODO, VDU, storage and so on and so forth. Moving to another segment, things like the industrial domain where we will have different type of VMs that needs to run. For example, we will have HMI or some kind of OSs that presents some dashboards for the user to monitor things that is happening. It can also be real-time VMs that we can dedicate some hardware resources to those VM and those VM would be responsible for controlling robotic arms or even connected to PLC devices and so on. In more advanced cases, we have things like safety critical application that can run as a pre-launch VM and those have the task to monitor the system has for any application that have high availability requirements and so on. Finally, if we do not really care about resource sharing and we just need to run a couple of VMs directly on top of ACRON, then the logical partition would allow us to do that. And in this case, we won't have a device model and we don't have a service VM in the traditional sense, but just safety VM and user VM that is partitioned in a way and segregated in a way to run seamlessly on top of ACRON. So this was an introduction about ACRON, different use cases and so on. Now when we jump into fuzzing and understanding what is fuzzing and how does this relate to ACRON. So Stefan. Thank you, Mostafa. So in the next part, let's talk about fuzzing for security validation, what does it actually mean? So as you just said, there's a lot of requirements for ACRON to meet functional safety, to meet security and various use cases. For security critical products, Intel has a so-called security development lifecycle that makes sure that all of our products adhere to the current best practices and kind of systematically designed and evaluated and implemented. In particular, you may be of course aware that you should do security architecture, you understand the usages, the assets, the goals.  Then there's a big implementation phase, but then there's of course a question, has the implementation been correct? Has it been according to the detailed design? Has it been according to the architecture? Do the mechanisms that I have specified actually meet the security goals? So there's always so-called security validation phase where there's a separate team that will perform software review, static analysis, vulnerability scanning, all the dependencies of the software, such as libraries, crypto libraries, and so on, analyzed. And we have teams, for example, so-called red teams for penetration testing that basically take a fresh look at the system and see everything so far it has been designed according to best practices and standards. And of course, then at some point when the product is being ready for shipment, the cycle may repeat as new threats, new vulnerabilities turn up, new use cases, they might be a second generation of the product, so that this whole thing is a circle and will iterate over time. And on the other hand, of course, for each of these steps, we're always trying to meet the best practice and to improve our state of art. And one item in particular here is fuzzing. Fuzzing has been getting a lot of traction in academia as well as industry, it's becoming state of art for software validation, in particular security validation of software. And of course, as you will see, it can be quite difficult actually to apply fuzzing when it comes to low level software, firmware, or even things like microcode. And this talk is what this talk is basically about, looking at the different tools and trying to understand how they fit to Acorn and what we are doing in this space to improve on the state of art. So how does fuzzing, what is actually fuzzing and how does it work? Fundamentally, when you are testing a particular product or a sub system of a product, you would call it a test target on the right hand side that might be software or firmware or just a library and interface that is part of a bigger product. And in general, we can say that this piece of software will execute in some way on an execution platform that might be a Linux platform. It might also be a system on chip where firmware is executed or an emulator. And then we have test drivers which has the task of actually executing the target. Or exercising the target. So the particular API calls to actually launch them based on some input that is provided externally. Now, in the case of fuzzing, this input tends to be randomized and it's usually based on a corpus that we provide in the beginning. For example, if you're testing JPEG library, your seed corpus of inputs would basically be a list or a set of JPEG files. Or you would try to have all kinds of different JPEG types, maybe black and white, different sizes, different encodings and so on. And then you might implement certain randomization mechanisms in the fuzzer to damage these pictures, to randomize them, to recombine them and feed them to your test drivers that then exercise the target. At the end of this, if you have a box in your target, you will get crashes and these are, of course, the things we want out of the testing. We want to fix them, we want to analyze them and achieve a bug free product in the end in this way. Without any further feedback loop, just doing it like this, this is called blind fuzzing. It should be kind of intuitive that with blind fuzzing, you do not have a good chance to reach really inputs that are far away from your original input purpose. Unless you define a very good test driver that implements some of the formatting that your target expects, some of the input formats, so-called structure fuzzing or grammar-based fuzzing, mutators in the fuzzer itself. The random binary mutation of the seeds will usually be discarded and lead to invalid inputs early on and not reach very deep code in your target. So there's been an extension which made fuzzing much more popular and effective recently, which is the so-called feedback fuzzing or coverage-guided fuzzing. In this case, the platform actually observes what is going on or maybe the software target has been instrumented to record what it's doing, to record the coverage, for example, that is achieved by a particular input. And that coverage is returned to the fuzzer as a kind of feedback. And the fuzzer may use that to say, oh, this input has actually triggered some new interesting branch in my target. So I'm not gonna throw it away and take a new seed input and randomize it. I'm gonna keep that input because none of the seeds that currently exist that I know of actually reached this particular branch. So doing this many million times, the fuzzer will iteratively build up a corpus of inputs, randomize them, splice them against each other and find more interesting inputs that solves a lot of the typical branches and conditions and corner cases in our typical software products. And doing this, of course, we will get a much better coverage out of the overall fuzz testing. That in turn, of course, leads to more interesting corner cases that are being covered and therefore also potential crashes that might turn up. And that's, of course, what we want, right? We want to find these crashes and bugs before the software is being ready for shipping and for the deployed in products. Finally, for those of you who have already heard about fuzzing, there's, of course, a lot of other aspects which are not so much in the focus of this talk. Once you have this system in place, there is, of course, a lot of additional aspects such as the overall automation and integration of your fuzzing campaign, the triage of the crashes that you find, you need to sort them out, you need to find out, you need to diagnose what is going on. And if you imagine if you get thousands of crashes out of such a campaign, you actually need to sort, you need to start automating this process and sorting out which crashes are relevant, which are the same crashes and who should fix them. So there's a lot of work in all of these areas and many of them are also subject of extensive research in academia right now. But let's get back to fuzzing for software validation. If we are, basically, if we are looking for product assurance, we have slightly different goals than maybe you have heard from fuzzing that is used elsewhere in the community. Our goal is not to run a fuzzer once, find the bug and report it or get the bug bounty for it. But our goal is actually in increasing the software validation, increasing the assurance that you achieve when deploying ACON as a security product in IoT. So what counts in this case is actually much more the overall usability of the tool, the long-term return of investment. So given that you have this setup, how easy is it to automate it? How easy is it to integrate with existing validation backends? How good is the overall performance? Can it be easy to debug, to automate? And in particular, the developers or the software validators who are in charge of the product, is it easy or manageable for them to actually understand the solution and extend the solution? For example, writing sort of test drivers that exercise more of the code. So these are all very important aspects for product assurance, which are not necessarily the highest priority when you do fuzzing for bug bounty or penetration testing. And with that, I give back to Mustafa, who has looked at various tools to see what is the most effective and state-of-art tools for fuzzing a hyperwriter. Thanks, Stefan. So as Stefan mentioned, there are different goals that we have when we try to, when we decide to integrate fuzzing into our ACON offensive security validation cycles and so on to make the product much more secure and robust. So the first thing we asked ourselves before even starting to pick tools and choose what to start with, the first question was, we took a deeper look into the security architecture and we tried to identify what are the most critical software components that we should look into. Basically, we have three main component as part of our TCB or the Trusted Computing Base and that is the hypervisor, which runs directly on the hardware resources. Another kernel module, which is called VHM, which is the virtual IO and hypervisor service module. And finally, the device model in user space. All these, these three components communicate with each other. So the device model is nothing but something similar to the QEMU-like application, which has two main tasks. First is to create or configure new guest VMs and start those guest VMs. And the second task is to have an emulation backend or provide some of the emulation services for those guest VMs in order to utilize and do resource sharing. For the VHM, it's a middle layer between the device model and the hypervisor that supports the device model in making communication with the hypervisor through down cones and also support the hypervisor to notify the device model with some of the requests through up calls. Of course, there are other components which is not part of our TCB. However, they are very critical to the overall system operation, things like the virtual IO front-end drivers, which is how the user VM try to achieve part of virtualization, how they actually achieve sharing of hardware resources, how they communicate to the device model backend. There are also the emulated device drivers that some of the devices are directly emulated using trap and emulate, and those are memory region in the hypervisor that are tracked by the hypervisor in a way in order to see if there are any requests or operation happening on them. Finally, we have the pass-through drivers, and the pass-through drivers are some of the devices that, for example, ACON hypervisor have the possibility to dedicate some of the PCI devices to the VMs. You can think of a real-time VM where we want to assign a device directly to this VM and be directly under the control of this VM. In this case, we assign this device to the VM and then we use the pass-through driver in order to make those VMs directly communicate with those devices. So that looks great. So even going one step deeper and trying to see from each component perspective, so from the hypervisor perspective, the hypervisor run in a more privileged mode called VMX root mode, where any software that runs on top of it, either in the service VM or in the guest VM is considered untrusted. From the VHM perspective, the hypervisor is the only trusted component, but any other software that is running in the service VM user space or even in the user VMs is considered untrusted. Finally, going to the device model where it runs in the service VM user space, it considers the VHM and the hypervisor as trusted components, but anything outside those components is untrusted. So any kind of data or code that is coming from the user VMs and so on. So that is the security architecture for ACRON. Looking to this security architecture from a different perspective, so if we focus on the interfaces and how these components communicate with each other from security perspective, we find that we have different set of interfaces. First, we have an interface between the service VM, VHM kernel module and the hypervisor, and they communicate through hypercalls. The second interface is how the user space application in specific the device model or any other user space application that might communicate with the VHM through our second interface, which is normal syscalls or traditional OS way to communicate between user space and kernel through IOCTLs and generally system calls. The third interface is the device model, as we mentioned, one path for the device model is to configure new VMs and configure new guests. For that, we have a command line interface that allow us to pass some of the configuration and the parameters that we want to pass to the device model to create those VMs. So there is a command line interface, a standard STDIO interface where device model accepts some input through the command line. From user VM perspective, so we have the virtual IO front end driver that exists in the user VMs that communicate with the device model or the virtual IO back end drivers, and that is happening through virtual IO devices. So in part of virtualization, those virtual IO devices are nothing but PCI devices that contain some of the MMIO region and have PIO, so PIO and MMIO region in order to emulate those devices. So some regions are used for emulated devices. We also use MMIO, DMA, in order to access some of these devices, which is not part of virtualized, but rather really emulated. And we have pass through. So pass through is also using PIO in order to communicate with those hardware resources. So the last three interfaces, so four, five and six, focuses more on the memory regions that is trapped and emulated or handled by the hypervisor in a way in order to be serviced either in the hypervisor or by the service VM. So those are all the security interfaces that we have. So the question now, what tools fit better to those components or what tools would work better to target those interfaces? So the first tool that we picked was the syscaller and syscaller is a coverage guided structure aware kernel fuzzer. It is very well established with tremendous Linux support. And we use syscaller in targeting two interfaces. One, which is the traditional usage model for syscaller, which is the kernel module. So here is the VHM. And another approach that we tried is also extending syscaller in a way that we can target the hypercalls. So the interface between the VHM and the hypervisor. So what is the VHM? So let's take a deeper look into the VHM and hypervisor. How did we actually target those, the code base in both components? So the VHM, as we mentioned, is a middle layer with some services task or some modules or some handlers that would allow user space device model to talk to the hypervisor. Those kind of handles do diverse tasks, things like VM management to create starts of VMs. It can also work with introps and handling introps. It can even go further to handle the IO requests and try to match it either from the hypervisor to the device model and so on. So picking the VM management. So we have API like hypercode create VM. As you see in this API, we have two parameters or payloads that is being passed as part of the hypervisor. Those two parameters is a potential candidate that can be fuzzed. What if we inject some malformed pieces of data or malformed kind of data structures into those parameters? How would the hypervisor behave? Another kind of services that is happening at runtime. So if a service VM, if a guest VM is running, this guest VM has to send kind of request to the service VM. So these requests usually require things like notifying the VHM and knowing that those certification has been handled and finished and so on. So as you see as well, it has some payloads or some parameters that is potential candidate to be fuzzed. So the goal here is first to fuzz the IOctals going from the device model down to the VHM module and then later to propagate some of the fuzz input from the VHM down to the hypervisor. So how did we do that? So syscaller, so how does syscaller actually work? So syscaller can operate in what's called remote mode or isolated mode where we can have some of the syscaller component run in a separate machine and then it start fuzzing the target machine which is totally remote or isolated. Syscaller contain different components, some of them run on the host, things like the sys manager which is responsible for starting and managing those VMs that we want to test or that contains the software that we want to test. It's also responsible for copying some of the other components things like the sysfuzzer and sysfuzzer is a process that is running on the unstable VM or the target VM and it is responsible for things like input mutation, input generation, minimization and so on. So some of the input that's generated by sysfuzzer is propagated to what's called the sys executor that takes a single input and creates some kind of coals or fuzzing programs that can be propagated to the target. In our case, we implemented a test proxy that lives inside the device model and takes those input and then propagate them to the VHM first and later to the hypervisor. So the idea here as we've seen in the last slide those APIs like for creating VM or identifying the end or the finish of IO request can be fuzzed first in the VHM layer. So we would see if any issues or any crashes or missing validation is happening in the VHM layer. And the good thing is that we, syscaller instrument the kernel or it gives a possibility to instrument the kernel. And with this, we would get coverage information from the VHM module. So we really know what kind of execution partners have been experienced and so on which would allow us to reproduce any issues that happen. Sorry. So later those kind of coverage information would be propagated back to the sys manager and it keeps a little database that knows or that manages some of the corpus that created those crashes in the VHM. The second part is what the extension that we made is to take those fuzz input and even propagate them one layer down to the hypervisor. So we take the fuzz input from this executor, try to fuzz the VHM. If something happened, we would know because we get coverage information. If not, we can propagate it to the hypervisor. So it would allow us to fuzz all the hypercalls that is happening between the VHM and the hypervisor. So in general, syscaller is a very stable tool. It has a lot of these disadvantages being well supported by the community. It's easy to use. So it provides some declarative description for those structure that we want to fuzz using a language called syslang. It allows us to do fault injection, sanitization and even automation using components like syspot and a possibility to reproduce those things and minimize those fuzz input using things like sysribro. The downside for using syscaller with fuzzing the hypervisor is that we do blind fuzzing for the hypervisor in the sense that we do not get fuzzing, so coverage information from the hypervisor. And in case we need to do that, since hypervisor is not a user space application, it cannot use the compiler libraries in order to get coverage information. We need to put some of the coverage libraries into the hypervisor source code, things like gcall and so on in order to get raw coverage information from the hypervisor. So this was syscaller. The next component is the device model command line and here we want to fuzz this user space application and libfuzzer is a very good candidate to do such tasks. So it's a coverage guided fuzzer that is working as an in-process fuzzer and we use libfuzzer in order to target the command line parameters. So can we fuzz those command line configuration and see if that would impact the device model? So how we did that? Before we talking about how we did that, so the device model from inside, as we mentioned, has three main components. First, it configures VMs or pass those parameter to configure the VMs. And the second thing is to provide those emulation back ends for legacy devices or for virtual IO devices. Starting with the VM configuration. So how we implemented libfuzzer is first you instrument the device model binary. So the device model is just a command line utility. You can pass some of the arguments to configure, for example, the graphics, some of the, for example, the graphics configuration, things like fence registers, graphics hidden memory, memory aperture and it allow you to assign some of the devices as you see in the minus S command. So we would give some of the virtual IO devices and enable them for a user VM. It also allow you to configure memory sizes for the user VM. So that's how a device model worked. We took those configuration and we inject them to the libfuzzer entry function, which is called LLVM fuzzer test one input. And this function in our implementation does nothing but taking this configuration VM file, try to transform it into a data structure and then inject one of the fuzzing input to one entry of this configuration file. Then we call the main function or the main function inside the device model, that's the organic starting point for the device model. And then those fuzzing input would be propagated to the parser or the part of the device model that is responsible to take those input and try to cut them into smaller parts and propagate them to the main function. In case we have a crash, then this function would retain, return back to the LLVM fuzzer one input function. If not, then it would continue to propagate those input to the other main functions, things like creating VMs or like loading the software or doing all the core parts of the device model to configure a VM. If this kind of operation works or fails at the end, we would know because we would get feedback from such binary or from the device model to this function. And then we would be able to mutate the input and tackle new parts of the device model code. So the first part of the function, as you see, is initializing those configuration based on fuzzing input and the second part is executing the main function based on the fuzzing input. So LVM fuzzer has a lot of advantages being very fast. It's a standard tool, it's a state of art tool. It's very easy to set up and it gives us also the possibility to enable different sanitizers, things like M-SAN, AppSAN, K-SAN and so on. One of the or some of the disadvantage that we faced is that we could not really fuzz the backend emulation part of the device model using lip fuzzer because there is a lot of validation that happens in other software layer, things like the VHM or the hypervisor before we reach the device model. And for that, configuring lip fuzzer to target the emulation back and would result in some kind of false poster, which is not the best approach. Also with lip fuzzer, we might have to do a lot of code refactoring to remove any kind of exit statement. We need to do safe returns to the single entry function, which is the LLM fuzzer test one input. So this was lip fuzzer and our experience with lip fuzzer. The last component, which targets the three remaining interface in the guest, which is the MMIO, port IO kind of interfaces, which is considered very critical because a guest VM is not trusted in the ACRON software stack. And for that, we used HyperCube or it's a research kind of OS that do fuzzing in a different way other than SysColar and lip fuzzer and all these tools. So HyperCube OS is a multidimensional fuzzer that is targeting more the low level software. It's consists of two main components. So one component that enumerates different interfaces, so enumerate the assigned PCI devices, it enumerates things like the local APIC, input output APIC and all these low level stuff. And also it contain a fuzz engine that after we enumerate those interfaces, we would be able to inject militias or melt form data into those interfaces. From integration perspective, it's very easy. So HyperCube lives as a normal guest. So it runs in any of the ACRON scenario, either in industrial or IVI, it's nothing but triggering a normal VM using the device model and it lives on top of ACRON to experience those different interfaces as we can see here. And from organization perspective, the HyperCube OS starts at the beginning by the device model. So as we start normal guests, if we want to start HyperCube, we have to define what kind of devices we want to test. And here we define some of the virtual IO devices that we think if we assign it to this guest and then run HyperCube, it would be able to fuzz them or inject bad input to those devices or the memory map region related to those devices. So after HyperCube boots, it do some initialization, things like setting the global descriptor table, the interrupt table and so on. And then it starts to register different regions. First, it start with the MMIO and Port IO region. It then goes to the PCI devices. And if we see here, those are the PCI devices that we have assigned initially by the device model. It then tries to enumerate or register some of the other regions like APEC and local APEC. It creates a fuzzer and then it uses an integrated kind of soil random number generated in order to have a seed. And then we use the seed to start fuzzing. This was our experience so far with different fuzzing tools and so on. So Stefan, maybe a bit about our future work and what's ahead. Okay. Thank you, Mustafa. So as you saw, HyperCube is quite easy to get started with and it's actually a really nice tool to test the low-level interfaces for the low-level VM interfaces with Acron. But the problem is that it is actually just a blind fuzzer and we want to have a coverage feedback and coverage guided fuzzing using something like HyperCube. And the way forward that we are looking into for this direction is basically kernel fuzzing using KFL. KFL is another research prototype also from Ruh Universität Bochum, the same people who also developed HyperCube. And the idea of KFL is actually to launch the target inside the virtual machine using QAMU and KVM, modified versions of them. And you would basically, again, you have a test driver that stimulates the target operating system inside the VM. It receives its input via shared memory from QAMU and from the actual fuzzing content. And then using Intel PT processor trace, we can get feedback for each execution that is being decoded by QAMU and returned back to the KFL fuzzer. The advantage of this is that it is nicely scalable. It uses a well-known interface, which is the virtual machine interface for executing and scaling VMs in the VM backend. And in the latest version, which is called NUX, we also have advanced teachers such as very fast snapshoting, fuzzing of multiple interfaces, as well as a notion of grammar or structured fuzzing of the VMM targets. So in this case, ACORN would run as a nested hypervisor inside KVM, and we expect to achieve this way the feedback also for the hypercube fuzzing case. So to summarize, we have seen that there's a range of targets that we need to pursue for Syscola for ACORN, sorry. They're actually quite different in nature. There's the device model in user space, the hypervisor itself, and there's a kernel module, the VHM, and they all interact in interesting ways and produce a number of security boundaries that have to have good security testing. Syscola has been very promising on the kernel side and hypercore validation side, while hypercube has proven to be very effective and scalable on the guest VM side to validate from the perspective of the guest VM. And moving forward, we are very interested to run this experiment using KXL and NUX using hypercube as a guest in a nested ACORN setup. Litfaza is very interesting to fast the command line interface and configurations that are possible with ACORN, but in general, it has been a little bit difficult to get it working for the device model, for the emulation of devices, which is in user space on the service OS. Right, so overall, I think we've seen that there's a lot to be done for improving, fuzzing, even for low-level software. There's no tools that cover all our bases, but of course, we're working on it. We like to also collaborate with academia and other projects in the space and extend them. And since this is a source project, we also plan to upstream integrate and automate all of these tools. So with that, I thank you all for listening and we're interested to hear your questions and comments. Thank you. Thanks.


Hey everybody, Jason here for jaspyscasebreaks.com. Oh, that's right, huh? Technically. 2022 tops tier one baseball, 12 box case break, previously number two just popping off here at the national here in Atlantic City. Again, here's all the names there. So from Andy, from the D-backs down to Rick Thomas. Jesse, took one for the, I mean, sorry, Jesse. Terry, I thought I read Royals, Jesse. Terry, took one for the team and got the Astros. So let's see what we can get there. Actually, sorry, I don't want to print this. There we go. It was fun. Yeah, take your lunch right now. Yeah, so I can just dive right in to there. I mean, you guys can do it right after you guys. Yeah, my break is after you. I mean, before, once you're gone, then that's my break. Kind of to open. All right, here we go, guys. Terry wants a podcast. Put them on the podcast. Cedric Mullins. A lot of information. That's what they want, they want content. Jackson Kaur, for the Royals. I got to hold 80, 60, 70, 70 bucks. Really? Yeah, so I got to see. What are those now, like, $200,000? Wow, that's cool. Did you see the PSA card already? Ours and the 10 around here? Toscar Hernandez, yeah, even the 10 and the SQC 95s here. I think the male is out here. I think the 9.5 sale is gonna influence whoever owns the three 10s to sell the 10s. I mean, supposedly, can Golden offer like $30,000 for one of them, like I said, no. 30 million, you mean? 30 million, sorry, yeah, 30 million. Imagine. Imagine, dude, it's crazy. All right, nothing much. I had a wanderer in the first case, that was nice. Is it 2020 too? Mm-hmm, just came out today. Oh, I didn't know it was now. Yeah, it came out today. It came out on a dude, though? Oh, so what is it, dude? Oh, it is my day, all right. Lance McCullers, no, it was about the safe. Can't be breaking early. Kibri and Hayes, yeah? I mean, last year, the Pavilion's always on the side, though. Have you used this your first national with this building? No, I was, I went in 2016. I like this one, I should have known just more, like. Yeah, I like that. The other one is like to the side. I like to set up here better. Yeah, this is a nicer convention center, I think. Last now, for the Rays. Rays is Matthew Wood, there you go, man. Hopefully we get your wanderer as well. And nice, Juan Soto, I just seen this guy a couple of days ago. Juan Soto, where's he gonna end up? Guys, he has to, well, August 2nd? Or I guess the Nationals have to, August 2nd? 26 out of 175, that's Rick Thomas. Good, can I have more 75s? Do we have any more? I guess I can put them in 100s. I didn't even use it. I can put them in 100s, I guess. What, 100s? No, I can put them in 100s, it's fine. That's fine, no problem. All right, next one. Where's my knife at? All right, we got a Nolan Aronato, the relic there for the Cardinals. We have a Nick Fortes. It's a 299. That's for the Miami Marlins, which is Donald. And then we have a Joe Carter. Nice signature there, Joe Carter. That's for the Blue Jays and Austin Caruso. I think it's supposed to be one patch auto in every case. And that was in the last box, last case. Joe Musgrove. We have a UL Bozo. UL Bozo for Texas.  All right, next one. We have Johnny Bench to 1.99. Mason Thompson. Washington Nationals, another one there for Rick Thomas. And whoa, a one-of-one Donnie baseball. That's gold ink for the Yankees. Who has the Yankees? Gary, there you go.  Tier one talent. I wish Joe was here. He's taking a lunch break right now. He'll owe you a train whistle. Very nice. So we got our first one-of-one in this case. Very nice. Is the internet fine? It works. Does it have a computer too? Or no? I haven't had any. Computers fine though? Yeah. Is the charger actually charging while you're doing it? Yeah, right. Yeah, it's charging, 60%. Okay, that's good. Mookie Betts to 1.99. Terry, I got you. I got you. Jim told me to 3.99. Miguel Tejada. Shit. Like, send someone for Nick. And we got Fernando Tatis Jr. to 49.  Incu, there you go. Yeah, stand up and walk. Yeah. They're better than like those white party ones. Yeah, yeah. These ones are horrible, dude. I sat on them at the Del Martia like that. I was dying. See, if we didn't have to like slouch over, then it'd be cool. Hold the L2, oh, sorry, not hold L2. Austin Riley. There you go. What've you got? What? I'm getting a little hungry too. How's that run? We have a Devi Garcia. Nice. Gladdy. Little junior to 125. 45 out of 125. That is for the Blue Jays going to Austin Caruso. We're halfway through, guys. Five more boxes. Go figure right now. We're not seeing no Astros, right, Terry? It's like every other break there's Astros. Come on, we got to get Terry and Astro, one-on-one. Rafael Devers to 299. Kyle Lewis. That's Chad Dolls' favorite player right there. Made of steel. Mariners, that's Lonnie. And then we got DJ LaMeu. Yeah, we like Jordon. We like Jordon. Jordon was not part of all that scandal, so. I think a lot of people like him, instead, right there. Randy Johnson. Oh, what's up, man? What's up, Jason? How'd it come to say what's up, bro? What's up? Are you guys set up here? Uh, walking. Walking? Buying some stuff? I don't write that as I am. I know, actually, it was crazy. I hope he comes back strong. I hope he comes back strong. I have some PSA-10 prisoners, because I don't want to sell just yet. All right, see ya. Lucas, Geolito. It's because he remembers my original card account. And for the Ray's, Shane Bass. Shane Bass. There we go. Bass is loaded. What's going on? See your big hit already? Or am I stalling it? All right, three more. Tori Hunter. You heard it? Nice. That was pretty nice, though, man. We got Yasmani Grandal. Ryan, what's going on? I know, thank you. I'm never really on the East Coast. I love how the East Coast is, though. I always mention it, you know? You got Atlantic City, Philly, Baltimore, New York, just all within like an hour and a half, two hours from each other. It's great. And we got a Rosni Contreras. No problem, man. Congrats. I'll do a recap right now for you guys after if you guys missed anything. Pirates, that's Mike. All right, two more box. Come on, we gotta get some hits for some people that haven't got hits yet. This case hasn't been a little different from the last case. I know, dude, right? Exactly, great. All right, we got Boba Shat to $2.99. We have a AJ Alexi for Texas and a JT Dromuto to $25, little bronzing. Phillies, Phillies is Rob. All right, here we go. Last box mojo, guys. Oh, we got a thick one right here. Maybe we have no extra stuff. Thick one right after this. We have Joe Musgrove to $2.99 and a Tony Gwyn. Look at that, five of five. It's a button. Look at that button right there. Mr. Padre, tier one button uniform relic and game used memorabilia. That's very nice. Who has the Padres? I don't even know if I could put this in a top loader. Do you have a slider box? This is probably too big for a 180. All right, I'll put that in a slider box if it doesn't fit in a top loader. We have wisdom, or wisdom, sorry, not wisdom. Patrick Wisdom to 25 for the Cubs. And last one here is Eloi Heminis to $2.99 for the Chicago White Slaw, it's going to Brian. I'm sorry, Terry, for you and other people, unfortunately, that went hitless. But there is more cases we could always redeem ourselves. But this also is a promo, though. This is a promo break, so you still have a chance to win some money back, or not money, but some boxes back, including the grand prize of $10,000 worth of tier one, four master cases. But we did get some nice hits. Like I said, that Tony Gouin was very nice out of five. These are all the patches we'll top load right now. All right. Oh, actually, you know what? We did get you an Astro. Lance McCuller, it's just not an autograph, I should say. Still got something, though. We got that, Eloi Heminis, Rammuto, Alexi Contreras, Eloi, or sorry, not Eloi. Grandal, Baz, Jolito, DJ LeMayu, Carl Lewis. Vladdy was nice. Little Tatis, Devi Garcia. One-on-one Don Manley, there you go for bases loaded. That was a very nice one. Thompson, Elbis, Bozo, Carter, Forthes. Soda was very nice as well. And there you go, Kowar there at the bottom. So I appreciate it, guys. This is break number two.  So we're gonna go to the NSEC list. There was tier one number one, and this would be number two. Two number two. All right, so I appreciate it, guys. Jasby'scasebreaks.com, let's get some more tier one.


Hello everybody today. We are here again with near early from my profit about the full live Demonstration of how to calculate your profit on the right way in your ebay drop shipping business We will summarize here all of our previous videos together and we'll show you live How near we'll calculate all of his profits from his drop shipping business So enjoy the video under this video. You will have three links for our previous videos enjoy near and Hi guys, how are you? Great to be here again. Thank you Leo so before I will give the Full live demonstration. I would like to summarize the previous three videos So we talked about a Two major things the first one was a the green area I call it the positive flow of cash and we talked about the sources that money come from it was from PayPal The amounts from that we received from customers the second was from PayPal as well the return fees from PayPal and the third one was from credit card in cases of customer provide us When when the suppliers give us our money back in cases of refund after that we talk We talked about the red zone the red zone is the negative flow of cash negative flow of cash There were a five Major sources the first one was from credit card when we pay our Suppliers the second one was paper fees. The third one was eBay fees number four was a refund We saw it in PayPal and the five of the the five the last I called it The non-generic in ones the specific costs all the VA monitor software warehouse and things like that So now I will show you I created a very very Simple excel file. I will share it now and I will start the full demonstration So this is the excel files you can see The three green one is the positive flow of cash The the red one is the negative. So let's start with the first one. I will open my November 18 is the same example that we did in the previous videos and I will show you what data do We need to copy exactly from From the files to here. So I will share now and the PayPal Okay, this is the PayPal. I already marked the colors So I want to start with the first one the first one the first One is a column C the amount received a month received is the money that we got From our customers and the amount was I we can see it in green color in the green color 1560 euro. So we are going back Here and we will put one we write down one One hundred one thousand five hundred and sixty euro now we will go back to the excel file To the paper Okay, what do we see here? I want to continue with the green one in calling G It's the refounded fees from PayPal in case of refund from customer We see here five the number is 15 euro. We go back To the summary and we write down you see here PayPal return fees 15 euro Now let's go back to the PayPal Now let's go to the red ones. The first red one is fees. This is PayPal fees PayPal fees is 90 euro Let's write it down in the summary PayPal fees. Here it is PayPal fees 90 euro And the last data from the PayPal is Column F Column F is the amount refunded from customers. This is the total amount of 308 euro we go to the summary PayPal refund 308 euro Okay, now we have done. We have done working with the PayPal PayPal file now Let's go to the credit card file In the credit card file I will sort again the data as I explained in the previous videos and I will show you how to collect it right. So first let's Let's sort all the negative ones We will go to the filter in the amount and we will choose less than zero Less than zero if you remember the second step is to go to column C and A to Z it because we want to see if there's Nothing suspicious beside our suppliers. I will Try to find something that is not Aliexpress or Amazon Aliexpress or Amazon. I see that everything is okay Okay, so the sum of this amount of the refund is 594 euro. Let's go back to the summary to the green one 594 euro in this stage you can see we finished summarize all The positive one the positive one is the real The real a revenue of your month. It's 2169 euro now you can see we have three blanks missing after we will Finish adding them then we can get the total calculation So let's go back to the excel file of the credit card And now I want to see to sort all the positive one We will go to the amount and we will choose greater than zero and Again, we will A to Z column C. I See here again This one PayPal. This is the PayPal invoice. It's not one of my suppliers I delete it and then I see there's nothing else. It's suspicious. Okay, so after I Sum all the numbers together. I will write it down in the summary and the summary is Outcome cash. This is the net that I called it 950 euros This is a generic the in the generic. It's all the specific cost How much I pay my monitor VA and cetera if I have something like this I pay a sum of 95 euro and the last date Data that is missing is the eBay fees. I showed you in the previous one. I will show it again Just open. I will show one of my eBay Users we are going to account settings We just Choose the right invoice from the right side Let's see. Let's see where it is. Okay, November last November. It was 34 euro Let's write it down in the summary 34 you 34 so in this stage This step we can see we finish Marking and write down all the data in the red zone the negative one so the summary of the negative one is 1577 euro so all you have to do is take the green amount and To a minus the red one from it and the calculation is simple like that to 2169 euro less 1477 euro and the real total profit of this specific month eBay store is 692 euro, that's it You have to do it every first day of every month manually like this. There's no other way of doing it This is the only way that I know and Basically, that's it. Yo Well, I just want to let people know that we will add a link to this Excel file under this video So you will be able to download this Excel file to your computer and use it for the calculations Thank you very much for the explanations near. It was a great video. Thank you very much for being here with us every time and See on the next videos people don't forget to subscribe to our YouTube channel and leave a comment What do you think about this calculation method? If you want any other explanations or if you have any other questions Just lie down under this video and they promise that we will answer any of your questions See on the next videos. Bye. Bye. Thank you.


QuickBooks Desktop 2023, Memorized Profit and Laws P&L Income Statement Reports. Let's do it, we're then to its QuickBooks Desktop 2023. Support Accounting Instruction by clicking the link below giving you a free month membership to all of the content on our website broken out by category, further broken out by course each course then organized in a logical reasonable fashion making it much more easy to find what you need than can be done on a YouTube page. We also include added resources such as Excel practice problems PDF files and more like QuickBooks backup files when applicable. So once again click the link below for a free month membership to our website and all the content on it. Here we are in QuickBooks Desktop Sample Rock Castle construction practice file provided by QuickBooks going through the setup process we do every time maximizing the homepage to the gray area going to the view drop down noting we've got the hide icon bar and open windows list checked off open windows on the left hand side going to the reports drop down company and financial opening up the P&L profit and loss income statement range change from 010124 to 123124 January to December let's customize it fonts and numbers to change the font to 12 okay yes okay and then we're going to go to the reports drop down again and go to the company and financial this time take a look at the balance sheet standard changing the date but this time with the drop down this fiscal year and then customize to the fonts and numbers to change the font to 12 and say okay and then say yes and then say okay there's the setup process we've been doing every time our focus has been on the profit and loss so now we want to think about memorizing these reports we talked about how to memorize reports on the balance sheet reports so we won't go into as much detail on that process but now we want to apply the same system thinking about how can we possibly use the memorized reports so that we can systematically provide a set of reports to say clients on a monthly basis or a yearly basis you might have a similar kind of question if you're providing reports to say to a supervisor or something like that we might want to customize these reports possibly customizing them for internal use which might have different needs different customizations then for external use those presentation purposes for the end user the end customer to memorize the reports we can hit the memorize button here and we can memorize the report but we want to first consider the categories that we have set up which we have set up in prior presentations will review how to do that and then we'll come back here and start to memorize some reports so I'm going to close this out I'm going to go into the reports drop down let's go into the report center and within the report center I'm going to maximize it so it's not minimized again because it does that every time for some reason go to the memorized area and then we have our memorized reports now we set up the categories of the memorized reports meaning if you just open up like sample rock castle construction it might have different categories where you might not have any categories if you're starting your own file and practicing with them you could then go to the edit the memorized reports list here this is where we set up the categories you can remove categories if you would like and reports themselves by clicking on them right clicking and deleting and deleting so we went in through here before and deleted all of the reports and then put in these categories for our reports by going to the drop down rise up adding new groups and then memorizing the balance sheet reports within those groups and now we're focusing in on income statement reports our concept would be that we're thinking that we're going to provide reports to our clients possibly at the end of the month or the end of the year and trying to think about how we can format our reports so that at that time frame we can then just generate them possibly have someone else generate them even and format them and provide them to the client as quickly and easily as possible one way we might do that is to set up a category possibly month-end reports for example and then format each of our reports possibly numbering them so that we can then order them in some way shape or form possibly numbering them when we generate the reports so that when we attach them they'll be numbered or put them in a zipped file of they'll be numbered possibly putting them into an Excel file so we can put them in one PDF file in an organized fashion so that's what we will be practicing here and once we save the reports we're going to put them into our folder which we set up and we'll do this in future presentations the rock castle construction we might save it by client and then we will we will save these reports in the same place that we saved basically our balance sheet reports and go through that process so now we just want to memorize the income statement reports to do that I'm going to regenerate the income statement reports and I'm going to put all the income statement reports after the balance sheet reports in terms of our numbering system we're all are now on number seven so if I go to the profit and loss report then we might say what kind of income statement reports we want one starting point we might want to put out and say hey maybe I should just collapse all of these expanded items if I am using a lot of subcategories and you could do that easily by going up top here and say collapse the rows so that collapses all the rows and it makes a much shorter smaller report and that might be your summary report so maybe that's kind of like the first report that you might put together the easiest report to look at the idea being that we'll start with an easy report with less detail and then when they ask questions we can expand on that detail that's usually how you might want to be presenting things to say a client so we don't overwhelm them at the start and then we'll customize that report let's go to our normal customization and I'm going to say okay let's go to the header and footer let's call it an income statement so we'll get in the practice of calling it an income statement for external reporting purposes and we'll say this is a summary report so let's call it an income statement summary report and then I'm going to get rid of the date prepared time prepared report basis these three items I'm going to put my name in the footer and then I'm going to go to the fonts and we changed the font to 12 so we'll keep that I guess and we'll say parentheses around the red numbers and we'll get rid of the pennies take the sense off and we'll say okay so there we have it that looks good and then if that's the format we want I'm going to make the red numbers or the negative numbers read too by going to customize fonts and numbers we've been making the negative numbers read I knew I missed something I couldn't think of what it was but there it is now let's let's memorize it I'm going to memorize the reports I'm going to call it an income statement summary I'm going to call it report number 7 I believe is the one we were on I'm going to save it into a memorized category these are the categories that we have already set up we're going to save it into the client month and reports and there it is we're going to say okay and if I go into my memorized reports and I refresh it I can do that by closing it back out possibly going back into the report center this way and then into the memorized reports I'm going to show it by list and so now we've got one through seven there's the income statement so then we might do a full income statement so if I if I go into that one now if I was to open that one up here it's it already has everything collapsed right so then the next one I'm going to close that I'm going to go back into the income statement we put together and this time I'm going to expand everything so I'll say let's go here and we'll expand all the rows so now we have all the detail and we might save this one as like the income statement standard let's say so I'm just going to change the name to the income statement standard or something like that customize header footer income statement let's say standard income statements we'll say something like that and everything else looks good I'll say okay and then let's memorize that I'm going to make a new category this time because I changed it new category this is going to be number eight that we have income statement standard going to put it into not the payroll reports but the client month in reports and okay let's check it out by going to the report center I'm going to refresh it going to the prior tab current tab and now we've got number eight there's the income statement standard all right let's do some comparative reports if I go back on over here we could do some comparative reports so let's say we want like the four quarters of the report and then the total years so I could say let's go from January to December but then I'm going to I'm going to change it from totals to let's hit by quarter and then okay so now we've got the four quarters and then summed up in the total we would then want to adjust the the name here so I'm going to customize up top fonts and numbers we're going to call this income statement by by QT or quarter let's say let's just name it that for short for now for practice purposes we'll say okay so we might want this report which will give us that information on a quarter by quarter basis and then the total summing up all quarters I'll save that will memorize that I'm going to make a new one and this was number nine I think that we're on I'm going to save it to the category of the client month and reports okay let's check it out in the report center going back to the standard tab and then to the memorize tab so there's report number nine that we could have so notice all the myriad reports that we could have with regards to just an income statement and then the balance sheet report so you may not want all of these reports these are just some ideas that you can put together and how you might want to group month in reports that you can be providing to someone at the end of a period of month or a year to a supervisor or a client for example okay so we got the month in we could do the same thing for for the months I could break this out by month I won't do it but you could you could do that right you could say okay what if I broke it out by month for all those months now let's do a comparative report that is comparing the last two periods maybe the last two quarters or the last two months let's do the last two months so I could say okay let's go from the last month which is 1201 to 4 to 1231 and then look at the previous period and the change and the percent so I'm going to look at December and then the month before it which is November so this is our comparative report we saw how to do before so there it is that looks good so let's then change the name income statement month comparative report or something like that customize and we'll say fonts we'll say income statement month comparison comparative maybe and we'll say okay let's keep it at that so that's a kind of comparison that we might want to take a look at that we could do a similar one comparing it a quarter a quarter by quarter comparison which now shows the difference the current month of the prior month we could do the current quarter and the prior quarter the dollar change and the percent change and notice that this doesn't make sense too much up top the December December I could take that off so I could say I don't really need that that doesn't really possibly help as much so we could remove that okay let's memorize it I'm going to memorize and make a new one and we're going to say I think we're on number nine I think we're on and I'm going to save this into our month in reports and save it if I go to my reports and check it out prior and then this one scrolling down I've got two number nines so I totally did that on purpose so we can fix it here so in any case we're going to go to edit the memorized reports so I can edit it up top if I want to change the name of that memorized report and then I can delete it or I could try to rename it so I'm going to right click and I want to edit the memorized report so I could delete it and save it again if I want I'm going to go ahead and edit and I'm going to make it number 10 so that's number 10 and then okay so so now it moved it up here so I could then pull it down so I'm going to take this one and pull it down so I can organize it you know manually so I'm going to say okay let's hit the hit the diamond and that's going to return it to its original order and then I can go in and manually reorder it so I can pull this down to the bottom for example but let's first go back and do the last one more report before we do that so I'm going to close this back out I'm going to go back to the income statement and the other kind of report we might want to do is comparing the current month or the current quarter or whatever to the same period in the last year so December of 2024 compared to December of 2023 for example so we can customize the report to do that and I could say instead of this one I'm going to use the previous year percent change in dollar change and I'm going to say this is December that looks good so we'll say okay so now we've got December 24 compared to December of 2023 we'll change the name up top so we'll say header and footer statement statement month comparison to prior year I'm just going to say prior year so we probably want to think about the name a little bit more to if you're going to provide this to someone but I'm just giving some examples of some different kinds of reports that you may want to put together and note again you could also do prior quarter to prior to the prior year or the prior years quarter of the fourth quarter for example and so on and so forth so let's go ahead and memorize that one the last one to memorize let's say this will be the last one we'll stop it here this time I'm going to make a new one we're on number 11 now number 11 I'm going to save it to the client month in reports and okay so now if we go into our reports we've got them in there let's go to standard and then memorize so I can see where that one shows up there it is but is it in order it looks like it is in order if it were not in order then we could adjust it by going to the edit item here and I could then kind of adjust the orderings now sometimes it might make you re like refresh it and then adjust the orders but but once you once you have the capacity you can take these little icons I'm just going to make it one two and then three four five six seven eight nine ten so there we have it now we have them ordered here and in future periods so now every month we can go through this full thing with the balance sheet reports we did last time and then the income statement reports you can see we have a pretty large list of reports we may or may not want this many reports and we might have other reports we want to include which would be subsidiary reports reports that might be given more detail on some specific line items of these accounts like income by customer income by item what we sold we might have some more by client detail in terms of the accounts receivable by client that kind of stuff that we can put together we'll talk more about them in future presentations but the general idea being you can kind of group your reports memorize reports so that you can use reports internally use reports externally and possibly be able to group your reports as fast and easy as possible to then be able to attach them to an email zip them and attach them to an email or put them in a cloud file so that they can access it by one drive or one note or possibly use excel and a PDF printer to create one PDF file with all the stuff in it and have one attachment to an email or one item in a cloud drive or something like that we'll do that in future presentations


In this study, a new approach to recycle defective ceramics was explored by incorporating them into composites for use in the automotive industry. The resulting centered clay microparticles were mixed with an unsaturated polyester resin matrix and then cast into molds. The bending test results demonstrated an elastic linear behavior which is typical of a fragile material. Scanning electron microscopy revealed a good dispersion of the microparticles and a fragile failure mechanism. A finite element analysis was conducted to optimize the design of a motorcycle brake lever, demonstrating improved stress distribution and reduced mass compared to the original part. Finally, a prototype brake lever was manufactured using the optimized geometry. This study shows that defective ceramics can be successfully reused in composites for automotive applications. This article was authored by Jorge Antonio Velasco Para, Vladimir Aremón Valencia, and William Javier Mora Espinoza.