2.6. First, I'm named Kokubin and from ARM, which is I'm a grad. My company was previously treasured data, which is acquired by ARM this year, so I'm now from ARM. And I'm a Ruby cometer for maintaining EOB, template engine at first, and then now I'm maintaining the main, maintaining JIT compiler now. So you may not know JIT compiler, but last year I experimented an optimization idea called JIT compiler last year in April, and then immediately after the Ruby 2.5 release in last Christmas, I prepared a pre-request to merge a JIT compiler overnight and just merged in early this year. And this was good for having a much longer time for improving the JIT compiler for Ruby 2.6, and I've derived that for about 10 months after March, and then I'm currently a number two cometer of the Ruby 2.6. I've been working a lot for JIT compiler. Then I got a revised 2018 this year. Thank you. So this trade stock is all about the Ruby 2.6, and Ruby 2.6.0, it has a JIT section. So what's JIT? Do you know that? If you know that, raise your hands. Okay. You know a lot about JIT compiler. So it's an abbreviation just in time and plus compiler, but that's not explained so much. So let me explain the history of Ruby implementation. So historically, Ruby 1.8 was parsing the Ruby core to a tree called abstract syntax tree, like press for return a local variable and b in the right hand and traverse the tree each time. So it's slow if the tree becomes a lot complicated or longer or higher, but after that, Koichi introduced the Ruby virtual machine. It compares that tree to sequential instructions, and it's faster than traversing a tree, and so it's a little faster, no, no, so much faster. So it's a current implementation of the Ruby 2.5. After that, so I did this. In 2.6, it's compared to native code. It's specific to the machine, which is running the Ruby interpreter. So it's less complicated. No, no, no. It's less calculation. So if we use this virtual machine, it needs to integrate the get local instruction and do a lot of things to get a local variable a, but in this time on native code, it just rows the first argument of the method to register and then fetches the second argument to register and then press just run the add instruction. So it's a lot of faster than dispatching the press instruction. So this is a JIT compiler. But how can we use it? So it's that optimization is still experimental, so it's not enabled by default. You need to pass a dash dash JIT option to enable JIT compiler, or if you are not using the Ruby command, if you are using it as a command, you need to pass an environment variable that has a dash dash JIT. So let's use that. So then we benchmark that. There's a benchmark called opt-cat for aiming to achieve the Ruby 3 by 3, which is aiming to achieve the three times faster Ruby 3 than Ruby 2.0. And basically, this is a NES, a Famicom emulator that achieves 20 frames per second by the Ruby 2.0, and it becomes the 60 frames per second that's good in the, hopefully in Ruby 3. In the machine, my machine is a little strong, so it's not 20 FPS by second by default, but it becomes already 2.5 times faster in JIT compiler. Thank you. Yeah, and it's even faster than the previous 2.5, 1.8 times faster. So the three black bar graphs are just this year, and left six bars are six years. So this year, we had a lot of progress for Ruby interpreter's performance. So we already achieved Ruby 3 times 2.5, only remaining 0.5. But how about other benchmarks? It's just a Famicom, and I guess nobody is running the Famicom in the production. So I got bug reports, and it throws them in a reservation, and Sidekick is thrown in by JIT compiler. So what's happening there? So JIT makes things slower, so today's topic is all about the JIT compiler's performance characteristics. So there's a lot of tradeoffs, and you may want to care about that for 2.6 since it's still experimental. In the future, I want you not to care about that, but for now, we need to care about that since it's still experimental. So the first topic is about when does Ruby become slow in JIT compiler, like Rails or Sidekick, like that? Yeah, the first thing is that when there are many JIT methods, it becomes slow. So in previous, Sam's report was compiling 10,000 methods in Rails because Rails has a lot of methods, and the number of maximum compiler methods is just 1,000 by default. And I think it's even too... At first, I thought it's too small for Rails or no serious applications, but it turned out to be too bigger for computers. So why? So this is because the JIT compiler by C compiler and dynamic loading of the current implementation. So currently, this JIT implementation is called MJIT, which is invented by another person, but I'm maintaining that, but that invokes RSO. After Ruby 2.6, when dash dash JIT is parsed, our native thread called MJIT workers spawned in addition to the Ruby main thread that doesn't hold a GVL, so that can run parry, and that spawns a GCC process under the Ruby process. So after Ruby 2.6 and Upass JIT option, you see Ruby entries a GCC or a CC1 on some strange, dangerous compiler processes, but yeah, it's intentional, and yeah, it's running in such kind of architecture. And it uses the compiled binary. Compiled binary is placed in the disk, and Ruby interpreter rows the dynamically rows the object file to the memory by dynamic loading a function called DR open, and it has some limitations. So I'll explain the limitation of the DR open. So by using such dynamic loading feature, native function is loaded like this. So this is like two megabytes, it's a little bigger for computers, and so in the middle it has a very large unused space, and if the three methods are loaded, heap structure is like this. So when there are multiple methods, we need to seek very large unused space to roll multiple methods. So it takes time. So if we compile the 10,000 methods to this, 10,000 methods multiply two megabytes, very big. So it doesn't consume memory because it's unused space, but it needs to seek the very large distance. So it's slow. The second one is about when there are still methods to be jitted, it becomes slow. So it's strange. The first one was when there are many jitted methods, it's slow. And also there are still methods to be jitted, it's slow. So when it can it be fast. Yeah, very difficult problem. So there's a lot of results. And we, so I solved this problem in Ruby 2.6.0, preview 3, a little. So I introduced a technique called jit compaction. I just named that, but it's not a very normal name. And jit compaction does, it's performed only when it reaches the max cache, it's a number of methods, limit of number, jitted method. And it reaches the 1,000 method, it invokes the jit compaction. When it's invoked, this kind of loaded memories are compacted to this one, only one, two megabytes. So it doesn't have a large distance. So it improves the memory load overhead. But still, there's an unused space. And so it's not ideal, but it's much better than just wrote in multiple pages. So I'm solving this issue a little, but it's not so completely solved. So if we wrote 10,000 methods, it's still slow. Then another problem is CPU memory resources. So if we run a C compiler, of course it uses this CPU resources and CPU calculation on memory. So if your computer does not have a lot of spare resources, it pressures the Ruby's resources and Ruby becomes, could be slow. So if your computer is not so strong, like my benchmark computer, it can be slower than my computer. Also, when GCC or other C compilers are running, we sometimes work on GCC or Red PID since there could be a rest condition. So when GIT is running, if GCC is running at the same time, it causes a segmentation of fault. So we sometimes look at Ruby's main thread on GCC invocation or with PID, because with PID waits for process that is invoked by Ruby script, but also there's a process GCC created by Ruby process. So to identify which is created by Ruby script or MGT worker, we are tracking the PIDs of the chart processes. So, yeah, it also has a rock. So when there is a method to be GIT, it has a rocks and memory issue by the GIT compaction since it's not compacted yet. So the last one is when TracePoint is enabled. Have you ever heard about TracePoint? Oh, I think, yeah, yeah. So I think you may not know this. So TracePoint is a dynamic instrumentation feature. So it is used by-bag or yeah, by-bag uses dynamic instrumentation to step by step debugging. And also web console gem uses the bind EX gem that enables TracePoint by default and also coverage. So if you measure coverage of that test test, you may be already enabling that TracePoint. So there are four development and testing. So currently GIT is designed for production and also last year Koichi introduced optimization that is available only when TracePoint is disabled, even while he implemented that TracePoint. So he believes that TracePoint is not used even while he implemented that. But anyway, currently TracePoint is not supported for now. So if that TracePoint is enabled, GIT is also disabled for now. But in the future, it will be supported. So in summary, there are three situations that may make Ruby throw. And in OptiCard, it's a short benchmark. It's just run within only one minute or so. So it doesn't trigger GIT compaction because it doesn't reach the 1,000 methods. So it's pressured by there's three methods to be GIT. But in Wells' benchmark by some, he compared 10,000 methods. So there's a lot of pressures for that. So this is a matrix. But OptiCard could achieve the very good result because this downside is overcome by the benefit of the GIT compiler. So let's talk about the benefit. So what is made faster by GIT? So almost all methods are fast by GIT compiler. So this is a silver bird part of GIT. Yeah. This is because when Ruby virtual machine is running, computer has registers and instruction pointers that calculates for the Ruby interpreter and virtual machine itself. And Ruby virtual machine has a stack pointer and program counter. You don't need to remember that, but that's just pointers. And just pointers are calculated to calculate the GIT local instruction and send instructions like that. And GIT local is executed. A local variable is pushed to stack and stack pointers moved. So these two pointers are moved by the virtual machine. But when we move to GIT compiler, we don't need to use stack pointers and program counter. So stack pointer and program counter is pointers, and so we need to control memories. And so there's memory pressure. So we can eliminate the pressure by using the GIT compiler. And GIT compiler just moves the native instruction pointer and the registers. Then the second one is basic operators on core classes. So there are some methods that is specially optimized by virtual machine. So things like plus, minus, multiply, division, and less than, less than and equal for things like that. So such kind of basic operators are optimized by virtual machine. And since it's optimized, GIT can easily optimize that by using the method in learning. So actually, virtual machine can't inline the method for now, but GIT can easily do that if it's optimized. So then by inlining the put object one and put object two, GIT compiler can, GIT compiler or C compiler can calculate the result of the one plus two because these bars are inline in this three method. So it just returns a three. It doesn't have an odd instruction. So C compiler can inline such kind of inline instructions which is optimized by virtual machine. So these are, if your Ruby application are using these kind of methods, it's likely to be fast. And also the third one is covering Ruby method. Ruby method means a method written in Ruby. So I really talk, I read about the method dispatch of Ruby virtual machine. And if there's a script like this, foo.bar, but when prior to reaching this line, we don't know what's the class of the foo and what's the method implementation of the bar. So at first, we need to search method from the receiver, getting the receiver class and troubles in the class hierarchy and get the method definition from that tree. And after that, we got the method was, we found that's a foo.bar method and which is written in Ruby. And so we switched the implementation virtual machine to call virtual machine again because Ruby method is evaluated by the virtual machine. So there's some switches and especially method search is slow because it's so complicated and dynamic because Ruby is dynamic. And but for that, virtual machine already has a cache for that inline cache. So for each line, it has an inline cache. And if it has a cache, so when the line is evaluated at the second time, it verifies the cache. And if it redefined, it searches the method again, but if it's not redefined again, redefined, it just uses a cache value and invokes a Ruby method. And foo.js compiler, we just, we would be able to generate a call that knows the foo.bar is a Ruby method and that's a foo.bar. So it eliminates the many branches like switched to the typos method and other things inside the verify cache and between the verify cache and method entry. So it reduces the branches and so it doesn't rely on, doesn't need to rely on branch prediction or it just reduces the memory access for reading the method entry because as we know, that's foo.bar method entry. We can inline the pointer or address to the method entry in the generated code. So it's faster. Also, similar things can be applied to instance variable. So the last thing is this one and so for instance variable, when there's a code at foo, we don't know which address has the variable for the foo. And we need to search the address and index of the heap. So this function has many branches and complicated and it's slow but after we find the index, we can see the instance variable. But if it, in second time, we have an inline cache and if it has cache and it judges it's the same class or not and if it's the same class, it just uses index two again and it because of the instance variable. But this, in this case, if it's jit-compiled, we can know, we already know that index is two because we execute the jit-compiler just in time. So we know that's two and it's inline and so we don't need to read inline cache and so rest memory access and also searching index has a lot of branches and so c-compiler is confused by a lot of branches which is the main path and by reducing such complicated paths, the code is very much faster. So in summary, there are four types of situations that can be faster and opt-carrot is a strange program which is very instance variable specific. So opt-carrot accesses instance variable a lot of times and so opt-carrot has a very hot method called render pixel and it reads a lot of instance variables and the performance of opt-carrot is very rise on the performance of the instance variable. So Ruby 3 times 2.5 is achieved by instance variable access optimization. And so for Rails, I don't know how often it could use instance variable compared to opt-carrot but other authorization can be still used for Rails. But I guess basic operators on core classes are not so used often on Rails because hash is core classes but hash with indifferent access is not core class, of course. So if it's wrapped or and also string has active, suppose, safe buffer and things like that. And so there's if there's we are using a wrapper for it or using areas like blank. So Ruby virtual machine is tracking the redefinition of the empty method named empty but we are not tracking the method called blank. So when we call blank, even while it's against to blank, empty, it doesn't become fast. So Rails has some brokers for optimization and so it could not defeat the 10,000 methods. So I want to improve that. So I want to talk about the future of Ruby. So Ruby 2.6 will be fast only on the Famicom. So the future idea is that so this opt-carrot benchmark is not using any creating any object. So it doesn't test memory allocation performance. So for optimizing the object creation, we can use a technique called escape analysis which is just judges this object is not used by other methods or outside this method. So if it's not used outside this method, we can allocate the memory to just this frame. And if the object is allocated to the frame, we can allocate the memory by increasing just a stack pointer to increment and release the memory by decrementing the stack pointer. So it's faster than calling Morocco free and so it could be fast. But this year, Koichi introduced an optimization called transient heap which is for short object but it didn't have a lot of impact for Rails application. So this may not have impact either. So this is hard. But still, there are, I believe there are still rooms for improvements even in Ruby 2.6. So I think we can change the heuristics to trigger or compact JIT because so if there are, so on Rails application, we could JIT compile the method that is never used on production or just used for initialization. So it just pressures the memory and if we change the heuristics or logic or strategy to close or release the JIT method which is decided to be never used, we can change the strategy to improve the JIT compile performance. And also there's an optimization in GCC called profile guided optimization. By passing the option profile generate to GCC, we can generate a binary to profile a binary. And once we run the binary and we can, so the profile binary generates a profile result and if we pass the profile result to the GCC, GCC can generate a better binary which is like, so if there's a method function which is not used so often, we can move the memory to not so used area and it can be fast because very important code that can be in the similar place and memory pressure is reduced. So it's good for branch predictions as well. And also I already implemented the experiment method inlining. So by using the method inlining, 10,000 methods can be just some methods if the paths are not so many. So we can improve the memory pressure by using the method. We could be able to, we may be able to improve the memory pressure by method inlining. And this is already, I have patched for this and so this is possible in Ruby 2.6. And probably three of Ruby 2.6 is already released but the Ruby 2.6 release manager is my colleague and so I asked until when I can introduce a big change for 3.6 and he said November is fine. So we can, we'll reduce RC1 in the beginning of December so until that we could be able to do that. So prior to December, we want to experiment and improve that. So to improve that, we need benchmarks. So last week I tweeted this. I want Ruby benchmark which is made slower by passing JIT option and I got many tweets. Thank you but I'm a little sad about that but it's not so bad because it's not, Ruby 2.6 is not released yet. We have one month or two months to improve that. So it's very welcome at this moment, at this moment. Don't do this after Ruby 2.6 release. And if you are considering creating a benchmark for Ruby 3.3, I recommend benchmark driver gem which is created by me by Ruby Gunn project. So this is located in the benchmark driver slash benchmark driver and this is very, probably you can't read this but this is an example for sending progress to Ruby gem. This is, so this is similar to, this output is similar to benchmark IPS. Benchmark driver has a lot of problems to configure how to measure and also how to output. So this output is by default like a benchmark IPS and also the characteristics of the benchmark driver is that benchmark driver can compare the multiple implementation of Ruby's. So this is comparing, maybe you can't read this but this is comparing Ruby 2.6 versus 2.6 plus JIT versus JRuby and JRuby plus invoke dynamic. So this kind of thing can be done by benchmark driver but of course it's not so easy in benchmark IPS or other popular benchmark libraries. So I recommend using benchmark driver. Also if you use a benchmark driver, most of the benchmarks in Ruby bench is already using the benchmark driver and since it's progable, output prog for Ruby bench is already there. So if you write a benchmark in benchmark driver, your benchmark can be easily added to this Ruby bench. Also I have a lot of benchmark continuous benchmarks that this monitors each commit. So there is a lot of, maybe you can read this at all but that's many time methods. So time class has a lot of methods and many methods are monitored, performance of many methods are monitored. And this year I experienced performance regression by moving to Docker because it calls a system call that reads the et cetera time and I reported that to my colleague and my colleague optimized time zone related things in time. So it becomes faster and it's monitored. And once we hit some regression, we can find that by this. And also of course this is achieved by benchmark driver and so you can add some monitoring by using benchmark driver easily. So we still have eight minutes but this is conclusion. Ruby 2.6, Ruby 3 is too early, sorry, experimental but there's one option if we couldn't improve the Ruby 2.6, we can use JIT max gas smaller compared to 10,000 methods to reduce memory pressure or early JIT compaction and also we still have time to improve Ruby 2.6. So benchmarks are welcome. Thank you. Questions? Yeah, that was, so the question was is this kind of example are real assembly instruction and this is some sued by, but it's assembly instructions. Yeah, correct. So that's achieved by using a C compiler for specific architecture and that's assembles for specific architecture. And the input is a C compiler. I said no, no, no, C source. So I'm doing just generating C code and C compiler abstracts away the architectures. The question was some strategy to improve the performance by utilizing the JIT optimization, right? And are you asking the patterns to improve the performance with this JIT or okay, okay. So there could be some ideas to improve the performance on JIT like using local variable instead of instance variable since instance variable could be allocated on heap and it's slower than just using local variable or stack, but I don't want to introduce that because that kind of thing would annoy the users of Ruby. And so I want you not to care about that by improving the performance. So all I want is just a benchmark for your use case and I optimize that for you. Sorry, sorry. So the question was is there just switch for compile all or just nothing or do we have a method to compile a specific method? But yeah, previously in my previous experiment in code error RB, I implemented a method to specify a method to be jitted, but also it would complicate the usage. So I don't want users not to care about the JIT compiler basically. So I intentionally avoided to introduce that method. So it's over nothing for now. And in the future, it's all by default and optionally disabled by nothing. So the question was how much the overhead of the JIT compiler and so the bottleneck of JIT compiler is for optimizing the optimization of the C compiler. So when we, when image worker generates a native code, image worker transpires the Ruby's byte code to C code, but it's not so slow. So it's some sub milliseconds and it's then it's C compiler is spawned and C compiler has a lot of optimizations and it's the most so thing. And so it takes about 550 milliseconds in minimum and about, it could take 200 milliseconds in my machine, in my strong machine that it could be slower on some weak machines and yeah. So some 100 milliseconds is the bottleneck for the JIT compiler. So I fortunately, fortunately, previously in privacy, I introduced the support for Visual Studio in the previous three. So it could work on the Windows, even when Windows Visual Studio and also it supports the clang, LLVM clang. So it can use the optimization of the clang and so the question was is it specific only available for GCC? And it's currently Ruby supports the GCC and clang and Visual Studio. Visual Studio, I guess most of people are not using Visual Studio for Ruby, but there's a Ruby committer who loves Visual Studio a lot. I supported that for him. The question was is it really slowed by, become slow by this memory pressure or locking and so things explaining or actually generated code could be slower or not? And the answer is that could be slow, especially for exception, internal exceptions. So if there's a Ruby block and inside break statement is there, break escapes the block to Ruby method, so it uses the internal exception, but on virtual machine exception can be simulated very easily because all of the Ruby methods is calculated in the same virtual machine code, but in JIT compiler, we can't escape easily from the one block to outside the Ruby method. So it could be slow, but usually it's not the bottleneck for that. Yeah, so that's my intention of the last feature and I may be able to create some heuristics to avoid such stoneness generated code. So the question was am I using my many benchmarks other than OptiCaret, I guess, and sometimes I use a discourse benchmark which is a Rails application and which is my main motivation to improve that. And yeah, I'm also running the Rails application in production and so I want to improve Rails application performance. So I mainly use OptiCaret and discourse benchmark because discourse takes a lot of time to measure, so I cannot use a lot of benchmarks, but mainly if I, when I do optimization, I test OptiCaret and discourse benchmark mainly. I think it's time. Thank you.


Okay, so it looks like it's time to start so hi everyone I think I have to get started because since it's video recorded also I'm going online We have to start in exact time. So hi everyone. I'm Elena And the topic I'm gonna be talking today is a work presenting a work We have been doing for now for about one and a half years and you see a lot of people who have contributed here I just wanted to call everyone out everyone out who even contributed a ton a bit to this project And I have given another talk last year virtual talk at Linux security summit on the same topic But it was a year ago and we were pretty early in the I mean, it was the same project pretty early in establishing our kind of methodology and fuzzing setup and they didn't have much results yet and so on so so now It's really the time to go to present what like what advance you've done in the past year and also Basically what I want to go over is this is to remind okay. Why are we doing this project? So why is it needed? What are we trying to do? How do we try to approach this problem and then walk you through different aspects of a kernel? Which we believe will need hardening under this confidential cloud computing threat model and and I'm sure I don't have nearly Like full list here. So it's actually like one of the point of this talk is to also initiate the discussion to kind of To see and and kind of to make you as a clinic security community start thinking about things about this things if you're interested in the use case and threat model and and and and Basically start discussions together is okay. What else is missing? What else we have to be looking into and so on and I'm gonna show some of the results also to end So let's start. So why do we need to card and whether we need to harden Linux kernel for the confidential cloud computing guest? so traditionally the legacy VM scenario like Like you can see here if a trusted computing base of a guest, of course, it includes everything that runs inside the guest But then it also it had the full dependency on the host side So the hypervisor, I mean if you're talking about a Linux based host So the KVM and give me a new user space everything used to be trusted So the guest would literally like the KVM had full introspection into guest memory Registers it could get anything It wants and and and this has been a model for a kind of decade or so now when we think of the Confidential cloud computing and we start to have this protected VM guests the threat model changes So now the whole point of this confidential cloud computing technologies like Intel has Intel TDX MD has MD safe And I'm sure there is more coming. So the point is that now you don't want to trust the hypervisor So you want to make sure that are your guests. So you have this hardware technology. I mean, they're different They implemented in different ways. Actually doesn't matter how it's done But what it guarantees you it guarantees you that the guest is protected from hypervisor It guarantees memory protection protection of registers and so on and so forth and For example, I can I know I'm interlocked with architecture Better much better. So in Intel case, there's some special software It's called TDX module and it's it's a software module which kind of plays a role of the shim of the secure streaming between the VM guests protected VM guests and we untrusted hypervisor and it kind of makes sure that we The hypervisor can't really do a lot of bad things to the guest And I'm not gonna go into details how TDX works. So anything with that is this really separate topic But when you think of it, okay So this protected technologies give us this nice protection of this VM guest But why why would we need to think about carding? But unfortunately the problem is that even if we have it all protected and let's assume it all works absolutely perfect So let's not go to try to think like what can be problems of all these technologies They still need to be able to guess still needs to talk to the host So it still needs to talk to hypervisor for actually quite many things So we still have a lot of this Interruption primary if we have a lot of this kind of powered Operations happening so if an every guest would need to read an MSR or perform a Mami or Porta You or anything of that it will actually it needs to go and and and and then listen many of his nurse It needs to go and request this operation to be done by the host and in TDX specific case It's done with one hyper call which is TDX specific So this is just implementation detail. It doesn't matter how it's done. All I can on the high level, but we But at the end you you start to get this input So basically all now if this MSR CPU ID reads all of this With PCI config space reads, but you just kind of consume now suddenly becomes untrusted So it can be anything. It's a malicious input and same applies for the shared memory because Many of these are protected kind of I mean at least a current both I think I'm D and TDX to tap the shared memory between the Protected guests and the untrusted host and anything that meant shared memory Which is usually a lot of DMA going through is also unprotected. So it's untrusted It can have any inputs quotes cost a host in hypervisor and controlled And if you think of this this is in and if you start looking into the kernel source code This is a huge attack surface So I have some numbers for 511 while it's old numbers, but it doesn't actually change that much for current versions So it's it's it's a lot of kernel locations Basically where you would perform all this different kind of MMI reads or port IO reads a PCI config space reads most of it and drivers which is to some degree good because As I'll talk a bit later there is a way to limit what kind of drivers can run and guest But still even if you kind of drop the drivers There's still huge attack surface even in the core like if you talk about x86 in the core x86 architecture code And just core kernel code which performs a lot of these things which is now can be untrusted And when we looked into the code, it's it varies a lot of the complexity of handling these untrusted inputs Now it varies a lot. Sometimes it's just simple bit reads Masking like nothing happening where really nothing to worry about but sometimes the complexity is very It's very complex. There's a lot of struct parsing going on and things like that pointers being passed and And and and the main point here is that that code in the kernel This is the standard mainline code, but it has never been written in mind that that inputs can be malicious Maybe with exception of things like USB and stuff Which has been kind of for some level of fuzzing in things networking also network drivers Kind of all best taken into account, but not with low-level things We never thought that like you know an MSI read or you know PCI config space read can be now untrusted and now it really can from a point of view of the guest So and and a single bug if you have you know buffer of a flow out of index array and something which is exploitable It's it can actually render with Technology with fancy hardware technologies protected guest technologies that can render it useless because yes You have protected all the memory and registers and everything But you have just consumed this malicious input You've got an exploit and and you took over the guest and you're running inside the guest and you have all of it control You see unencrypted memory because you're inside and everything so that's why it's actually very important to think about this It's it's a really a model change It's a new threat model But we have to start thinking about if we want to run secure Linux kernel guests in our confidential cloud computing So what we have been doing about it is if we developed like this this approach how we of course like our focus has been for TDX But it's it's it's literally applies for any confidential cloud computing and for kernel We try to kind of do this iterative approach with Different steps. It's really iterative. It's not like step one do second third and we are done We kind of kept doing it in the loop So we try to minimize the amount of things we have active in a guest So of course we kind of minimized the drivers only enable like a small set of virtual drivers to be active and so on Disable all the subsystems. We don't need in virtual guests For example, there's a lot of subsystems We would never use or like on course and things like that Like minimizing open airports and so on so whatever we can we try to minimize Then what has to be enabled for you know for the guests to run and interact we have to basically we have to go Through the code auditing we have developed restatic analyzing tools for doing this and tracking this information Using smudge and I have had actually took last year as I said which goes in more details How we did kind of the methodology we have and I have the pointers that we enter our both Documentation and tooling which is all open-sourced both for audit and fuzzing and And and and yeah, and after the code audit or at the same time as a code It was a new to do fuzzing because of course code audit is even like static analyzer driven. It's very error-prone We might not find all the places we might miss humans might miss and make mistakes and so on So we actually developed an extensive fuzzing setup for for reaching all these locations And most of it is only active during the boot So it's actually not very easy to reach you can't use any existing fuzzing tool Like sciss-collar so anything because they're really not designed to insert Like it's really sciss-collar is more like way for the sciss-collar path and inserting all that way And now we have to basically reach all these millions of code locations like PCI config space reads and such you just cut it through code through the Kernel source code and we don't want to do Instrumentation on each of it because that would be way too much work and very intrusive for the kernel so we want to have like generic setup where the fuzzing setup can be injected, but This is like an overall thing have been doing and then what I want to go now is is really just to go through this distant aspects How how did they approach it so as I said we have this a bunch of things which Which the guest kernel can read or can obtain kind of we Can do reads for all of these things and the input and that case will come from the host We don't care because it's kind of the other way around and and there are all these things And I'm gonna go like just one by one. They're all in principle in our case down using the specific hyper call But it's just a transportation method. It's not it's not actually doing anything It doesn't look in if you think that if you could use that hyper call for do any checks You actually cannot at that point because you literally don't know anything about of this thing What is the type of thing that you're consuming so you can see yes It's a part of your read and here's your value But you don't have a context at that point the context only matters in that source code location They actually perform this part I read so so it's just a transportation method. It doesn't matter So if I start going through this one by one, so I mean for MSR's I mean for us They're kind of separated in two groups. We have a bunch of MSR's which are trusted So, you know, but like they are controlled in our case by TDX module Some of them simply are not allowed to kind of you can't read them in the guest because the features have been disabled There are some which are context-switched And they're also fine. So because you you want to go like the value would be provided to the guest upon the read natively and you'd want to do the soldier tour for asking that value from the host But then of course, we also have a bunch of untrusted ones And in our case the if you try to read this MSR by if a kernel tries to read this MSR from any point of code You will in the guest kernel you will get the special event with virtual exception insert in the guest kernel We have handler there and handle a look and say, oh, yeah, this was an MSR read Let's just ask the host because it's obviously like, you know We don't get it natively and this is where the host will input your MSR value And this is where it can provide a malicious input. So the MSR is actually pretty easy We are we have done the audit and fuzzing and because the handling of MSR is usually so simple It's like bit masking and stuff where we haven't been really seeing much of the or actually haven't been seeing any issues With regards to MSR reads We did see things like you have to always be careful because like they might enable certain features Which you might not want to have enabled in the guest But otherwise there's not like in the index out of bounds or anything with regards to MSR So but we we still fuzz them and kind of treat them Miss pay CP address are very similar again So we have trusted an untrusted case and and we have the same thing that we try to kind of disabled Here is as much as we can and this was kind of the first strategy But then our recently it has been changed that We don't even allow most of the CPU ID reads out of protected guest To the host so they only around the small range at the end which is Which is intended like the software of software for KVM things of the software Controls CPU IDs and that kind of shows most of our Problems for CPU IDs because this range is actually very little used in the core core or kernel code. So The portal is slightly more interesting. So I mean we don't support portfolio from user space So if that happens user space tries to read a portfolio from this this will be not allowed But then of course for all the kernel portfolio done in the kernel code What we did is that we have this small kind of a low list of ports Which we are allowed to read from be from the host or to minimize this part of attack surface minimization You have this example of just code snippets like what kind of port we leave open for this protected guest and And this portable filter applies for early and normal portal operations not no active and decompressed mode we kind of so far think that it's kind of reasonable compromise that we It's not very big fat in that mode We do out it and fuzz it regardless Of course taking into account that these things are only once which are opened again Porta is is is also pretty simple and And we haven't had any interesting findings here, which I guess many people could kind of think this in luck But this would be the case But but regardless you can have and because we also have to remember that all our findings are limited only to the core Kernel and set of drivers we enable so we are not gonna go and analyze absolutely all the drivers is a lot of drivers in a kernel So of course if you are running, I'll let's say if you will be running your Linux guest with some Drivers which is outside of virtio set and they would be doing some port a you and and you might need you might need to add this This portfolios into this filter least But you will also have to perform this kind of fuzzing rounds and things you can use our Methodologies to do that, but we haven't done it far between drivers. It's it's way too big kind of thing to cover So so every time I'm saying that we haven't made any findings It's always limited only to the subset of drivers and the core x86 code we tested So it's it's not of course like, you know all yes config of all the drivers and stuff So moving to MMI you are Similar situation to the port a very similar report to you but so we don't enable it from user space We have this patch which busy or the set of patches which are which makes the MMI you're kind of opt-in so so that you have to explicitly kind of Indicate that you want to share this MMI region with the host because by if we I think early on we had the Implementation was kind of the other way around that all of this MMI Automatically became shared and it is very dangerous because like you suddenly is now don't have any control of what kind of regions you're sharing with the host from the guest and And you don't know what is your talk surface now? It's all all opt-in is kind of not wrapped nicely for PCI devices So if your PCI devices authorized to have device filters are set if its devices authorized from this allow list It will automatically all of its MMI or mappings become shared But for these devices we already do the fuzzing and auditing and then we know We kind of we went through that attack surface and we know that it's okay So every time you're kind of similar here from MMI you every time you will add a new device to your allow list and things you will have to you will have to do the additional hardening and And basically follow the same process so but for the Virtio and for virtio drivers and for core kernel again We haven't had much of findings on the MMI yet But the next one is actually one which which has been the most one of the most Problematic so the PCI config space so so PCI config space. I don't know how many people know So it's pretty low-level kind of kernel code It's used to kind of obtain in in a lot of its in Device probing stages and things that gets used to obtain a lot of information about device configuration And it used to be like, you know, this is basically you really really hardware So nobody thought this being untrusted, but now all this his PCI config space is host controlled So of course can insert anything. We're kind of limited by only allowing this this traditional early this probing for CF-8 We don't allow this mcfg space type of PCI config space But still it's you can do you can do this PCI config space access which is going to be host controlled and And we have actually found quite some issues with even with the drivers We have enabled and in the core code. So I have example here, which is for 515 And this is the code from the radio or core This we've been modern map capabilities. It's it's an overtail one of overtail core code and And then you can see here it performs and this is very typical actually scenario for a lot of PCI config space So it performs bunch of PCI config space reads and now all these reads like a Highlighter just one the bar where it gets a bar while it always reads a host controlled essentially So it can be anything up to the value of the that variable now It's u8 variable and then and it's not known of course like case was presenting all those things Which like you can try to determine it compile time or it's of course nothing is known at compile time It's fully runtime and it's runtime only when it's like, you know If you actually start fuzzing and kind of reaching with the fuzzer But what what happens later if you ignore this addition for now Is that it will go and use this bar values to perform different calculation on this resource allocation? And it will use it as an index into the resource arrays And and and you can easily see and and and this resource array something like five or something like that So so you can easily have like out of bound accesses straight there and and for like simple fix like example for 515 is that you can add this checks now to basically control that your the bar value that you got now is Is is is basically have this very trivial kind of value check on on on before you start kind of passing it later and kind of Sometimes it's not even trivial because you can path this consumed and when you can do some operation It's in reassigning things and at the end all this host input propagates And when you can do some indexing and based in that value So it doesn't apply I think for 517 or 515 the code was changed was actually check already there. So this this slowly kind of gets fixed In the main line, but there are a lot of this kind of style issues and with especially with PCI config space with what we saw and are and we have the patch also which kind of I mean we do out it in fuzzing And this is how we found these things or for our methodology, but what we we find out that it's very difficult to Again, we have this such a future tax service. So we're trying to minimize it as much as we can So we have a patch have a link here Which tries to block this PCI config space access also for similar like for mmio from devices, which are not allowed But there is actually a bunch of problems with this patch So still have to figure out how to make it work correctly is currently make some issues and live with it And then some of very weird kind of side effects. We don't them yet sure what happens So it certainly works from security point of view blocks it But like the side effects are not quite currently kind of are very desirable. So So, yeah, so that's the PCI config space. So definitely problematic and are very actively used by a lot of drivers in probing stage So the I can imagine many drivers having kind of the similar issues because again from the driver perspective from that code has been written like the Person who wrote the driver never thought but lucky not can get malicious input from PCI config space read So it's this code has never been written with that in mind and the last the last one from this The list of different kind of types of inputs you can get is it's KVM specific KVM has its own hypercodes and CPU IDs And and this are kind of they were very kind of drastic about it because they all entrusted obviously call comes from KVM So we just disabled pretty much everything you can and left only the things whichever like trivially secure or Essentially trivially secure so we Kind of see if you need it to to think about it later if it functional is actually needed We haven't needed it for now, but it's definitely one of the things also to think about if you think of securing your your guests So moving on into shared memories. I said there's another big big area where you Typically what you call do you typically use your protected guest is that you set up a shared member because memory is all protected So it's private to the guest the host and hypervisor cannot seem to memory So if you want to do some DMA or anything you need to establish a shared pages Which is essentially kind of unprotected unencrypted or maybe encrypted under a different key, but it's accessible to the host so and and and and and this is also the And a lot of DMA happening as I said and all the vertio happening at least in our case It's all of the vertio is gonna happen over that shared memory pages So we have spent We have this list of virtual drivers which we enable you can see the list here And maybe we'll have some more in the future and we have to basically inspect for all of these core drivers and things We have to inspect how it handles all this kind of Data structures and things it keeps in the shared memory because again, it's all can be now entrusted and and we have false So vertio is complex kind of in many ways It has many modes and supports and kind of types and we have tried from one side We've tried again to disable as many as possible. We said, okay We don't need the virtual PCI legacy mode. We can disable your mama you and so on Support only this it has this different modes of types of weird cues and things it supports So we kind of disabled everything which is not we don't need and not commonly used and then we basically again We audited and and fast the rest for the drivers. We allow not for all there's more vertio drivers We just don't need them yet but then again here it's the catch is also to remember that we We do this for course for the for the core driver code and how the driver handles it But the driver to the end is going to obtain some payload data from the host And it's going to send it up for somewhere to process and that payload data is interested And it has to be verified separately if for a moral point of view it's considered application data You can protect it in different ways if you want like it's just like you know application data payload But we can't do anything about it. It's out of even colonel hands like after it's it's it's has been delivered In other aspect you had to think about is that of course everyone is usually like you know We want to run secure computations inside of this protected guest and so on and and and we need randomness everyone is like crypto needs randomness secure randomness and We don't want our randomness to be controllable or observable by the host now and and of course for Linux It's we have the Linux RNG which runs This is the primary source of cryptographic randomness that a lot of users base in kernel uses and its main It has a number of structure It has like a number of sources of interrupts But we if you don't enable any hardware support of things and so on the main source is going to be just just interrupts And interrupts are host observable now, so they can't really rely on this like It's it becomes kind of scary to think that now you're kind of your Entropy and and things inside of your guest this is can be controlled But at least it least observable by the host a predictable to some degree, so But fortunately if you think of deaf you random which is kind of gets an output out of this Charger 20 base here in G It has this option Implemented ages ago that you can you can indicate through the config random trust CPU You can say that you can you trust CPU hardware and the number generator Which present on different types of CPU Intel has it on this through the same early run ten er deseed instructions And then it will on every iteration of this every time you ask this random Bits out it will add it will kind of inject a fresh entropy From these instructions in our case on and and this is not a host controlled thing And then also it will do an early seeding of the generator with church a base there in G using this The third year on ten er deseed instruction again. This is not a host controlled So at least by forcing this forcing this options are basically forcing to trust this Hardware number generator. We have some way of arguing at least that we have some Independent source of entropy, which is not in the host control It's not like perfect because horses still will observe the interrupts and stuff But it's already better than just saying that lucky hosts can observe all our entropy at least to some degree and Then we also like when force things like looping and things like that to make sure that we We don't we always like we always require is to succeed and we don't go want to fall back to things which are untrusted Going further Another thing is the timers over on one secure time And we for us we have only limited trusted clock inside a protected guest it's based in TC and it's kind of limited So it's not full clock. It's lit has just properties like sync with synchronize and monotonic and There's no guarantees that it will match a real time But if you want to do it, you can kind of do your own setup inside a guest and use some Time server and then and perform using this local TC and so on you can kind of establish this process of getting the real time But what we need to make sure is that we need to disable the other clock sources Which which all kind of fall back one way or another going back to the house so the KVM clock obviously in KVM controlled and the ICP IPM H-pad and so on so so we really kind of we went and then disabled a bunch of these things to make sure that we kind of we always The only use it is see because this is this is something we have at least some Reasoning or we can have pride some guarantees about what what what the time is In other interesting kind of aspect I'd have to think about if you try to kind of think of a whole picture of protecting the secure guest is the ICP I So there are a lot of ICP I tables which virtual firmware will pass you like we passed to the guest to the kernel Apollo startup in our case TDVF is a virtual firmware is basically just a special kind of breed of We call it TDVF, but it's a special breed of EDK 2 and and and it will pass all these tables Luckily for us with tables a part of our remote attestation So you can't just pass a garbage tables and see what happens to your ICP I and I am my interpreter and your drivers And we do an additional thing of again We're going defining this allow list of tables which we even want to consider in the guest So I mean virtual firmware can try to pass as many but this is the only list We're gonna actually like allow it to be run To just minimize the attack surface, but even all of that said some of these tables are very complex and and and we have a lot of features and It's not clear like they probably don't even need all these features and It's not clear if like even if these tables that let's say attest it It's not clear if like how much of the attesting party would understand like, you know All these details where and and and you can kind of who's going to be developing these tables and you can have vulnerabilities and Interpreter and so on so this is like kind of an area We haven't really dived into the kind of we understand somewhat the risk there But we haven't had any bandwidth to go and start because I'm an interpreters complex There's a lot of things on all this ICP I kind of Table structures are complex enough and and we need to we need to be able to also in the future to think of how we want If we want to properly harden because I expect not not for all confidential computing Yes, we would have this setup that the ICP tables are at least like a tested or trusted to some degree But if especially if they are not and you can you know our input arbitrary garbage event this becomes actually a very big problem Ah Going next when when you have to think of things like what changes on things like interrupts and panics and For us like the only we in 46 to epic for interrupt handling We have some limitation. We have some additional limitations in what kind of non-nma NMI interrupts are allowed to be insert But still it's it's like the host has KVM and host has some control over what what what they can input where supposed to interrupt mechanisms so on and The bigger problem is also that all now IP eyes like between the pro different communication between the processors The virtual CPUs in our case because we're running visuals. They're all still host controlled fully So the KVM of hosts can just drop them. They can't like you will never think that they can deliver And and and this is what is used during panic So if you want if you and maybe let's say you you you reach some security issue and you want to be understand Now you have to panic or kernel the panic is kind of complex operation It's not atomic it issues with different IP eyes to different error in our case virtual CPUs to notify that this happened and the host can just drop them It's just they would never be delivered and and where is this it would be nice to make it kind of reliable We've had some discussions in it. It's not clear how to do it Like properly and and we don't really have any kind of concrete Case we can show we found where it could be a problem So we because denial of service for us is is not this outside of threat models So we host anyway is in control of kind of you know starting and stopping this protected guess We just can refuse to start it So we don't care about denial of service, but we haven't we try to think of where is the case We can think here where the consequences would be something worse than denial of service So but this is on the fact that we have kind of we're panicked in one CPU virtual CPU And the rest has never kind of keep going and we never got this notification If we can actually end up in security scenario where this is gonna be kind of we're gonna actually have a security problem So it feels bad. So that's why we want to try to think about it but then kind of ask put people kind of if people think of they'll find a way to kind of to see how we can we can make this reliable or Provide a kind of good use case For for judging why this this should be okay or not. Okay. Like like what not, okay? a Bit in the private memory management. So this is probably the only thing which is very 2dx specific even more like MD and also hundreds the private It has kind of similar somewhat similar concept where the memory also so for this protected guess the memory pages Which are private memory pages in a guest they have to be accepted before they can be used it's a security kind of property of the architecture and It's typically a lot of it is accepted by virtual form there when it starts building with kind of memory layout for the guest But not everything because it's very hard. It's very kind of performance It's not optimal to start accepting a lot of memory in advance So to some of it has to be accepted by the Linux colon in runtime And it's important to not to re-accept twice because what is going to happen in kind of you know by hardware is that if you Re-accept the memory page twice the page is going to be fine, but it's going to build zeroed.  It's a page. It had some secrets there I know your keys and and and you re-accept it You kind of you you you tricked your guest to re-accept a page And then suddenly the page would be wiped out and you're running your AS with zero keys It's not going to be very secure and you have no way to even detecting it as this from the process which uses it So so there's a logic implemented which kind of takes care of that And then we have also another angle is that we need to make sure because we constantly get this virtual exception events when Something is not behaving like like a shoot inside a guest kernel And and we have this crucial code sections like syscall path and stuff where if we get the this virtual exception It can be security problems So we're also trying to block this basically we have a couple of cases here It's not even maybe very interesting which will generate us which we can get So the host basically the KVM and or Malik Harperweiser and the host can trick the guest to get to access to do certain thing and and And then it will generate this point of the and you can try to time it when the guest isn't executing in particular place Let's say it's entering the syscall path and it's just about to switch. We it's still running like with the Kernel stack and and and you can have all kinds of bad things happening So we're also trying to kind of fix fix that one to make sure that we don't get these events But it's it's still kind of this kind of things you also have to think about but this is a bit more specific to this memory management of the of the hardware And then the last one last slide before going to results. So we also started to think Because everyone always thinks about this transit execution attacks So all these speculative side channels And and it's actually very complex topic and i'm not even fully qualified to kind of talk about it But we started to think that Our guests again like guests is run inside. We protected guests They do have we have responsibility for mitigating some of these attacks. It depends what the list is kind of depends on the On the type of attack on what mitigations are done by the hardware And in our case what is the rest is done by tdx module and so on. So it's kind of complex But there's one group which is spectra we won which you definitely know we have to protect There's no additional mitigations happening anywhere And there has been a lot of work and past done to protect to kind of study this attack surface between the user space Again between user space in a kernel and try to find all these gadgets and try to secure the kernel But now we actually have exactly the same problem. Just the attack surface is different So now we have to try to find respectory one gadgets not in the user space kernel path, but between the vmm the hypervisor and the And the guest and and what we did is that uh smudge the static analyzer It has a pattern an existing pattern has been for a while where it's called check specter You can use it to and it was a while As I said, you can use it to check the potential spectra we won gadgets between the user space in kernel And we have and it used uh It basically what it did is is used this internal smudge structures to track the We basically all the inputs you can get from user space and how it propagates through all of the kernel To the extent smudge can manage of course and then it would always ask Okay, is this index where kind of you know of a bound is this index belongs to Has been influenced by the user space input So now we've done the change. So we have now the similar basically tracking done for the host input So we have a way to ask okay Is this input is this variable and the kernel has ever been tainted with the host input So that is all this msr and so on and mmyos input And if it has been You can use this flag and you can run this spectra pattern It's already like a merge to smudge and everything you can say to it Okay, please just kind of tell me put this analyzed whose input on and it will show you potential Can you run it for a whole whole kernel? Source tree and it will show you potential gadgets for Essentially this new attack surface. There's a lot of gadgets there. I mean potential gadgets I'm not saying every one of them is a sexual problem. But we have to start Looking into it. Of course, there are false positives there Luckily most of the findings are again in drivers and we disabled most of the drivers So but there are also findings in the core code and this just started looking in it It's it's always so I kind of keep repeating this but it's it's it's a big attack surface Nobody has been looking before which has exactly same set of problems I mean they're very kind of similar Maybe there's a lot less going on of course like because there's no huge user space. It's still like, you know this hypervisor Guest interaction, but still it's a big attack surface and they have all kinds of problems We have worked for years From user space and now we have to kind of start thinking it on where the attack surface and it's not tdx specific or anything It's it's it's actually like it's it's linux specific for this threat model So let's see how much time I have so just to kind of very quickly This has been kind of overall hardening for the code that we actually have enabled we call it tdx guest code But again, this is just linux mainline. We kind of we we run it for most of the work we've done It's 515, but it's all transferable. So as I said, we use we use they do the round of audit We do runs of runs of auditing. We produce this Automatic list of findings all these code locations where this this input is consumed We run it for some filtering with the manual analysis We kind of classify this different with different kind of We have different rates So we we can make it suck some code looks safe and easy handling we can market a safe or some code looks concerning We can market a concern But all of this is just manual audit We need then we do the actual fuzzing setup and our fuzzing setup is pretty elaborate using this kfl fuzzer For most of the fuzzing we also use the kfx fuzzer, but it's it's usually it's just limited But we basically do the fuzzing runs on all of this Kind of all of this attack surface and then our normal fuzzers what we give you they'll give you a code coverage And and linux linux source code is huge. So if you just collect a code coverage and you get some number Let's say 40 percent It's not going to give you any information about how many of these actual points that we want to reach this ones Which we labeled let's say safe and concerned because we don't want to reach excluded code or Something which we trust we want to actually have precise information How much of that code we can reach and and we have also kind of this cross-referencing and and matching and Against this auditing results and at the end we actually get the coverage information The need so that we know that how much of that code of our interest we reached with our fuzzer And then we can start tuning fuzzers and and and get better at it Don't have time to go through this just to show that like this is an example of what we end up with The boot we have about 24 harnesses and we started to create also user mode harnesses and all kinds of different parts of the boot and and They depend like sometimes we have very big harnesses when we kind of okay, there's not too much things happening there of interest So we are care of big level harnesses for something which is like, you know, virtio and things we want to have much more precise harnesses to To make sure we can because there's a lot of things going on There's a lot of input consumed to the host so we want to kind of fast this faster So that's why how kind of harness this logic is done and we started to look on this user mode harnesses Which also kind of very important for us Some results since I have to wrap up. So this is for 515 Just to give you kind of an idea of what kind of things we're seeing so we we have like we have our Auditag distribution So a lot of code we likely can exclude and and this is only for again Most of the drivers are already dropped and non x86 code is dropped So this is i'm not even kind of including this into numbers But even of that core x86 code which is left and the drivers we enable We have this kind of distribution of attack How many concern kind of places we found and so on we have the fuzzing coverage This is already adjusted per this thing Above so how much of this safe and concerned code locations we can actually reach the fuzzers And it's not you can see it's not hundred percent, but we are working on it And then especially like for concern items sometimes this very corner case is how you need to reach this This particular location you need certain things to be enabled and so on We have found a bunch of back We have had so the audit basically this concern items means that we have found some issue in the code And then the we have found also some bunch of things fuzzing And and we actually like we're finding things much faster when we're able to and process them because This is that nobody has been looking into this before So we have we mostly trying to look into different Kazan warnings and things like that and we And and and very we have patches already and patches the public Some of them like visible numbers for 515 as I said some of these things already got fixed later in 517 or 515 So it might not apply But just an example like of of of things we had time to look into And many of these things like like any of these fuzzers or any of these back fuzzing Kind of Foundings you would need to kind of actually fix the problem to be able from fuzzer to reach deeper So it's it's it's kind of to some degree. We are filled with still scratching the surface and and We are and this is where I want to go to a discussion point because we I had like a bunch of people in first kind of slide saying people who have contributed in past and so on But at the end we're very small team now. We have I mean probably about four persons not even working on this full time So Going forward This is actually a big afford if we want to have this Attack surface secured From the point of view of the so if we want to linux kernel to be secure under this confidential cloud computing threat model So we really need to kind of collaborate with community and kind of establish some kind of a community Project or I don't know how to call it But like we need to really establish continuous fuzzing of this attack surface. We have the tools now. So which which Which work because like things existing tools and stuff we had to develop the The new tools because things like cscolar as I said and stuff. We're not applicable here We need to do a lot of boot time snapshotting and things like that for it to work efficiently and And and we really need to start doing it for mainline kernel because code is she being checked in all the time Every code being checked in could introduce this with problems People have a way should have a way to easily kind of find these things and there are probably even many more Hardening aspects that we even haven't had time to think about. So we just I repeat we just released kind of We're still Regardless of that who have been working on for a while the surface attack surface is so big that it's still like Very early on in the kind of you know, really trying to So kind of claim that we have this attack surface secured So what I'm hoping to do today in the afternoon. I think we're both sessions Would be so I would like to invite anyone who is interested to discuss about this to a both session for this confidential cloud computing securing guests so And I have a bunch of reference here. So as I said, we have a documentation public Follow methodology same things about all these different hardening aspects of the kernel It's all kind of written down. So it's easier not slides All our tools there's one repo, but it pulls like millions of different repos and pulls from the fuzzers and things So but it's kind of nice setup. You can try it doesn't require any hardware You can try it. We have a minimal emulation setup for our like tdx case So you can run it in any hardware you have essentially more than more than enough Like not super old system, but it requires some pin and stuff And then we have the tdx case kernel is just where we have security patches and stuff which haven't been sent to mainline yet So, yeah, so that's I guess all I think we're out of time somewhat, right? So maybe if you we have a both session as I said that we end and you can catch me I think on the brakes and things like that. So thank you


What's good y'all Chris from jassescasebreaks.com doing our 2021 upper deck skybox medal universe champions one box break number 19 Random first name first letter. So the way it's going to work here We have our list of letters from a all the way down to z plus other letters Every get a LeBron James first name LeBron first letter L go to all the L spots Tiger Woods T goes to Tiger So that's how it's gonna work. We have a couple randomizers to do first We're gonna do our early bird spots the first 10 people will get the last four spots So our fresh dice so Michael a Michael Allen and Carl grabbing the last 10 spots total in the spring to get it done Thanks again y'all Carl big shouts out. We'll do our first dice roll for this our only bird spots. Good luck three and two five five times for the early bird one two three four and last one five top four names only is Gonna be Michael a Carl Carl and Carl. Thanks. You too. Congrats still in the break. We will add you now To our early birds to our now a full list This is now our full complete break all of our friends Here's our fresh dice and the letters From spot a to z plus all other letters this dice roll you for both of those. Good luck. Thanks again y'all Four and five nine times for both. Let's do it one two three four five six seven eight and nine ninth and last time Carl Carl and Michael those first three spots from the early bird Every and nine times for the legs. Here we go one two three four five six seven eight and last one nine nine times so P Spot P down to spot s So we'll show it off here So Carl will grab spots P and R Michael with that Z plus all other letters Carl with W And C Michael a with N Carl with B J D and M Michael Lucia with T Carl with a Lk Michael Lucia with E plus G Carl plus H plus V and S We'll do quick chill quick little trade discussion if anybody is interested. We will get to it We will pause the video when we come back. We will pick One of our six boxes here with our dice and get to breaking. Thanks again y'all be right back All right, we are back no trade discussion as Carl said let's bust it Bustin makes me feel good. I'm gonna make sure Keep doing this Yep Every time every time every time There we go. That's better So go F and F so that's being printed out Here are our six boxes one two three four five six We will use our dice Corresponding dice. We'll pick the box. There you go. Good luck Come on dice banger here It'll be two So one two We will upload the other this was number what? 19 Man We will upload 20 on the site after this here is random a letter first letter first name number 19 Thanks again y'all. Let's do it There's a quick one is an easy one one box break. Good luck to everyone involved Carl Thanks again. Shouts out to you for giving us break Busted get off me Plastic and here we go Three stacks of pack one box Skybox number 19. Thanks again y'all. Good luck. Well, that's Scottie Lewis fresh foundation Allie rushman I want a nice rookie season Fresh foundation sky or sky brown Justin Fields on the arc weld Rookie for the Bears spot J. We'll go to Carl J for Justin. That's the brawn there Joey Kim fresh foundation Corey Kisbert refractor Jalen Johnson another spot for J another one for Carl in the furnace last furnace Nice died cut be J Boston reaching for the stars Spot B going to Carl. Very nice number, but The die cuts one per box of Justin Fields.  I We do get a jumble. I Can Griffey junior Spot K Carl. Nice one man congrats. I haven't seen one of these actually Oh, these are nice Jambalaya Look at the background moving around that's a sweet one nice Carl congrats. Thanks again all these little Galaxy by ourselves to Coco Gospotsy Carl Triple Lawrence blast furnace Omen Diaz Fractor W Carl and the last little stack here. Thanks again y'all. Good luck So if we can grab one more somewhere intimidation nation sack well should spot Z all the letters going to Michael Olarcio That's Isaiah Todd purple 46 out of 75 That's gonna go to Z plus all other letters Michael Olarcio working for the Wizards, I believe purple 46 of 75 very nice Got a Francisco Molinari and a Danny will it nice nice break y'all It's a jambalaya can Griffey junior going to Carl Handsome card right there young young Griffey to That cut a brand of boss junior Justin Fields are quelled and the 46 of 79 purple Isaiah Todd to Michael a C++ all other layers nice quick and easy. Thanks again y'all. We will put can't believe it number 20 of Skybox metal on the site at jaspys case breaks calm. Thanks again. Bye. Bye


the pigeons showed almost perfect transfer of stimulus control from the red green lights to the vertical horizontal lines that is they omitted oh ah chapter 21 provides a detailed discussion of modeling imitation and observational physical guidance is a response prompt applied most often with young children learners with severe disabilities and older adults experiencing physical limitations


Go ahead, Peter. Thanks, Jen. Following up on Ouija's question, a lot of focus on the economic pain in Russia, potentially, from these sanctions. But what about the economic pain here? The Russians are saying they think gas prices in Europe are going to double. How high could they get here? Well, I think, again, Peter, as I said to Ouija, I mean, some of this depends on what President Putin does. So as he's suggesting what the impact will be around the world, it's all based on what his actions are, just to be very clear about it. But even without all this going on, gas in California is almost $5 a gallon. I think, as you heard the President say last week, standing up for our values is not without cost. What we're trying to do is minimize that cost. So I don't have a prediction of it right now because we're trying to minimize the impact. Let me tell you right now, the only people that are going to be heard by this plan are Americans and Europeans, who clearly thinks he can ride this out longer than us. And do you honestly think he's wrong? We're already dealing with Biden's inflation, much less $8 a gallon gas.


Well, Friday is finally here and so is the Bemidji Youth Baseball Tournament over the next three days. Four-age divisions will flood the diamonds all over the city. Day one for the 14 U-Squad. They're playing over at BSU. Their first game against ESCO in northern heat. Stars strong at the plate. Keelan Downs gets one past the shortstop here. Run scores. We're tied at one. Later in the inning, Grant DeKluzen goes oppo to ride. That's going to play two more for a single. Five-spot in the first and more in the second. Nolan Rutledge is going to find some green. Past the second baseman. Another run would cross the plate. Northern heat, they win in their opener. 7-2, they continue it tomorrow. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.


Hi, it's Bridget. Welcome to Above Life Channel. The purpose here is to inspire your spirit and to fill you with hope. Today I have kind of a unique video. I'm going to share with you a historic location I recently was able to visit. I was actually on vacation and I thought, hmm, I can't be at this hotel without giving you some video of it. And it's not in my usual genre of pick me up positivity, but we are going to definitely focus on the positive and the good in this particular historic video. Alright, and I am going to channel this person. So we're going to talk just briefly, hopefully briefly, with former president Richard Nixon.  This is a very current events, very historic time in the United States and related to current events that are going on right now in the US. It just so happened I was actually at the hotel in Walt Disney World Resort in Orlando, Florida, the contemporary resort, right at the ballroom of Americas, where I actually stayed at that hotel, where Richard Nixon gave his famous speech, I am not a crook, that whole speech. And I want to show you the footage of that. So I will I'll show you the footage of the area of that ballroom so you can see what it looks like today here. And we are going to talk now I was going to try to talk with him there, but I just couldn't do it. It was too, it seems so weird, you guys, like I'm in this most magical, happiest place on earth, you know, Disney World with creativity and and positive energy and all sorts of stuff. And that's like imagination. You can do anything, anything's possible dreams. And then totally with this energy of former president Richard Nixon, like resigning today, I imagine Disney World. And I'm at the contemporary resort. And of course, there's a very historic place here that we are going to visit. I'm going to share with you and I don't know that it's going to inspire your spirit, but it's historic. Okay. So we are going to check out the ballroom of the Americas. The ballroom of Americas is famous for something in history. Richard Nixon, he gave his famous I am not a crook speech here at the contemporary resort in Orlando, Florida. I don't know why I'm whispering. Here it is. Wow. This is the ballroom that Nixon gave his famous speech in. Isn't it awesome? How beautiful. Look at the ceiling tiles. That gorgeous. Look at that. Look at the decor in here. Isn't it awesome? Look at the chairs and the setup. Isn't it just beautiful? Wow. Impressive. All right, guys. Wow. Awesome. Awesome. So I am going to talk briefly with former president Richard Milhouse Nixon, not to get into drama, not to create chaos, but to give us some insight, you guys, because there's some stuff going on right now in the U.S. as I've mentioned, and it will be interesting to connect with Mr. Nixon and see what kind of energy he can share some information or messages he can share with us. So let's bring him in. Yeah, he does really have like this. Oh, I don't want to be rude or impolite, but this grumpy old man face, like really stern kind of waxy. His physical appearance looks very tired and very worn and stressed. It looks like he's like a heavy duty smoker is what he looks like. You know, he just looks so much older than his years. And understandably, I mean, you can understand why that might be, right? Because of the circumstances and drama of his political career and the way he left office. So I don't mean to be rude, you guys, but you know, I really describe things because I'm clairvoyant, so I see the spirit and he's showing up as he did at that time. All right. And now he brings me into like this room. It looks like it might be the oval office. It's this big, beautiful room like White House looking room with big, long curtains, huge room. So it looks like the oval office or an office that he used in the White House. He actually is telling me he just told me the state office building or the federal office building or something. There's another building that has this that he office in is what he's telling me. I don't know what that's about. I don't know if they're in Congress or on the Capitol Hill. There's another office for the president to use. There must be. Please be showing me that. And then it's like long curtains and there's like a blue, a lighter blue, thick, heavy curtain. And there's a gold. So it's blue and gold with it, but it's not a bright blue. It's a light blue, like a sky blue. That's important because color really matters a lot to me when I'm channeling because the gold is usually connected to our spirit and our intuition, like what we sense and know and what our spiritual purpose is and our life purpose also could be connected with that golden yellow color. And like being on path, this is on purpose. And then that blue energy is very much a truth energy. What is true, which is kind of interesting, isn't it? Because that's where this whole like the troubles that that former president Richard Nixon had was all around the blue is energy of truth, which is very appropriate here. So Mr. Nixon. And then he shows me like something he's proud of like China, working with China, trade, business. He does, there are definitely some patterns or theming between him and the current president of the United States, Donald Trump. And with that, like a business mentality or a business acumen and trade, he's talking about trade and trade efforts. It's a big deal for him. Mr. Nixon and he's drawing that comparison to the current president, the sitting president. He says the sitting president. Interesting. He uses the word the sitting president. Okay. And at the time I'm doing this video, it's Donald Trump is the president of the US. Okay. And this video is recorded on December 15. I should share that 2019. Okay. So so is it true that there were how should I say this? There was a lot of question about your mental state and your mental health and your clarity in the later part of your presidency. Mr. Nixon, is that something that you would be open to speaking about, especially because mental health is a thing, a big deal now, trying to get us as a society to be more accepting and compassionate toward other people, other people, because it's not really about our own selves. It's about everybody else and their labels and issues, right? Yeah. No, really it's about all of us. Mental health is something as important as our physical health and wellness. And the society is kind of changing to become a culture of we're hoping for more acceptance. I think most people more kindness and understanding about that and cutting slack to each other and ourselves so that people will ask for help instead of suffer in silence. Okay. That's my soap box thing. But will you talk about that? And he says, Yes, I will. Yes, I will. Yes, I will.  It looks like a Scotch or Whiskates a darker colored liquid. He's drinking a lot. And he's showing you like tremors in his hands like he's shaking. He says, I wasn't well. I wasn't well. He's not suggesting that you guys as an excuse or anything to the actions or what may have taken place. But he's just saying I wasn't well. I wasn't well. I wasn't well. A lot of stress, you guys, a lot of stress, anxiety, tension, a lot going on. It looks like other people in his own party are having conversations with him to encourage him to make the decision to leave office that that would be in the best interest of everyone is how kind of he was getting a lot of pressure is what he's showing me. He doesn't want to talk about Watergate though. He doesn't want to talk about the scandal. He doesn't want to take accountability. He doesn't want to defend himself. He doesn't want to talk about it. It's almost like he just he does not want to even doesn't want to even stir the pot is the words that the phrase that comes through stir the pot because there's not it's not he says it's not the same as what's happening in your America today. It's different. It's very different. He says times are different. Times are different. There's a lot that has changed in the US and it's a different time. It's a different time. It's not really fair to make and it's not even adequate or it's not an adequate comparison to make and I'm sure there are many parallels being drawn and discussed in media and in politics and and yet so many of those arguments are those things that you're seeing played out now live on television happened. It happened before as well time and time again the same kind of cat and mouse game occurs. The difference is it's behind closed doors behind closed doors and it really is like a game like a show and there was always dirty laundry. There's always dirty laundry in Washington. There is don't kid. Do not kid yourself. He said do not kid yourself. It's always been there. It's now that you're seeing it and I'm thinking you don't want to see it. You don't want to see it. Then don't look. Don't be looking if you don't want to know the truth. Don't look. Don't look if you can't handle it. If you can't handle knowing that it makes you feel pressure like you have to do something about it doesn't it. It has a really put pressure. See you see you see most of America they just want to be left alone. They want to be left alone to live their own lives and then every four years go to the polls and make a vote and the truth is is they don't really care in the long-term future of the democracy it's just all it's all just for show it's not there's not really it's a cycle a pattern of behaviors it's not something that idealism doesn't really thrive in the political arena it does not it cannot in fact it cannot survive it it can't anyone you show me any politician or any any possible possible candidate and I will poke holes all over their idealism their dreaming mentality and attitude will not survive it will not they will become hardened they will become cynical they will they will understand really fast baptism by fire that in order to be successful at all they have to play by the rules that are already existing and those current structures are the very things that you are frustrated about that you have felt and known for years but care to turn the blind eye to so you would think you would have learned something from my time but you clearly have not in fact you're a glutton for punishment and want to know more more more more why why is that is that because you want to feel better about yourselves and your own miserable life and compare and point fingers and blame yes it's because everyone in America needs a scapegoat needs someone else to blame for your own unhappiness and the truth of the matter is is it's your own self loathing that creates the the need to devalue others and the lack of the ability to reclaim or to lead through all of this smoke and mirrors is not an easy thing it is a difficult thing and it's not for the faint of heart and that's why most Americans will ignore this they will choose not to have any kind of responsibility or accountability and that is the word of the masses it is it is it is that's the popular opinion okay i did not expect any of that you guys okay he needs to stop we need to stop now it's not that he's angry or upset or an evil person it doesn't feel like but he has a lot of very strong he's a strong personality mr. Nixon does and his energy feels very much um on point as like a dominant fee figure he doesn't feel reincarnated he feels like he finally has an opportunity to say see kind of like see i told you so this is the way people are this was happening all along and nobody notices or it happens in in different to different scales in in anything in business and government and healthcare and politics it happens everywhere but when it's on the big stage and the big light is being cast on it then it becomes a performance and both sides get everything gets heightened and it and it distorts the image of what's what's truly happening it distracts from the individual voters or Americans responsibility and puts blame and focus outward instead of acceptance inward and that's where the power is and so that's where the contrast is it's kind of coming from mr. Nixon sharing this insight is kind of to push us on the edges and i do agree with some of the things that he's sharing in the perspective that he's sharing it and it's again his personality is very bold very dominant but he's not trying to control he's simply almost it's almost like karma as to what it feels like where he's coming back to say i guess i wasn't so bad was i huh you know kind of a thing like see we've always been in america where there are things that are best left behind closed doors and desperate times call for desperate measures or behaviors you behave in a way when you're feeling threatened or you're afraid and there's fear in ways that you wouldn't normally do it's irrational he says it's a rational choices that are made and then there's all these cover-ups that go on and there isn't despite what you may think it's conspiracy theory is what he's saying there's no conspiracy theory here it's just what is a truth and the fact is in the matter of public opinion there is no such thing it doesn't exist this is Richard Nixon it does not exist truth does not exist it's simply a matter of popular opinion the majority rule and when the majority changes the rules change the environment changes everything changes then it happens over and over and over again so if you're looking for stability forget it you're not going to find it out here you can't find it in a structure or system even a democracy the only place you can find that is in yourself and that's where you need to find your own peace you need to be able to live with yourself at the end of the day and can you do that if you can't then you need to step down and that's what i did that's exactly what i did that has more integrity than any of the truths that you claim today oh okay so i'm not sure you guys how this uh channeling video with the former president Richard Nixon is gonna like land for people and i really don't want people to be fighting in the comments but i know you guys hmm at above life channel i can do my best to create an environment that is positive focused on inspiring your spirit and giving you some hope but i also really value authenticity and so i give it to you right here in this context this isn't about politics it's about people and it's about you you as a spirit being a human you as a spirit being a human not your mind not your anger not your your limited beliefs but about your spirit understanding how does your spirit understand this message what is in this for you as a soul as a gem of wisdom go ahead and put your gems of wisdom in the comments below now i know there will be some political banter ignore it please this is not a political channel this is a people channel this is for your spirit so this is frigate thank you so much for watching this channeling session and remember as much as my job is to inspire your spirit and that's my purpose your purpose is to live your life because this after all is your life so live it just live it and ignore the comments it's not worth it thanks for watching you guys


Trade it and upgrade it now at Appalachian Wireless and receive up to $700 off when you trade your old eligible device in and upgrade to a new iPhone 14 Pro Max or Samsung S23 Ultra on an unlimited Appalachian advantage plan because we are you. We are Appalachian Wireless. The Mountain Girl experience unveiled their new logo and announced this year's date for the festival on Saturday, February 18th. So every year we try to celebrate a different instrument, a different genre of music and just a different outlook on the Mountain Girl. So this year we have brought the rock and roll and we're so excited to unveil our new Mountain Girl logo. Woohoo! This year's festival will take place June 15th through the 17th with June 15th being their first ever bluegrass night. Another big change for this year's festival is that it is entirely free for anyone to attend. So we hope that the community will, you know, want to partake. Remember, Mountain Girl Experience, our mission is celebrating the artistry and creativity of Appalachian women. But this is an event for everyone. We have all different genres of music. All genders are welcome. All people are welcome. You don't even have to be Appalachian. That's right. For more information and updates on the coming Mountain Girl Experience festival, you can visit the Mountain Girl Experience Facebook page. For Mountain Top News, I'm Kelsey Dean.