Good morning everybody. Thanks for showing up. I think this is actually the biggest stage I ever got for presentation much appreciated So let me let me quickly introduce myself My name is Jan Eidenberg. I'm a trainer and project manager at linotronics, which is a Service provider for doing embedded linux development. Well, you yeah, you might have heard of us So we do a lot of mainline development. For example, we we've got the x86 maintainer at linotronics We we do a lot of different Mainline development and actually one thing we do is on behalf of linux foundation We're trying to get the real-time capabilities of linux into mainline and that's basically The topic of my talk today. I would like to give you a brief introduction on using real-time systems with linux or how to make linux real-time so This is not gonna be a very technical talk This is basically a overview of what you could do what's possible what not and giving you some historical details about real-time in linux, so I Will talk about basically About the definition of real-time so first of all before I start to look into linux and real-time We need to get an idea basically. What's the definition of real-time and who needs real-time systems? This is how we we're gonna start it today Well, then we will look into linux and real-time you will see we have several approaches on on making linux real-time We have several different approaches like we have real-time systems with linux around for more than 15 years right now So we have a lot of different historical approaches on how to make linux real-time So I will show you the most important ones and then the ones which are still well developed today in which I used in the industry We will also look into Yeah, the results and the latencies you could basically achieve with a real-time linux on a specific platform So basically We will look into a cortex a9 dual core system and see which latencies could be achieved with different real-time linux approaches So this is basically what I'm gonna talk today about So now let's get started Now let's quickly rethink what real-time is about and and for me this this definition is pretty important because still when we do Presentations or trainings on this we can see that there's still a lot of confusion about the definition of real-time So basically one definition we get really often is well real-time is it's just about fast responsiveness It's fast execution Well, I don't know is anyone in here who would agree with that No one yeah your experts already So basically yeah, this is not the definition of real-time right, but but you get this definition very often real-time is about fast execution it is not Well another definition we get quite often is well, it's about performance Which is also not true right, but well We have to admit that World real-time might be a bit confusing Also, it's not fast execution. It's not performance But well if we would go outside the hotel and ask people on the road what's real-time they would tell us well Maybe it's the real-time Just the time of the day Basically, well we have a clock real-time in Linux right, but that's the time of the day But this is also not what we are going to talk about today So real-time is kind of confusing so that's this is why it's that important to get a clear definition Of what we're going to talk about today and and to get it in the correct context with Linux So it's not about it's not about fast responsiveness. It's not about performance Basically, it's all about Determinism so when we talk about real-time we talk about timing guarantees So you need to execute something in a specific time frame and you need a guarantee for that So basically the definition would not be as fast as possible It would be as fast as specified and this is the thing which is important about real-time So well if 30 seconds is enough for you. Well, it's okay It's your definition of real-time, but you need a guarantee to meet this So this is really important to get this point when we talk about real-time. It's all about determinism So when I talk about Linux and real-time today, I'm gonna talk about Deterministic timing behavior with Linux This is basically the idea of this talk So well, could you help me out here? With a second Let's try that again Just let me know if it's making trouble again So well, yeah, sorry for that Okay, thanks. Thanks for telling me So once again, we've been talking about determinism so Now when we start about talking about real-time Linux the next question would be who needs real-time Linux so Basically when we talk about real-time We have to think that the correctness of your algorithm you're trying to to calculate It's not just based on the correctness of your algorithm. It's also based on the correct execution time Which means if you don't do The calculation within the correct time frame This will lead to an error condition So this error condition in the real-world example might be The product you're trying to manufacture is broken Your machine will break or something so missing the time slot will lead to some error condition And actually the worst case situation would be if you miss the time frame actually someone would get hurt, right? So if you get into that situation that the machine gets broken or your product or someone gets hurt This is basically the moment where you need to think about real-time requirements So Now one last point on the definition of real-time and This is something we here really often when it comes to Linux and real-time We get some word like soft real-time so I Don't want to get Really deep into that topic, but please please forget about this word So we don't talk about soft real-time. Let me give an example like if your wife would tell you she's kind of pregnant Would you believe that no you wouldn't right so well she can be pregnant or not so same for real-time Let me give you another example like if you think about the use case and missing the time slot will lead to an error condition So basically I'm interested in not getting hurt Well not getting hurt for most of the times wouldn't be acceptable for me So basically that's the same for real-time. So you can be deterministic or not, but there's nothing in between, right? So please forget about this soft real-time word. It's it's just crap, right? So we can be real-time or not. So When I'm talking about real-time today, we talk about hard real-time and deterministic timing behavior. That's it so well when you get Into a new technology The first thing you do is you're trying to check out who else is using this at this very moment So basically you don't want to be the first one, right? So well for real-time Linux, we actually we have many users around in this world Which are currently already using real-time Linux systems. So we have a lot of industrial Applications we have the automation industry Actually already automotive industry we've got multimedia systems. We even get well aerospace systems in non-safety critical applications And we actually we have financial services, which is pretty strange, but They run it on big server machines for example for high-speed trading and stuff So they need a real-time operating systems and these guys are actually using Real-time Linux. So we already have a lot of users of this system around so When it comes to real-time and the real-time operating system We need to check a couple of requirements for this. So we need Deterministic timing behavior And to get deterministic timing behavior One of the most important features operating system needs is preemption Well, basically you need to be preemptible at Most of the parts in the operating system because the high priority task needs always to be able to to preempt the low priority task, right? So this is the most important requirement for a real-time operating system And well, you also have a couple of special situations you need to deal with One of these is the so-called priority inversion so some of you might have attended the Real-time summit yesterday. They had a couple of discussions on this So in real-time systems we have This classic error scenario. So Basically what could happen is Just think about static priorities We've got three tasks Task one is the highest priority task Task three is the lowest priority one So basically now these two tasks need to share a resource. So we need some kind of synchronization So what would happen right here is task one is blocking because task three is holding that resource Which is basically not a problem at all But now what could happen is task three gets interrupted by something in between Before it can release the lock So the result would be A high priority task is waiting for a low priority task and the lock never gets released until task two stops running So this is a clerical classical error situation in real-time systems And actually a real-time system needs to deal with that So we have a couple of strategies and actually linux can already deal with this situation You don't even need a real-time extension What linux does is in this situation We've got the so-called priority inheritance Which is basically to to keep it very simple In this situation what we would do is we would boost task three to the priority of task one Until the point it releases the lock so we can avoid task three to to get interrupted by someone So it's not just the preemption. We have to deal with a lot of different use cases Which are very specific to to real-time applications And we need to think how we can get these features in a general purpose operating system like linux so Traditionally We have two approaches To make linux real-time capable And uh, well the oldest approaches are the so-called dual kernel approaches which are basically not Doing real-time in linux. It's like having real-time in linux on the same system So these guys just introduce a micro kernel Which is doing the real-time And well the other idea is just to find a way To make linux itself real-time capable so Let me try to explain that so this is the classical dual kernel or micro kernel approach So what these people do is They do a simple real-time kernel. So this part is just Well doing the real-time stuff And the idea is that linux is just running on top of this micro kernel as a low priority real-time task So if you don't have any critical real-time applications running Well linux can get some run time So this is the basic idea of these micro kernel approaches which Looks quite clever at the first glance Um, but well if you think a bit More about this approach you've you've got two problems you have to to solve, right because Someone needs to maintain that micro kernel, right and needs to port it to new hardware So just think about the amount of arm socs which which are coming up right now So this is exploding. So someone needs to maintain a micro kernel needs to port it to new hardware This is a huge effort and well, basically these communities are not as big as the Community of the linux kernel. So this is basically something you have to deal with in this situation Well, and we've got another point right here Well, you can see linux is run not running on the on the physical hardware, right? So you need some kind of hardware abstraction layer Inside linux just to make it capable to to run on the micro kernel So basically you have two things to maintain You need to maintain a micro kernel and you need to maintain a hardware abstraction layer to adjust it to the most recent linux versions So also this is a big effort and basically this is one reason why most of these approach Approaches are usually a step behind of the linux development because they just can't keep up with a fast linux development but well these were the First approaches we had for real-time in linux and and these approaches are still around them I'm going to give you two examples on these implementations so The other idea would be instead of moving the work to the micro kernel finding a way to make linux itself real-time capable, which basically Looks like a huge task Well, basically that means you would have to touch every single file in the kernel And I can tell you we had to But basically you can manage that so at the first glance looks like a big big shot But at the end of the day the result you have right here is Well, it's just you have to real-time in linux So you have no microcone to maintain You have no hardware abstraction layer to maintain And you you can just use the standard tools. You can just work with linux So basically if you would have the choice to make linux real-time capable That would be the way you you would want to go So, well, let's have a look into some some Real implementations of real-time linux So one of the very first approaches we had for linux and real-time was rtai The so-called real-time application interface This comes from Italy from the University of Milano Actually, I think actually they used it for a couple of aerospace Applications and this was actually the first real-time linux. I was using right 15 years ago or something. So This this approach is around for years. So rtai is a classical dual kernel approach. So basically This this kind of implementation So Rtai had a couple of drawbacks when when using it. So basically when you when you write So this is actually still around and maintained, right? But the basic idea when using rtai is You write a kernel module you write a kernel module which is basically scheduled by this micro-curl but basically Well, it is like kernel development. So it's Not like writing an application. So it's really hard to deal with Hard to debug hard to get into it and basically uh one one thing we I had to face I had to do this for an industrial customer Well, since it's most of the parts are kernel space. It's gpl code Which is not a big deal if you do open source software But well for an industrial customer, you usually also have some closed source parts So you always had to think about which parts you can put in user land without real time and which parts you can put Into the kernel with real-time approaches. So Basically doing real time with rtai was pretty hard because most of the stuff is done in kernel space um, they had some interface for doing real-time In parallel to user space tasks This was called lxrt Well, I tried it several times. So for me it was not Working pretty well. So I got told to get more stable these days I have to admit I haven't tried it for a long time, but for me never worked pretty well. So basically Usually for rtai you you have to stick with the kernel space um Another problem we had with rtai was one of the design goals Is basically not portability. It's more or less for a given number of hardware To achieve the lowest latencies you could possibly get That this was basically one of the rtai design goals. So This means that the number of supported platforms it's Well, it's it's quite limited. Yeah, it's it's basically x86 32 bit 64 bit a couple of arm platforms and that's it So if we look into the rtai design um Basically looks like this and this is the classical dual kernel approach And basically you you can identify that the two problems I mentioned about micro curls You've got to maintain this real-time kernel and actually you can see This real-time kernel is just handling everything which is Related to the timing critical things in the system. So you have to maintain this real-time kernel and right here You have to maintain this hardware abstraction layer So these are the two problems I told you about so this is a classical example And linux is just completely in a different world The only thing you've got is a simple communication mechanism to synchronize both worlds But that's basically the idea and you can Nicely see in this figure that basically it's not real-time in linux. It's just real-time in linux in parallel on the same system That's basically what you do with these approaches Well, there's another approach which is called senomai and this is Still pretty popular. So this is still used in a lot of industrial system. It's still around on the market It's also quite old. So I think it was founded in 2001. So it's around about 15 years or in 2000 Like it's it's more than 15 years now. Well, um, the reason why it's still that popular is that these guys Had one advantage. They always had a proper solution to have real-time in user space So you didn't have to do everything in the curl So userland is much easier to deal with So this is basically one thing these guys made pretty good and they had a nice idea for real-time in user space they implemented the so-called skins which is basically a emulation layer for the api of different real-time operating systems So you have a for example a psos skin. You even have a posix skin So if you have a pre existing code base for some operating system You can reuse it with a tsunami Which is pretty nice. So basically these layers are not feature complete. It's it's a subset usually even the posix layers Just a subset of posix, but well, you can actually reuse a pre-existing code base for rapid prototyping porting your application very fast to send them I so that was basically a nice idea um They have a lot of supported platforms way more than rta it is maybe also reason why it's More widely used in the field than rta. I but It's still A dual kernel approach. So also these guys are using a microcone for doing the real time. So Basically, we we suffer the same problems. We've got the microcone to maintain we've got a hardware abstraction layer to maintain and Well, even if you can have real-time in user space There's one additional problem. You can't use the standard c library You need special tools and special libraries to deal with senomai so Even for posix you need to link your application against the senomai posix skin So it's not standard glibc. You always need special tools and special libraries To deal with these microcone approaches. So the handling is way more more complicated So if we look into the structure of senomai Basically looks like this. It's actually not that different from rta. I you can see it's a classical dual kernel approach we've got that Microcone which is dealing with the hardware Right here You've got the real-time tasks running on top of that microcone But you can see a small difference You can see that part which is called nucleus. You've got and you can see that code path, right? So we get the real-time into user space Which is much easier to handle than the kernel part So this is a big advantage but to use this once again You can't use any standard tools of linux. You need to use special libraries and stuff So it's much easier, but still some special tools are involved So Let me summarize The dual kernel topic for you So well the dual kernel topics the dual kernel approaches have been the first ones We had around on linux for a couple of reasons because when when these approaches have been invented We didn't have preemption and stuff in the kernel. So Implementing real-time was pretty hard. So basically this was the obvious approach These approaches are still around on the market But we have a couple of drawbacks. So we've got a special api special libraries. You have to link against so This is basically one one drawback. We have Well special tools and libraries We need to maintain a microcone and the hardware abstraction layer So basically you usually step behind of the linux development So this is also some big issue with these approaches and last but not least just remember We also have high-end uses like Financial services and stuff for a little time linux. So these guys are not running real-time on a two or four core machine arm These guys are running on a s390 with I don't know how many cpu so For these guys we have with these microcone approaches We have a bad scaling problem because basically these microcones are known to to scale pretty bad on big systems So they usually scale up to eight or 16 cpus But if you get more cpus, they actually don't scale pretty well So we have one approach to make linux real-time but Basically these these approaches have a couple of issues So for that reason People started to rethink if there just wouldn't be a way To make linux itself real-time capable And well actually we have now one approach Which is called preempt rt or the so-called real-time preemption patch So the reason for this name is what we basically do is we just introduce a new preemption model to linux Which is called real-time or full preemption So this is why it's called a real-time preemption patch This is a classical single kernel or in kernel approach making linux itself real-time capable And well, basically this approach is now around For 12 years. So it's also quite old. Um, it's it's widely used in the field It was basically founded by thomas glijksner and ingo mollner Um, these two guys in 2004 started a lot of real-time linux development They started to coordinate their work and the result was the real-time preemption patch Well, we have a huge community for this development right now So basically the reason for this huge community is that actually Most of the parts of this development are right now already included in the linux curl So 80 percent of their development already made it into linux So that's the reason why they have a huge community So basically if you're using linux nowadays and look into The timers the interrupt handling tracing infrastructure Priority inheritance and all that stuff all these features were basically originally developed for the real-time preemption patch And we're pushed back into mainline So this is the reason why we have so many users for this approach um Basically Since this approach makes linux real-time itself. You don't need anything special to know about it It's just if you write a real-time application You just do a simple posix application Which means you can buy a 30 year old os book like About posix real-time programming and basically you know how to to write a real-time application for preempt rt You don't need any special libraries or stuff. You just use your standard c library So this is the big advantage of making linux itself real-time capable So um, we also have a high acceptance in the community for the real-time preemption patch So what's the reason for this? So basically as I told you we have a lot of features Which came from the real-time development into the mainline development? So basically what we did we added some powerful features Which are not just useful for the real-time users, which are useful for everyone So basically these guys made linux a better operating system also for all the other users So this fact got them a lot of acceptance In the community and the other reason is that well when they made linux real-time everything got preemptible So we found a lot of race conditions and locking problems fixed these and pushed them back into mainline So actually we improved the stability of linux a lot and this is also one reason why these guys have a high acceptance in the community So they're not just not just focused on their work. They just get some more quality into linux So basically the mainline integration of real-time started a long time ago And actually I just check this these days the mainline integration started in 2006 So this is now actually 10 years ago We started to push the first features into mainline So this is a original statement from linux torwalds from colon summit in 2006 so The reason for this statement was Usually if you want to get a new feature into linux He wants to know a use case for this so Who the heck is using real-time on linux? So actually One developer was telling him. Well, I've got a customer who's running welding lasers with linux already with a real-time extension so Well, that was the answer of linux like well Doing a welding laser with linux is kind of crazy, but well We are all kind of crazy. So i'm fine with you doing the real-time stuff So sounds funny But basically that was the starting point of the mainline integration of the of the real-time linux development round about 10 years ago Nowadays you you still need a patch to have the real-time stuff. It's like 20 percent of the original size we got most of the stuff in the mainline kernel right now You can get this patch for every second kernel version And currently the linux foundation is pushing the mainline integration So the main goal is like having the real-time capabilities within linux within the next two years So this is basically we're doing that development on behalf of linux foundation So this is basically the idea so How did these guys manage to make linux real-time? So we need to remember again What's the main requirement for real-time operating system? And the main requirement for real-time operating system was preemption To achieve deterministic dehyming behavior. We need preemption. So these guys Had to find a way to maximize the preemptible sections inside the linux kernel so and basically Well, this is a very simple Explanation, but this is basically what they did First of all, they had to rework the locking primitives To well minimize the atomic sections inside the linux kernel and to maximize the preemptible sections inside the linux kernel So basically what they did was reworking the spin locks And finding another method to deal with these situations One idea Yeah, just basically reworking the locking infrastructure Basically at some point the real-time preemption patch was also called the sleeping spin locks patch That was the reason because we had to rework all the locking primitives Well, and they had another pretty cool idea Another section in the kernel where you are not preemptible is basically the interrupt context So what these guys did was instead of running the interrupt handlers in hard interrupt context They moved it to a kernel thread So what you basically do is when an interrupt arrives You don't run the interrupt handler code. You just wake up the corresponding kernel thread and the kernel thread will run your interrupt handler Which is two advantages basically The kernel thread is interruptible main advantage And well, it shows up in the process list with a pid So you can even put a priority on your on your interrupt which is arriving So well, you can give a low priority on your non important interrupts and give a higher priority on your important userland task So running it into a kernel thread was actually pretty nice and pretty cool idea And actually this is also one of the features The threaded interrupt handlers which are already available In linux mainline So even in mainline linux nowadays you can decide if an interrupt handler can run in hard interrupt context or in threaded context So actually also one of the features which came from preempt rt so well, this is Just last but not least a quick overview how the real-time preemption patch this single kernel approach is working So basically well, you can't see any difference to standard linux. It's just standard linux Right just with a new preemption model. So there's nothing special about it. So if you do a kernel driver It's just a linux driver. If you do a real-time application It's just a linux application. So you don't need any special libraries or special api You just do a linux driver or posix application That's it so now i've promised that i will show you some Results in a real-world example, which Latencies you could possibly achieve Um with these real-time linux extensions So what we did was we took a simple onboard With a altera cyclone 5 Actually, we took the rocket board which is a community evaluation kit of that soc And we did a couple of latency measurements and what we did was we did the same measurements for senomai Like a dual kernel approach and for preempt rt a single kernel approach So um, well, there was actually a special reason why I wanted to do these measurements because um Well, if you google for these topics You will find a lot of papers comparing senomai preempt rt and rta i And basically most of these papers claim that the dual kernels are way faster when it comes to latencies So and actually for most of these papers. Well, I was able to figure out that Yeah, the preempt rt systems are most of the times pretty misconfigured. So My idea was to get a fair comparison of dual kernel and single kernel approaches and see which latencies could be achieved So we actually took one senomai expert and one preempt rt expert Gave them the same bird gave them a task and then just tried to compare the results. That's basically what we did So I think that was pretty fair, right? So, um, but yeah, let's wait for the results So, um, what we did was, um, we did iacq tests So we fired iacq with a frequency of 10 kilohertz We took the os adl latency box to do that I will tell you how this box is working later on and we put the system in a worst-case scenario So basically since you want to guarantee timing behavior, you need to guarantee this Well, even in a worst-case situation, which is usually high cpu load high memory load And a lot of work for the scheduler Well, we achieved that with a tool which is called hackbench So this is basically how this tool works You've got process groups With 20 clients and 20 servers and the idea is that each client sends 100 Messages via socket to each server. So you've got a lot of inter-process communication And you can tell hackbench how many groups to run So for example, if you take 40 groups is 40 times 40 processes So you have 1,600 processes doing weird inter-process communication So basically this is a real worst-case situation because yeah, the scheduler continuously needs to take decisions You have high cpu load. You have high memory consumption. So this is basically the real worst-case situation for your system And while doing this while running this we just started to Fire interrupts and measure the latency. So this is basically what we measured like Once again, I'm taking myself as an example like we get an external event Like this morning the alarm clock was ringing I need to react to that and get out of the bed Basically, this is what we what we did on the embedded board We just fired an external event and did a simple driver who who needs to do this reaction And well, what you can do with the latency box is this latency box can Record all the samples you get and do a nice histogram for you because this is also quite interesting Just not to get the worst-case latency also the Variety of latencies you could get. Well, basically once again taking me as an example today. It was quite easy But well, I might go drinking tonight. So tomorrow it might be quite hard to get out of bed So it's not always the same latency. So you have some Variants in it, right? So this is the so-called shitter And that that's why we want to record all the samples and put them into a nice graphic And this is basically what the os adl latency box can do for you So this box can fire interrupts in a preconfigured frequency. So in our case, there was 10 kilohertz So it has an output pin Triggering a signal you send this signal to your Test system your test system needs to trigger a pin Which is connected to an input pin on the latency box and the latency box will just measure the time in between and record all the samples so We did this for 12 hours, which is Not very long, but it's the bare minimum to get a good idea of the real-time behavior of the system So this is really the bare minimum. You could take it's not really long, but it can give you a good idea So this is basically the measurement we did firing interrupts with 10 kilohertz Getting a reaction from the target system and measure the reaction time Do every measurement has been taken with the latency box So actually what we did was we had two use cases And the most important one was the reaction time of a user space task And this is basically the real world example If you need to react to an external event, basically you need to synchronize Your user space task with some interruptor stuff like some field bus interrupt or whatever But basically you want to do your work your homework your real calculations in user space So this is basically the most important example So that was the reason why we took this as the first measurement Measuring the latency from the external event arriving into your application And and triggering the GPIO which goes to the latency box again So well We started this these measurements on senomai These are basically the results So as you can see we did a nice histogram. Well, you can see the latencies on this axis You can see the number of samples right here so Now The average was like 30 microseconds reaction time But basically the average is not the number we are actually interested in right So we want timing guarantees. So if you want to to look into timing guarantees We need to check for the worst case Well, you can see the worst case latency right here the small peak So we had actually one sample which was around about 90 95 microseconds That was the worst case latency we got on that device with the 10 kilohertz interrupt So let's say 95 microseconds worst case latency, which is actually not too bad For user space application So now we did the same measurements On preempt RT You can see Well, the variety so the chitter is a bit bit bigger. Yeah right here But you can actually see this is the worst case latency we got which is Pretty much the same we had for the dual kernel like it's even slightly better But I would say it's pretty much the same bit less than 100 microseconds what we could achieve So well, basically this can prove that we don't have that big difference You can find in a lot of measurements Which are around in the internet between dual kernel and single kernel approaches So in this test case we had pretty much the same latencies on the dual kernel and single kernel approach And well actually one thing we tried right here is well since Preempt RT is a standard linux system. So we can actually use All the standard linux features So we were wondering What would happen if we just isolate one CPU that was a dual core system What would happen if you start isolating CPUs for just handling that interrupt and see what happens So basically that was the next measurement we take we've taken Doing the same measurement with an isolated CPU. So you can see it gets slightly better like from 90 to 80 microseconds But this is basically well, you don't hit a specific worst case because it's running on a different CPU Well, this is done pretty often in real-time systems that you just isolate a specific core for For some specific tasks. So well since there was a standard linux use case It was quite easy to isolate this core. So we've got this comparison I think the most important Graphs are these two right here the green one and the red one the red one is just the preempt RT without any special configuration and the green one is senomai Without any special optimizations, you can see basically the latency we got for the user space task was pretty much the same Which is actually pretty cool when we compare a dual kernel and single kernel approach so We did another measurement And well, actually I was not really happy with that Measurement. Well, you might get that if you look into that slide, right? What we also did was we measured the kernel latency And what was the reason why I was not happy with that measurement because Basically in my opinion, what what you basically do you compare a both Full-featured linux system with the reaction time of a micro call um Well, you can discuss about that, but in my opinion, it's not a fair measurement because yeah, we compare linux in the micro call So basically in my opinion we compared apples with peers But basically, um, the results were actually quite interesting So I would like to show the results right now because, um This is the result for senomai and well, if you if you look in the internet When we we got a couple of measurements in the internet where just the dual kernel approaches were 300 or 400 percent faster than a single kernel approach But basically, yeah I was expecting some something similar right here. So we got um 30 microseconds for senomai Well, and if we check out what we achieved with preempt RT without any optimization was like Well, it's slightly better than the the user's case base latency But well, we we need to remember we just go into a kernel thread Right here. So it's different. It's not that big But now if we start to optimize this use case and just once again start isolating the core We even get down to 60 microseconds Latency for the for the preempt RT, which is basically if you compare a micro call And well a full featured linux system. I think this comparison is not too bad 60 microseconds is actually Pretty fast on that kind of cpu. So the difference is not that big as you would expect And not that big as a couple of measurements in the internet claim to So actually we're getting pretty close with the single kernel approaches So now we did one last measurement On the on the kernel agencies Well, and and basically the idea was we were running on an ARM platform And well, I was saying well we compare Apple with peers But well if you go with the micro kernel, you might be willing to to accept a couple of limitations on your system So I was thinking about Could we also put some limitations on the preempt RT system and see if that could somehow improve the latencies Well, the the idea I had was just I was just curious how that would improve the timing behavior Just trying to use the fiq feature of the arm cpus Which is a hardware feature. It's a special interrupt vector on ARM Which is running in its own context. So it's very limited in the things you could possibly do. It's living in its own world So it's very limited what you can do, but well when I'm willing to to accept Limitations basically this this might be an option. So what we figured out was well, we've got 10 microseconds Or even less Average latency and you can see the worst case is all around about 30 microseconds And that was actually a pretty interesting number because If you compare this it's also 30 microseconds. So We are quite sure we just hit a hardware latency in this measurement But we didn't have to the time to track this down right now But we've seen this on similar architectures with cortex a9 cpus. We see the similar latency. So Really looks like a hardware thing so well This is the final comparison of the kernel site. So taking into account that we compare a Full-featured operating system with a micro kernel. Actually these numbers are not too bad And well, we need to remember and that's that's the conclusion With the pre-empt rt patch we have a single kernel approach Which is basically nowadays the defect of standard for linux and and it's getting integrated in the mainline linux development Um, it's pretty simple to use you can use standard c library knows nothing special It's available for every second kernel version. So it's the standard. It's easy to use and it's Basically for the most of the use cases It the micro kernels don't have better latencies. So basically the latencies you can get Really comparable So basically if I should give you a recommendation, you know Um Pre-empt rt would be the way Because it's the standard and the latencies are pretty good. You could possibly get And last but not least. Well, if you're willing to take limitations Instead of going with the microcone solution, you could also check out Which features your hardware is offering like the fiqs on arm You can use that we've we've got support for that in linux But you need to be aware of that the things you can do with these hardware features are actually pretty limited So well There was basically all I wanted to tell about real-time linux. So it was pretty high level I know but hopefully it was quite useful for you to get an overview about real-time linux So well, basically, um, thanks for listening and well feel free to ask any questions Any questions we've got one minute left. Uh, is there a mic? No Yeah, yeah, it was yeah. Yeah Yeah, that's true. Yeah, basically yeah that the what we basically did was pretty simple Just we booted with a isolate cpu. So we just got the housekeeping stuff on on the real-time cpu And all the rest was just running on the second cpu. So basically it's half the load, right? Yeah, that's true Any other questions? Yeah Yeah Well, it it somehow is I but I agree with your point This is basically a problem we have nowadays. So basically, I think your point is this With these measurements, we don't have the mathematical proof of that We we we can't miss things But the the main reason why we don't have this nowadays is basically we for Modern system like the board we've been running on we don't have the option to do the mathematical proof It's nowadays. It's not possible anymore It's like in the good old days of micro controls you had a fixed frequency You didn't have any caches so you just had to calculate the numbers of instructions of the code path calculate Which code path that you go and it was quite easy to to to calculate the latencies you get Basically the problem you have nowadays We've got even hardware features which are not reliable. So basically we we don't really have the the option to do a mathematical proof But you're right. So with a 12 hour measurement Basically, I can't be sure that I have a timing guarantee So what we basically do is to to hit this point to address this point since we don't have the way of a mathematical proof We try to run Long-term tests and not just 12 hours. So like you you you might have heard about the os adl They they run a a test farm Which is doing like long-term tests about one year two years three years And you can actually get all the timing samples for a long time on these boards This is basically one one point. We try to address this but in my opinion. We don't have the The option nowadays to do the mathematical proof Well, yeah, yeah, so I agree. Well, the use cases might be limited But well, there's there's still we still have a lot of use cases where you might need just a small RTOS system which can be mathematically proof So I agree on that But basically for the huge socs and the complex system like linux. There's no way to do a mathematical proof Well, yeah, actually I never thought about that approach But I think that the main problem is not just the code And once again the main problem we would suffer nowadays in my opinion is the hardware which is which is behind You just need to think about we've got up to three or four different caching levels nowadays And and actually we've got CPUs nowadays, which are reordering instructions for us So basically if we even if you have to prove for the code You still have the problem that you don't have to prove for the hardware And for linux actually we don't know on which hardware the linux system will run at the end of the day So that's basically the problem in my opinion here But actually if you're interested we could continue the discussion in the break Well, yeah, you've gotten a couple of other issues there like safety requirements and stuff But yeah, yeah, I agree But well, I think we're running out of time right here But if you're interested we could continue that discussion in the break So well, yeah, I think time is over. So once again, thanks for listening You


So a little background on this presentation. Karen and I noticed that there were quite a lot of conversations happening in the library space around business models, digital infrastructure, evaluations that were happening, and not everything was always sort of making sense when we were hearing the conversations and listening to folks. So we thought that we would add to that conversation about business models financial sustainability. So we could help our colleagues, all of you, with evaluating all of these different digital infrastructure services. So for the purposes of this presentation, we're borrowing our definition for digital infrastructure services from actually IOI. They did a really great report, totally recommend reading it on digital infrastructure definitions. It's probably the longest definition we'll read, but it was really great. They did an analysis. We are slightly modifying it by adding in preservation to their definition. They are explicit about preservation in their report, but we thought we would call it out specifically for digital infrastructure services. So you can see that here. I'm not going to read it to you because we only have 30 minutes. So Karen and I actually decided to sort of focus on digital preservation service providers. And so here we have a list of different service providers in the digital preservation space. You can see the names of some of them here. Now, these service providers come into sort of two camps. You have the non-profits and the for-profits, but really there are actually subtypes underneath non-profits and for-profits. And oftentimes that gets lost in the conversations that we have about comparing and contrasting these different services. So you can see the subtype list and I'll talk a little bit more about those in a second. So again, this presentation is to help you with identifying criteria for reviewing these different types of services. One thing to note too, even though I did put Islandora, which is Canadian on here, we're really going to focus on US-based organizational models. That's what Karen and I know best. So we're going to talk a little bit more about that. So first I'm going to talk about financials, and then I'm going to hand it over to Karen and she's going to talk about governance. So I'm going to start with a definition of course for for-profit and non-profit. I know that like we should all know that, but just to be safe, for-profit of course is a company that is working to make money in theory. You can always have a for-profit go wrong. But underneath that there are a variety of different ways that for-profit companies can gain funding. You know, they can be a sole proprietorship. They can be, have like a partnership. They can be a corporation, an S-Corp, an LLC. Some for-profits can be get their funding from venture capital firms, and that all places certain regulatory requirements on them, right? Like or business requirements on them from who they're getting their funding from. On the non-profit side, non-profits are organization that do make money, but their goal is to reinvest that money within to the communities that they have set out to support. So that I think is a very important thing to remember throughout this conversation is that's really their goals, is to take their money, reinvest it in their communities, and lift their communities up. In academia, in libraries, we are very familiar with 501c3s. There are around 34 or 35 different types of non-profits in the US, and at the end of this presentation there is a resources slide, and you too can go to the IRS's website and look at all of those. And so how those different non-profits are structured can vary. Even within non-profits like a 501c3 can be structured in different methods. So that's something to keep in mind, like legally, which then has implications on the financial, you really want to think about how your non-profit that you're engaging with is structured, because it does have implications on the services that they're offering. So when you're looking at a for-profit or non-profit, you're going to want to think about their financial health. Not all companies are required to provide information about their financial health, but some things that you could ask for are your balance sheets of a company, income statements, cash flow statements. These are things that most companies will generate as part of doing business. Getting access to those can then help you create financial ratios that will help you understand how well a company is doing financially. I was going to go into financial ratios, but you're lucky this isn't a 45 minute presentation. But again, at the end of this presentation, there is a link to how to calculate financial ratios and some of the different types of financial ratios that exist and what they can help you evaluate about a company. On the non-profit side, there are a variety of things to consider around financial health. First is the funding model. How are they getting their money? Are they diversified? Are they only getting money from one space or are they going off and you know, they have different spaces in which they're taking income in? The other piece of this is the cash flow and sustainability of the cash flow. There's different schools of thought about how much money nonprofits should have and reserve. Some will say six months, some will say one year. But really the non-profits should be thinking about why they're keeping that cash flow, why they're keeping that reserve, what they need in order to sort of sunset the organization if something were to go awry. Those are things that they should be thinking about. Keeping too much cash on hand that doesn't help you with that goal is not good, right? You don't want somebody keeping 20 years and never reinvesting their money into the communities that they're aiming to support. The other thing to keep in mind is expenses and organizational efficiency. If non-profits are not organizationally efficient, then they are doing a poor job of stewarding your money. That's just flat-out true. We can go to battle over it later. The last thing you'll want to look at is debt. How much debt an organization is taking on? Regardless though for financial health is you really want to make sure that your organizations are being transparent with you. It's important to note that some non-profits are not required to be transparent, right? Like a 501c3 has a 990 form which is available on the internet. You can go look up your non-profit and check out its IRS tax form. But some non-profits which are subsidiaries of a larger non-profit, I'm going to pick on Bradley at AP Trust. Like AP Trust is not in itself a legal entity. It's underneath the umbrella organization. They actually don't need to be transparent in their financials. But one of the reasons I appreciate AP Trust and Bradley is actually they're very transparent about their financials, why they keep how much cash on hand they do, and here's the information about what we've spent every single year. Bradley tells me all about it, and he openly shares that with anybody who asks. So that shows an organization that's doing a good job of stewarding in my mind my funds. And with that I'm going to hand it over to Karen to talk a little bit about governance. All right. Good morning everyone. I am going to speed through this because we do want to have some time for questions. So I apologize as I'm about ready to go on my speed journey. So come with me. The first thing that I wanted to point out is governance is something that Rosie and I have been involved in in many different projects and actually an unexpected find of us doing preparation for this report is I dug too deep. And what happens when you dig too deep? Well, you find out that boards are not only inextricably linked to capitalism but they are inextricably linked to the slave trade and to colonialism. So as we think about all of our structures and behaviors, boards are based in a racist society to make differentials about people and people treated as not people. So this is a very important thing to know. Thanks for the clap there. So that said, we live in a capitalist society in America which highly provides and promotes this. So you are actually legally required to have a board if you are for profit publicly traded company.  The only option where you can kind of get out of it is if you have a private or closely held company. So a group of individuals, if you're running your own company, if Rosie and I start an amazing business, we'll probably keep it close. So here are the things to note, though. With both the for-profits, there is the intent to make a profit and so really you're doing what's in the shareholders or the members best interest. In a for-profit public model, that is almost entirely synonymous with growth. So as you look at companies and back to Clis' remarks yesterday about what if this company gets sold to this other company or how does your data get through? Part of it is they have to. If you have to grow, you have to buy more, you have to have more services in order to get more money for your shareholders. Non-profit boards, Rosie has dived into a little bit so I'm going to just skip past this, but really it's more about advancing the mission. So when you're looking at boards and you're looking at your digital preservation services, we are now looking at apples and oranges. We're looking at for-profit. We're looking at non-profit. What are some of the things that we can look across them? One of the things is how are their members selected? Do they have an election? What is the representation of that board? Are they compensated? One of the big struggles we have in boards and companies have as well is what's known as a weak board. A weak board is where the CEO goes to the board and says, please tell me all this is great and they say, hey, yeah, you're great. Part of that comes down to two things. Usually it is either that people have been handpicked by the CEO or subset of the organization, or they don't have enough time in their daily lives, so they're giving five percent of their attention to the board. So when board members are compensated, it is usually supposed to say, we value your time and you are worth compensation. Right? Kind of the opposite.  That's not the rationale. It could be what it is in practice, but the rationale is you are buying their time because their labor is worth it to make your company or organization better. So who really has the authority? A strong board is a board wherein often CEOs will rotate quite frequently because they do not have the support or the function of the board to continue in their work. So when you're looking at companies and organizations for digital preservation services, take a look at the board makeup and see if it is a strong or a weak board. I'm going to touch very briefly on fiscal sponsor organizations because this has come up quite a bit in our digital preservation discussions and one of the things that is really tricky here is there's a wide spectrum of services that they provide. At the very minimum, usually it is the financial services, but they can also provide administrative services, healthcare, employees, legal services if you do get sued to Morris's point earlier. So again, I'm going to kind of slip past this, but fiscal sponsor organizations really can have a tight relationship with the project or a very loose relationship. It all depends on what the agreement is. So I decided you couldn't have a presentation about capitalism without clip art, so I hope that you appreciate that I made sure there was plenty of clip art in this presentation. One of the things I think we do in our library and our IT communities is we often convolut advisory groups with legal entity boards. And so a couple different things that are important about advisory groups. They are not legally required. They really should be specialized experts. They can provide things like a sounding board, fundraising, mentoring, program evaluation. They should have a very specific scope and sometimes many experts would argue a short duration as well so that you are refreshing that advice. The other things to look at is what is the organization doing as far as outreach and or market research? If you are an academic library and the organization is only talking to museums, is that problematic? If you're a museum and they're only talking to academic libraries, is that problematic?  So like I said, I want time for questions, so I really am zooming through this, but we have provided in the slides will be available a checklist. And this really is focusing on things that you would ask yourself when you're looking at evaluating digital preservation services. This is not the will my data get exported. That's something else. This is in addition. So yes, there's a lot more work for you, but is the board fully informed? Is the financial health available? Are the reports out there? What are those health indicators that Rosie talked about? Are the reserves aligned with business growth? A lot of times in higher education we like to keep accounts of reserves for like three years, six years. Really the best practices in nonprofits is six years. So one year and four profits, it could be reserves that are as little as three months. Advisory councils that have a divine role, if there is one, what kind of fiscal sponsor relationship is their selection and compensation for the board that you should be aware of? And really are the goals and program evaluations that come out communicated with the community and the audience spoken with. So with that, we're going to go to questions, but I also want to put up, we did an extensive kind of list of references and resources that will also be available after this presentation. So thank you. Questions? Was it too fast? Okay, I had to check in with my friends. I'll come sit by you Rosie. I'm always good for a comment. Michael Siegel, University of Berlin, and I am Executive Director of the iSchools, so it's relevant what you were saying. And as a Chief Executive, I love this passive board. Exactly for the reasons you were saying. We don't generally have very passive board. We have elections, we have the rules listed in our bylaws, which I think is very important. Persuading people to run for board is a non-trivial challenge sometimes, because we don't pay people. And in the academic world we rarely pay people what their time is worth. I think one of the most important issues though from my perspective is the degree to which the board members must understand international practices. We are a highly international organization with members, it's schools of information, relevant here, worldwide, universities that have doctoral programs in this field. And I will tell you a story that my wife tells.  And I really do want to have time for questions. I won't tell the story, I will just say getting, especially the North Americans to understand European and Chinese practice is a real challenge. That's an educational issue. And I will say two things in response to your comment. One, we specifically did not look at higher education boards. We are looking at companies that provide services in a different way. So I would be careful about some of the things we did not do research in that area. We are not higher education, we are 501c3. Okay, great. But I think it is really interesting to look at what that board is made up of, right? Who is on that board? A proper board is going to have a diverse amount of experts from inside and outside the community. And that's what we are looking for.  Now I scared everybody off from asking questions. Thanks, Eric. Hey Rosalind, hey Karen. I really appreciate your presentations. I am Eric Mitchell of the ULUC San Diego. You started with talking about digital preservation. And so I would actually just love to ask kind of an open question about how you see board membership kind of lending sustainability to that kind of that big, you know, 100-year challenge of digital preservation? Yeah, it is an interesting thing to consider, right? It's one of the things, you know, I know Karen said that like six months one year of reserve is good practice. But when you're looking at an organization that's hopefully stewarding things for the long term, like it's one of the things like I called out AP Trust before, they do have longer reserves. But that is mostly for the long tail, right? Like if I have all this data and I'm trying to get it out quickly and I want to steward not only my organization's money well but also their content well, like how much time do I need to like sunset, right? How to do that work versus something like actually the San Vera community, you know, for us sunsetting would probably look like how long do we want to keep our community manager on for? And how much time do we want to give her to be able to find some new career or new position? Because we aren't stewarding content the way AP Trust is, right? And so it's different. You don't want to end up in a situation where, you know, suddenly AP Trust just turns off the AWS bill. It's not. We don't want that. So that I think it depends, right? Like use your best judgment, think through some of that around sustainability and what they're trying to sustain to for the long haul. Jeff. I'll just add, I mean, when we were creating the checklist, a lot of it is asking questions, right? What is a parent from this board? Is it strong? Is it weak? Where does it fall along that? So it's not trying to provide guidance on which one that you pick, but it's a very important question. And I think one of the things that I do think we struggle with is if we have a very weak board and we concentrate on cult of personality for a CEO for any of our digital preservation services, that is also very problematic. And we are as a group prone to cult of personality. So I do think that's another thing just to watch for. In the scholarly communication space, we're seeing a lot of reduced number of vendors, more complicated situations where this company owns this company, but it operates separately, et cetera. And I think we're starting to get glimmers of that in the digital preservation space as well. And I wonder if you have any advice for somebody who doesn't know anything about how these laws work and how the companies work. How does navigate that environment? You start. Okay. Well, so I would say one of the nice things is that because of the U.S. and legal rules, most of that has to be transparent for any non-profits. So you're going to see who the parent company is and where that sits in a or a sponsor organization. And honestly, Googling it is probably the best thing. I think the thing to think about on the corporate side is the more and more you learn about whomever that parent company is, and this is something, of course, we run into with B-Press and Elsevier, Ralex is a data distributor of a different variety, and their goal is to make money for their shareholders. It's a very different goal. So understanding who those companies are and what their goals are is important. But I honestly think the best way to do it is to Google, and I say that as a librarian. I think, too, I'll add that the other piece of it is engaging with the company, right? So we have the big companies, but there are some digital preservation services out there that are smaller companies, and the way that they do their business can sometimes be much more, you gain more trust as you're having those conversations. I had an organization that I did a contract with, and it was the most pleasant contractual experience I've ever had. It was great. Procurement was very happy. I was very happy. They were really engaged and open to hearing feedback on their contracts. And then the one thing that I had a problem with, they said, oh, yeah, we don't put that here. We put it up on our website, and it's all here, and you can peruse it at any time you want. And I was like, oh, this is great. Right? And so it's through those conversations of the initial conversations trying to gain an understanding of how they operate and who they are and how they're open to conversations, right? If they just ignore you when you try to ask questions about your contract, that seems problematic, right? I have legitimate questions. I would also say if you have a moment, schedule a meeting with procurement, that was like, it's the best 30 minutes of your life. They can really, I mean, they do this, right? That's their job. And they can really help you understand how to hold organizations accountable through the contractual process. And how to make the contractual process open to different types of organizations. Right. Really your takeaway. Make friends with your procurement officers. I think we're out of time. So thank you all for joining us. And through the speed review, the slides with the links will be available through CNI. So much appreciate it. And enjoy lunch.


Γεια σας, καλωστάω και καλωστάω. Είναι Vicky εδώ και σήμερα έχω ένα βίντεο επίπεδο. Έχω δεύτερη ευκαιρία που έχω δεύτερη ευκαιρία στο σχέδιο μου και έχω δεύτερη ευκαιρία σήμερα που είχα δεύτερη ευκαιρία στο σχέδιο. Έτσι, είχα δεύτερη ευκαιρία να κάνω ένα βίντεο επίπεδο, να σας δείξω τα κόμματα και τα πρόσωπα και ελπίζω να εξηγήσω όλα τα ευκαιρία. Τώρα, το ένα που χρησιμοποιηθώ είναι από το Tonex Studios και μπορείς να δείξεις ότι έχει δεύτερη ευκαιρία να εξηγήσεις και να εξηγήσεις τα πρόσωπα, αλλά θα μιλήσω για το πρόσωπο. Έτσι, αυτό είναι σχέδιο σχέδιο που δημιουργεί ότι είναι πολύ δύσκολο και στις ευκαιρίας, είναι πολύ στήριο και υπάρχει πλαστική πίτα underneath, όπως μπορείς να δείξεις. Λοιπόν, όταν βάζω από το σχέδιο μου, όμως τι κάνω από το σχέδιο, δεν μείνει από εμένα. Τώρα για χρόνια και χρόνια, έχω been working on self-healing mats which I do love still, but I found them really hard to clean and they don't take heat, at least the ones I had. Λοιπόν, είμαι going to measure the one that I have so that you can see how big that surface is and I'm going to give you the measurements on both inches and centimeters. So that's 60 by 36 in centimeters and it's 23 inches and 3 quarters of an inch by 14 and a quarter in inches. And again, there are many different brands of glass mats in the market. So I'm just showing you what I have at home. Now the main grid area on this glass mat, the one that I'm having by Tonic Studios, is in centimeters. So all the grid that you see is in centimeters and it's perfect for Europeans. I got mine from Amazon UK. By the way, this is not a sponsored video. I'm just showing you what I'm using just because I get so many questions about that. Now there are also measurement areas that you can use for creating your stardart card sizes and again, that's in metric system. And there are also on the other side, there are also different rectangles that you can use for creating your mats all the way from a bigger to a smaller one both in rectangles and in squares. Down below the description area you will find links to this glass mat that I'm using but also to other brands that you will find in the market. And instead of having the centimeters, they will have inches. So there is an inch ruler here, the one that I'm showing you here as well as that tiny little grid is in inches. I have learned to work both in centimeters and inches. So this is not an issue for me but you need to keep that in mind if you want to get this specific glass mat. Now I'm going to show you these corners, these are plastic corners and that's how the glass mat came. So you can place them there but I find that the corners are nicely rounded and I like how they are. Actually I tend to grab my hand on those corners so I decided to remove them and I won't be using them at all. Now this is a glass mat that you are supposed to cut on top of it. And I'm going to show you how easy it cuts. It really cuts like butter. It was a big surprise for me since I have been working on a self-feeling mat for so long and I didn't believe that it would cut so well. But anyway, the blade just slides on top of that and this is the number one question that I got since I have started using this glass mat if the knife creates scratches on top of the glass mat. Well, it doesn't create any scratches at all. It's totally smooth. Remember this is tempered glass. It's like the glass that we have on the screens of our mobile phones. But you need to remember that the blade of your craft knife might dull sooner than if you were using it on top of a self-heeling mat. Now I don't cut paper too much with my craft knife on top of my surface so this is not an issue for me but depending on how you use your surface then you need to keep that in mind. For me, the most important thing on a glass mat is to be able to apply heat for embossing for example which this glass mat can take and also easy cleaning. Self-heeling mats are not as easy to clean since they have all those scratches where the ink can go inside. So for all the mess and for many techniques like blending I had to bring in my ranger craft mat but the glass mat combines everything and I can do all the work on top directly. So you see it's very slippery just like that ranger craft mat and I can do my blending beautifully and I don't care if I apply ink on top of my mat. This is not self-heeling mat. The ink is not going to sink inside those scratches and it will be easy to remove even without water. So I'm just going to bring in a clean cloth here and everything comes right off. Which makes it the perfect surface for mixed media and for tons of techniques. So for example I can do the famous technique of team holes where I am squeezing a little bit of ink. I'm applying water and then I can bring in a watercolor paper and just place it on top of that and it works perfectly fine and I will have no issues cleaning. Now here is another thing that I love about this surface is that it is white which means that when I am applying a color on top of it I can see the real color as opposed to the craft mat that I used to work on which was brown the ranger one I mean. So I can use it as a palette and I am going to apply a few colors with my zig markers there and then I can easily use my water brush to apply the color on my paper as well as I can do the mixing and I don't care at all if I make a mess on top of my glass mat because it's super easy to clean. And if you find that some of the ink doesn't go away just spray it with a bit of water and it's brand new again. Now one thing that I had trouble with was alcohol markers. When alcohol markers touch a self-healing mat then it leaves a stain forever which is not the case with the glass mat just because it's so slippery once you let it dry it's super easy to remove. I am going to apply a couple of colors here so you can see I'm going to bring in my towel which is going to easily remove the stains from the alcohol markers. Now if you find that some of the ink doesn't move then you can easily clean the surface with some isopropyl alcohol this will clean both ink stains as well as adhesive. And since we are testing things out I'm going to bring in some dye inks apply a few colors just to create a palette there and check out how easy they are to clean. And I can easily remove stains that are more persistent such as stazon ink for example.  And the number one thing that I love about the class mat is that you can apply tape adhesive on top of it and it's real easy to remove. You can remove it with an adhesive eraser or again with your isopropyl alcohol. Another great thing is that you can apply your mediums directly on top of your surface I see if you were doing if you were working on top of the range craft mat so I have applied some embossing paste there and you can use your gesture on top you can use your acrylic paints on top you can see I'm working with my gelatos there again, work with it as a palette and they are all easy to clean even when they are dry. And another thing that I need to point out is that the grit that you see is not printed on top of the glass it's underneath the glass which means that no matter what you do or how hard you clean your surface it's never going to fade out. So you can easily scratch any bumps and things that you might have dried on top of your surface with a spatula just like I'm doing here. Don't worry it's going to clean the glass perfectly and it's not going to leave any scratches. And finally it takes heat beautifully which was not the case with my self healing mats so you can even emboss on top of it you can use your fuse tool on top of it and you can even use your heat gun and the truth is that the silicone is super easy to remove once it dries because it's such a slick surface. So I hope this review was helpful I tried to cover up as many things as I thought if you still have questions let me know below and I will answer them and I hope I have helped you a little bit to decide if a glass mat is for you or not. As always you will find links down below to this glass mat that I used as well as to different brands and sizes You can check out the prices in different online stores Thank you all so much for watching and I'll see you tomorrow with a fun event.