All right, good morning, Cyber Traders. Happy Thursday, everyone. Welcome back. Market's closed tomorrow, looks like. It looks like it's actually passed over for a lot of people here. It's all victory shawley, happy Passover. Good to see you all. Good morning, Samuel, Ryan, everyone. Josh is here every morning. Staff is here. We're ready to move on. Looks like we'll get a little bit of down on the mark. We're down about 38 points. Job claims came in regarding, it was a little bit more than they expected, but you know what, today is gonna probably be one of the slowest days in the market. Now, we had a couple of good moving stocks yesterday. We'll talk a little bit about it. We're getting a couple of good movers this morning. This OCTO is moving up pretty nicely this morning, $2 to 270. We'll check it out. We'll talk about some of the big winners, what we had so far as of yesterday. We had this unbelievable run in SCLX. I mean, this thing was just a rocket ship. I mean, right here in the middle of the day, it goes from $9 all the way to 16. This is why we come to Cybertrad University. This is what we look for. I know a lot of you here kind of enjoy it. Even if you showed up late and you got our tweets, or maybe you got my alerts being in the trading room, if you jumped in and saw it at $12 or even $13, still got that nice little run going in there. But we'll talk a little bit about that. We'll talk about the iceberg orders and everything. I want to start off just talking about, I mean, regarding about my meeting at NASDAQ, and they told me something very interesting. I thought I really wanted to share it with you. NASDAQ is kicking the idea around that they want to have another opening of the stock market. And they're just kicking the idea around, I don't know, what are you guys' thoughts on that? I'm just curious. So basically, I want to do it because we have a lot of people, they want to explore into the Asia market. Some of you know that I trained a lot of people in East Asia how to use NASDAQ BookViewer and stuff like that. So they wanted to do an opening around like two o'clock in the morning till like another time, try to get that time zone because there are some stocks that do want to trade overseas, trade on the NASDAQ market. We know the market trades in pre-market, but always want to get kind of your thoughts what you guys thought about that regarding that. Now for us, does it make any difference? Listen, there are people on West Coast. There are people all the way in Europe.  So something that I look a bit kicking around, but if I find out, I'd be happy to kind of break the news to you. But anyway, regarding about today, like tomorrow the stock market is closed. Happy Easter, if everyone's gonna be celebrating Easter this week and good Friday, but I'm not expecting to be a very exciting day. I mean, I'm just looking at the attendee list right now. My son is home from school today. I know in New Jersey here, kids will go to school this week. So, you know, but overall, we had a couple of good movers. As in the pre-market goes, what's going on with audio? Oops, that's the audio. So what we have going on regarding about going into Good Friday and see what's happening on, you know, having the market close yesterday, just be very careful today. Today I don't think it's gonna be a super exciting day, but just kind of repeat what I just said. I had a meeting with NASDAQ yesterday and the big news with them was they might consider doing this 2 a.m. market to accommodate the Asian market. And like I said, I'm always curious to see what you guys feedback is. You know, what you guys think. Just out of curiosity, I know I put it in social media, but what everyone inside the group, what do you guys feel about, you know, the market opening up at 2 a.m. in the morning, just have another opening? What do you guys think? Think it's pretty good? We think it's pretty bad? Well, for us, well, for us, the reason why you're saying it's a little too early is because Manuel is because we're here more in Central America. So yeah, sorry, I will be sleeping. But you know what? I think it might be a pretty cool idea because I think what this will do is it will cause more volatility when you wake up in the morning. It's like something might trickle in that was in the Asian market that might kick into here. But that's really, you know, that's our goal is the volatility. That's all that matters. But like I said about these a couple of stocks, let's go over and see what moved as of yesterday and then we'll go through the watch list. So first of all, we're gonna get a big run up right now. There's a stock that's running up right now. I had on my watch list is the one from yesterday and she just made a big move right here is the IFRX. So this one's going back on the watch list. This had a really nice move yesterday, a $4 stock up to seven. And you know, listen, here's my watch list.  You can see it right there. That was our watch list yesterday. You can also on Twitter, if you subscribe to us and if cyber group members, you know you have that on your alerts. But the stock moved pretty nicely. I just saw a big spike right here in volume. There was a nice little iceberg order showing up right around here. It was holding here pretty strong. You could see it here around $565 and made a nice little move to about $6. Decent pop. Look, buyer just showed up right now. You see right here, look at that. Buyer just showed up right here for $14,000 at $576. What does that tell us? Right between the lines, fellow traders. All right, another stock that made move pretty nicely was a really big jump up and it just tanked. Was the BURU. I don't know what the hell happened with this thing, but this thing had such a great run in pre-market and then it crapped out. So I don't know if it was on your short list, but it sure turned out to be that way. And you could see it's already on the big losers list. Another stock also took a very big hit. I think they made an announcement. There was a news on this stock. Morning, but news came out and direct offering of shares. Right, so it looks like they're coming out with some more shares. The company needs to raise money to make back. That's not working out too well for them. So you could see it at how that worked out. Now, does anybody have kids here? Did anybody know what movie came out yesterday? Like my kids called me up and they went to go up to New Rochelle up in Westchester in New York. And they had these 4D movie theaters. He said it was like going to Universal Studios. Mario, right, Mario Bros. came out. That's right, Andrea. A nice little Italian friend there, Mario. So anyway, a couple of stocks are moving from it. I don't know if this has a lot to do with it, but we always talk about this. When there's a big hit in a movie, sometimes you probably have no interest in it. But AMC had a nice little pop this morning. So listen, people went to the movie theater. Obviously the movie theaters have big competitions now that you could stream stuff at home. So they took a very, very big hit. And you know what? They had to step it up. But I thought that was really cool, the 4D movie theater. I think that pay was like $25 to see the movie. They said it was unbelievable to another watch another movie ever again, other than back at that movie theater, being in 4D, especially like a Mario Brothers. And I think they even expanded it to kind of build on that video game.  I mean, that game is literally, I played that game. I remember playing the first one. And you know, when it first came out and it's still around today, after what?  So anyway, now that it's starting to get into the movie, it did really well. And AMC looks like it's benefiting from it. So if you wanna know why this stock is up, a speculation for me, that's the reason why. And you could see is the second biggest percentage gainer on the New York Stock Exchange. Another stock that's also making a nice little move right now this morning, which is a little bit more nicer trend. The stock obviously moved from yesterday to, is this stock right here, F-O-A. Now I just noticed that there was a big seller that was out there at $1.87. He just got filled, all right? Bonham is picking up. There's been a big iceberg order. It's been holding here for a while at $1.70. You could see it has a nice little push. We got some resistance levels right around here around $1.83. I don't really see that many resistance. I guess that was that, you know what? I'll show it to you. See a big iceberg order right here. It looks like it got hit. It came down a little bit, but it's trading within this range. You got a lot of effect that's going on right here. You see that big resistance levels right around here at $1.86. Those where the orders are, and it looks like right 90 right there. And you can see it got our resistance levels back here in January. But nice little moving stock, you know, cheap, which we always like, remember, which is here to make money. So definitely want to check that out. Another stock I was a little nervous on this, spreads pretty good, 3.6 million shares traded, $2 stock, $2.70. It's trending down in pre-market, but we'll see how she opens up. Not terrible, up 31%. And then at the second biggest percentage gainer of today, what do we got at 11% on the NASAC? That is horrible. All right, so not really that much action going on in the market. So we don't really have a very, very good list, which is very understandable.  And that's basically what we got going on right now. I think there was a court ruling on AMC also. Yeah, coincidental, Michael, they probably do it exactly when the movie came out too, right? But I heard it was a very, very, very big hit. But getting back to what I told you earlier, if you logged in late, I had my meeting with NASAC yesterday. We were talking about redesigning and working on some new ideas on NASDAQ Book Viewer. A lot of you see me on NASDAQ, on Joe Marin Jones' show, talking about Book Viewer, which is very similar, I just showed you here. And it's just very interesting conversation, you know, regarding about coming out with another second opening in the market, tackling China. And I think we're going to capitalize on that. I think we really could. And you know what's even better about that is that if they do do that, you know that a lot of trade, it's like the start of a new exchange, a lot of beginners, no one knows what's going on and you won't get as many shakes. So I'm looking forward to that. I know it might be early for some of us, but I'll keep you posted and you'll always hear it live here. All right, now listen, today at 12 o'clock on the top left-hand corner, you'll see a little QR code if you're on social media. I'm going to be doing a open house today at 12 o'clock. If you want to come join us for 45 minutes, I'm going to teach you how to use level four. I'll tell you a little bit about how to trade the volatility of today's market, how to find these stocks. Just take your phone and just scan it. Take your phone, scan that little QR code on top left-hand side and we'll get you registered for today's event. With that said, everyone, let's get ready for the market open up. We got a little bit more than 15 minutes. And before I go, I want to wish like I said, everyone a happy Easter, happy Passover. And if I don't see you at the end of the day, that's what you train for. So go enjoy yourself, all right? We'll see you back here. We'll see you back here at 12 and at 2.30. If I don't, see you back here on Monday. Happy trading, everyone, and trade safe.


All right. Thank you all for coming today. How many people flew into Portland or drove more than say 100 miles to get here? Wow. Wow. Portland is my hometown. I was born in a hospital a couple miles north of here. Went to high school a couple miles south of here. Went to the University of Oregon. So I'm a local guy. Thank you all for coming all the way here for this conference. It's been a great conference and here we are finally to the end of it almost. I'm kind of new to using open source software. It's been a few years and this latest project has been the most intensive thing I've ever done with open source. And I think the power above everything to me is the quality. And that's the passion of the people developing open source software just to make sure that it retains that very high quality and it makes it just wonderful to use. A little bit about myself. I started developing software actually about 37 years ago. I've done, I did some CAD cam in the early days of mechanical CAD systems where we drew vectors on a screen before there was pixels. Integrated circuit CAD. I've spent 28 years doing different kinds of embedded development from 3D computer graphics systems to things like live television and video processing for, for example, this projector might have some of my firmware in it. And also 2D and 3D printing. And in my current job I've been there about three and a half years and I'm leading the investigation into Xenomai. So today I'll go over a little bit about 3D printing technology and I'm passing around some 3D parts. Those were printed on the printer that we developed locally here just a few miles south of Portland in Wilsonville is where the 3D systems branch office is that develops this print technology. So there's some parts going around. I'll do an overview of Xenomai to maybe some of you it's just a refresher maybe some of you are new to it. I'll give a typical use case in this case a PCI device driver. And I'll tell you about my practical experience basically taking an existing very large code base that has been out and running software that's out in the field that's been through lots of hardening iterations. That's running on commercial RTOS and taking that code and running it on Xenomai under Linux and then hopefully there'll be some time for questions. I work for 3D systems. It was founded actually over 30 years ago by the inventor of stereolithography. And that's a technology that also uses vectors that takes a laser and draws it in a vector layer by layer by layer building up a plastic part. Later they developed selective laser sintering which is again a vector based layer. And then they started developing I guess it's 20 years ago multi jet printing using an ink jet style print head and this is a pixelated technology. And so like these screens here and everything we're used to everything's pixels and in the 3D world they call them voxels but it's basically 2D planes one after the other. There's lots of different technologies that 3D systems develop stereolithography selective laser sintering the multi jet printing color jet printing digital light processing which is basically stereolithography but instead of a vectoring a laser you use a projector and you project UV light to harden the resin and direct metal printing. Most of you how many people have done 3D printing maybe have a 3D printer probably what you have is the fused deposition modeling FDM type of printer with a head that moves around and has a filament. Actually 3D systems did offer one of those called the cube and we no longer offer that so we actually don't have an FDM printer that we sell today. This is the printer that we developed in Wilsonville it's multi jet printing it uses a print head that has hundreds of jets 880 individual jets firing at something around 40 kilohertz and so these little nanoparticles of material are being laid down in a layer of voxels and layer after layer until you build up the part. We take as our input a 3D model and internal to the printer we geometrically create slices and those slices are those pixelated layers or voxelated layers. So there's parts of that layer that's air parts of it that are the plastic material and parts of it that are the support material that are laid down to support the parts that might overhang. And the support material that we use in our technology is a soft wax and it can just be melted away so that's very convenient for sort of post processing you don't have to break off little bits or do anything. You just heat it up a little bit the wax melts away and you end up with your part. Here's a video that our marketing department put together to talk about this technology a little bit more graphically.  Building it up with wax to support the parts that overhang and plastic for the part and there's a lamp that fires to harden the plastic. And I think the rest of this video shows how easy it is to replace the material so just pop it out put in the new material. Each jug is only a few hundred dollars so it's very inexpensive. These printers are intended for like an industrial use typically so very high resolution devices very high resolution parts that are printed. And they can also be used to create castings for metal one of the big uses also dental they're used to produce dental like crowns and things like that. And the power of this 3d printing even though it may take many many hours to complete your part through the 3d printer to mechanical designers. That's way way faster than the maybe multiple week turnaround times they would get to actually get prototypes that they can test for fit and usability and things like that. But you can kind of tell by looking at this imagine all the real time systems that are here and and these systems include motion control of all the different motors, the thermal control. There's things have to be heated up some things have to be cooled down. The curing of the resin is exothermic process so we have to get rid of that heat. We have material delivery delivery so we're sensing the levels and the printhead and and in the bottles and things like that. We have to pump pump it through and open up valves and gates and the printhead itself has to be controlled and we to control the jet fire and the data load. And the before printing calibration things and and also as the data comes in it has to be reorganized because the printhead has an interesting geometry of orifices. And as that as those jets are firing during the motion you need to organize all your pixels so they're in the right place at the right time. It's not just a simple five hour something that where you just push the pixels through you have to reorganize them in real time. Now in this case also while all this real time is stuff is going on we're running this slicer that's taking this big geometric object very potentially very complex geometric object. And geometrically slicing it and figuring out what might be above it doesn't need to support and you know what are the boundaries of it and all that so. That's the slicer is in is very compute intensive it uses lots of RAM it uses lots of floating point. And it pretty much takes over the processor in our current systems. We have a Windows PC basically that's running the slicer that's embedded PC in the in the printer. And then we have another power PC 32 bit controller that runs a commercial our toss that runs all the real time systems. So for my project my technology assessment and this is not for any particular product to go out the door yet but just to see is it possible. Can we run this hard real time controller control on a on the same controller that we're doing the compute bound slicing. And that's a fairly big challenge and we chose Linux with Xenomai because and Xenomai specifically the co kernel version of Xenomai because that provides us this hard real time capability even on a system that's heavily loaded. And you can imagine these print jobs some of the big print jobs might take a day 30 hours. And if 29 hours into it you miss a deadline and you have to abort the print that's not going to make the customer very happy so we really absolutely need hard real time response and it has to be extremely reliable. In our early development we had a secret weapon and that was Philippe Jerome he's the originator of Xenomai as far as I know he's one of the main contributors to it. And he also does consulting so we hired him to kind of get ourselves kick started because personally I I'm not that familiar with open source development and I'm definitely not that familiar with Xenomai. I read some of the documentation but having Philippe along kind of give us a quick kickstart. We gave him all our embedded code and he kind of massaged it into into Linux land for us. He created this auto tools based build environment which is what Xenomai currently uses and so that was the natural thing for him to do for us. We could also do CMake or whatever but it's we use we're using auto tools and he delivered these real time driver model drivers and API functions to access the drivers based on this original code that we gave him. And again what he originally gave us was kind of an example that we used and then we looked at it examined it and then kind of threw it away and started over. And actually this was a great way to get started but anybody here who's interested in doing a Xenomai project you don't have to hire Philippe. You there's great there is pretty good documentation and there's great user community support and when you get the Xenomai download it has like example programs and test programs and things that you can look at as examples. And if you've done kernel development you'll be a lot more comfortable doing it. So to get started you have to make a decision in Xenomai 3 you have the choice mercury or cobalt mercury is single kernel cobalt is the co kernel what most people think of when they hear Xenomai is this co kernel. But if you don't have any hard real time constraints and you just want to use the libraries that Xenomai provides then you can use mercury and you can put that on any kernel and you can have the preempt our key patch and you can then get the benefits of the library some of them like that emulate VX works or emulate PSOS and some of the other libraries like that. If you're hard real time then you want cobalt. You patch the kernel with this thing called I pipe this interrupt pipeline that basically takes over handling the interrupts so that you get that real time response. And then you want to select a Linux distribution that's compatible with whatever patch you're going to use. And also note that the single kernel is only in Xenomai 3 so if you're familiar with Xenomai 2 that was only the co kernel so mercury is new in Xenomai 3. So you start with getting the kernel source and after being to this conference for three days I think everybody knows how to get a kernel from kernel.org or you can in this case I'm running an Ubuntu distribution and I can just grab the source for that distribution. And if I do the first method I can get the version of the kernel that exactly matches the patch that I want to apply. Now I could take a chance and try to apply a patch that's close to the version I have with my distribution and I've heard that may work but I've only tried it the first way and it's worked really well for me so. This is an example when you go to Xenomai.org and you see the different patches that are available you can see the kernel versions here that are for X86 version 4 of the kernel the different kernels that the patch will be applied to. So for example we were using an Ubuntu distribution just to get started we started with 14.04 and then later we moved to 16.04 and so we upgraded to a newer patch when we changed the distributions. To get to Xenomai source you can clone it right from Xenomai.org or they also have tar balls that you can download and then put onto your system. I like cloning the repository with Git you get the updates you can there's a stable branch and you can pull from the stable branch and get the latest updates as you're going along. Or you can even if you use the Git source there's tags and you can check out a tag and just work off the tag also. So Xenomai like I said is using auto tools so if you're familiar if you've done auto tools builds and I think people that do work in Linux probably have a lot of exposure to this. Typically bootstrap and that creates a configure file you run configure then you just say make and make install and then you've got Xenomai is now built but it's not in your kernel yet. You've just got the executables of Xenomai and the libraries of Xenomai built. Then you want to go get the I pipe patch whichever one corresponds to your kernel. They recommend using W get sometimes I'm having trouble with W get and I read somewhere that curl is more reliable and I've had success with curl but occasionally I've had failures trying to use curl to get it and I don't know why but it's a network thing. Either one works generally curl I found is more reliable. Then there's a prepare kernel script and you run that and then you configure the kernel and for me and I haven't done kernel development much before I've done a little bit of driver development in a project where there was already a driver and it just needed modification and I didn't have to configure anything or whatever I just look at the code and see where it needs to be modified and rebuild the kernel and it was pretty easy in this case I had to configure. And when you first come up you'll see something with lots of scary warnings saying oh if you have page migration on that may increase your latency and you might probably want to turn that off and I had to learn how to use the slash search for term in menu config and find what page it's on and turn it off. So if you're familiar with menu config it'll probably be really easy for you. It was a little bit of an adventure for me and but this is that kind of a one time thing you get started you configure your kernel and then you're good to go after that. And and I learned a lot about how to configure a kernel by going through this process so it was all good.  So you can like put a little some number of characters so when you say you name dash R you can kind of see oh yeah I'm running the special Xenomye version that I wanted to. And then you can then the next thing typically you can do is just to make sure that your kernel is actually patched properly you can run this latency script and you get some nice numbers out of it. It kind of gives you this warning I built with debug and so I have quote unquote high latencies which are like in the forties and fifties so that was pretty pretty much OK with me. And the other thing to keep in mind is that when you build a Xenomye application even if you build it as user Fred you have to do pseudo run your application as super user so to get into the into the Xenomye libraries. So now you've got your kernel up and it's got the co kernel you've got the cobalt core there and then you've got these RTDM real time driver model drivers. And then you have some libraries on the user space side that your application can call into to get to those drivers and. So. The real time driver model this is this is intended to be a unified interface for both users and developers of real time drivers and it's the Xenomye programming interface and it was also it's also used by this real time application interface RTI out of Italy. And when you create a device with RTDM it's in this dev RTDM RTDM structure. There's RTDM functions in the kernel to register IRQ handlers. There's ways to actually register your devices and here's the typical use case for a PCI device so let's say you have. An FPGA somewhere and it's and it's on a PCI bus and so you when you boot up the Linux will go out there and kind of probe the PCI bus and and then you can basically get called back and say hey. Here's this device that matches the vendor and device ID that that you want. And then this is just standard Linux there's no Xenomye in here at all this is just standard PCI driver stuff where you want to enable device and request the regions and. I get the start address of the bar and maybe I'll remap the bar into memory map space and but then once you have all that then you can call this RTDM function and you just say I want this handler to be called whenever this interrupt happens. And that handler is going to be called in this real time domain and. So. This is how you register the device so you can create a class and then you can use these RTDM functions again to set the system class and register the device and this is with an RTDM device. Data structure and then you then. You want to set up the device structure which references the driver structure and that gives you the basically the path to your device that you want to. Open and send write commands and read commands from and all that stuff. So a little more detail this is the RTDM driver structure and you set it up with what what kind of device it is and the kind of the key thing here at least for me is that there are real time and non real time versions of things like IO control and read and write. And so you can your device can act as a real time device but it can also act as a non real time device at the same time the same device. One thing to keep in mind is that if you're going to have a process is going to be accessing the non real time part of it you don't want it to lock. You don't want it to lock the parts of your device that the real time is going to want to access that could be a bad collision there. You can also create a POSIX compatible clock so let's say your device has a hardware clock. You just write these functions to read the ticks convert the ticks to nanoseconds or just read the ticks as nanoseconds or read monotonic in nanoseconds to be able to set the time to a specific value in nanoseconds. Convert nanoseconds to ticks and ticks back to nanoseconds and then ticks to nanoseconds with rounding and program local shot and global shot which is basically saying this is when I want to wake up. So this is the time in nanoseconds when I want to get my interrupt and you this is how you do that you write a function to do that you populate that in a data structure. Called XN clock or Xenomy clock. You pop you just give the pointer so those functions. And then in your interrupt handler you want to call this Xenomy clock tick and that's basically going to when you get when you get a wake up interrupt is going to do what's necessary to wake up any waiting threads or kernel processes. Then you you register it and again this is Xenomy cobalt clock register you give it the XN clock structure and you get back a positive clock ID. And then you can use that clock ID in these basically what look like positive functions the timer create timer set time clock now to sleep just looks just like positive. So that's that's very nice to be able to essentially have a real time clock that's your hardware that you can access like you're used to doing with posix write a user space application. Here's a nice thing once you've got got all this set up then this to me this was kind of like an eye opener I did not realize that in Xenomy that most of my hard real time stuff is going to be happening in user space. I thought all of my hard real time stuff was going to happen in the kernel but that's not the case at all. Very little actually is happening in the kernel all the hard real time stuff that's doing everything is happening in user space in user space threads. So you you create your user space thread and you open your device that you created in this KB case the FPGA control maybe read and write stuff to it to get it to do things. There's also a real time thread create see in our case we create a couple dozen probably 30 maybe even 40 threads that are all doing motion control and thermal control and material delivery and head set up and data control and all these other things. And so with all these threads running we there's real time mutex is there's condition variables which we use as a way to pass information back and forth between the different threads. And when you get it going this is a case what you can see this old dusty printer they gave us we just ripped out its brains and just left the hardware alone. And we put in this x86 PC basically it was just a little motherboard a little x86 motherboard the motherboard cost us 50 bucks we got a processor and some Ram and plugged in the PCI bus to the printer and here we're running three different axes we're running a head maintenance axis Z table of going up and down and then the printhead moving left and right so. That was just an early example of like hey I can actually not only can I control this motor in real time but I can have three of them running at once and then more and more and we kept adding and adding and we just basically took this huge. You know with I don't know decades of development person hours developing the embedded control code for the 3D printer and again this printer is shipping this is a printer that's shipping you can buy today that uses a different kind of controller. And that code that's been hardened by going to customers finding bugs going through QA cycles and release cycles and it's a very well hardened code. This is now running on Linux with ZMI so we were able to print a beautiful looking part. And it was actually a big surprise along the way we ran into some problems and these are kind of problems if you're doing ZMI you may run into yourself. Check the time here I guess we got plenty of time left good. So thread relaxes so. Your threads run primary mode is the real time mode secondary is running just like a Linux any Linux thread and this is really nice actually because if you need to do something in the Linux world you just do it. And the thread becomes the Linux character and it runs in non real time mode and it does its thing. And then when you come back and you access your real time functions boom it pops back into primary mode and it's a real time thing if you want it to always be real time. Don't call those other Linux functions. But when you transition from primary to secondary it's called a relax and sometimes you have a spurious relax is because something happened. That caused it to go into secondary mode but you weren't really expecting it to and these actually are you get notified by ZMI. If you look at the syslog you'll see you know relax on this thread and it's very useful and there's another utility that where you can turn on and get a log of all the relaxes and get trace backs of where the relax happened. And so this is very very useful. An example where this could happen is let's say I'm doing all this code almost all this code is C++ by the way this embedded control code was all developed in C++. So it's kind of handy to use a collection class and then if you start adding things to the collection class and it needs to grow if that collection class happens to be on the Linux heap. Then it's going to make a call into Linux and you're going to relax into secondary mode. There's you can create memory pools that are real time memory pools which is what we do now. But you get these relaxes happening just by saying insert something into a into a map and you're like how'd that relax happen. The other thing is if you memory map a file and then you have a pointer and you're actually accessing the file like you would memory. But the file system is Linux only you don't get the file system in real time mode. So you'll get a relax and sometimes just referencing that will cause a relax. So in our case we had this issue where we had a non volatile parameters we kept them in a file. We memory map that file so we could get to all the parameters like they were in memory but they're kept in a file so they didn't go away when we rebooted. And every time we read from that memory that memory or wrote to that memory we would get a relax they're happening all the time it's like oh we got to fix that. So we made a copy of that memory and so we just access this cache and then we found this really cool thing called the cross domain datagram protocol. It's part of Xenomai and it's where primary mode thread can push data out to a secondary mode thread. And so that was nice when we need to update our file we push it through this XDP socket. We never leave primary mode and the other side never leave secondary mode it's reading from this socket it stays in secondary mode it writes to the file in the background and everybody's happy. And if it's writing to the file and we need to do something now Xenomai takes over and does this thing and we go back to what we were doing. So it's very nice that solved it no more relaxes no more weird latency issues that just fixed everything for us really nice. We had a case where we wanted to do a time stamp of some material like when a bottle was put in and so we wanted to like time stamp when a new bottle was inserted. So we use this get time of day call in the kernel in primary mode and that works great because you're just reading from a location in memory somewhere at the time of day and it works fine. Memory access is no problem but sometimes the OS has to update the time of day and if the OS is updating the time of day you kind of go into a little spin lock until the OS is done. But the problem is I'm in primary mode Linux can't run when I'm spinning and so Linux can't ever update so dead and so we put in this little change to time stamp and everyone said I want the thing we just die and so we when we roll back to change that lock up went away so we kind of knew all right somewhere in these changes. This is problem we were to track track it back to get time of day and a big help by the way was the Xenomai forum it's like an email for my if you use Xenomai sign up for the email form it's extremely useful. And just searching through that forum there was somebody had a similar problem and explain the whole thing of what was going on so I like goes on I see what's happening. There is a real time clock get time call as part of cobalt and there's special type of time called clock host real time so that was our solution it's a different way to get the time where we didn't. And that that's in user space but in a kernel space there's actually an RTDM call that just gives you nanoseconds since the system booted so that's that's pretty useful. So the NK lock is an internal Xenomai lock that kind of protects some of its internal data structures. And so we also had our FPGA could have multiple threads accessing and simultaneously and so we wanted to block out certain references to the FPGA with this RTDM spin lock which is a very very useful kind of lock that lets you. Make sure that you're not clobbering yourself on multiple threads running. And so that's how we protect that resource with this RTDM spin lock is very simple and easy to use and very and it works fantastic. And then while we would have the FPGA resource lock maybe we use the Xenomai select signal to wake wake up a thread that's waiting on us on a select. Down under the cover somewhere it for a moment locks this NK lock internal Xenomai lock to do its thing for select and then it unlocks it and then we come back and we unlock the FPGA. But on the periodic timer when we enter the periodic timer function NK lock is already locked. And so then if we lock the FPGA we get the ABBA locking issue and boom.  You were asleep for more than three seconds why and it'll just like send you a signal. So to avoid it we we could have say that whenever we have the FPGA locked we just don't use these Xenomai or RTDM functions so that we don't ever have that problem of locking NK lock with FPGA locked. Or in the periodic timer function we just don't lock the FPGA and we only do things in the periodic timer that don't require us to lock the resource. Or we could build our own timer interrupt and not use the sort of built in Xenomai timer feature and then we don't have NK lock locked. We chose the second one our periodic timer thing was like a watchdog timer that had you know poked a register and we're the only thing ever poking that watch watchdog register so you don't need to lock the FPGA in there. We had this interesting issue where in this code base that we inherit you know that we brought in this big code base. It has the user space timer capability that's kind of a library function and so it it starts up all these timers and when it gets a timer interrupted basically says every timer that was set to go off before now I'm going to wake up. Even if maybe this isn't my interrupt I don't I don't know that this interrupt was associated with my timer but I know that if the time is T and somebody requested me to wake up before T I'm going to wake up all those things. But we had an issue where you say I want to wake up at time T and it converts that nanoseconds into ticks that rounds it down a little bit so when we finally get woken up the now time is actually before the time when we wanted to wake up. Which is not unreasonable but in our case we wouldn't wake up anybody would say oh the timer came in and nobody's waiting so then we'll wait for the next interrupt which then never will be one and we would just stop and go how come the timers aren't going anymore. That was a pretty easy thing to fix we just said oh when we read the current time we add one hardware tick in nanoseconds to that now time. What's interesting about this what I found really interesting is that on our on our proprietary system it had the same problem it rounded down and everything the same way. But the latency was big enough that that little bit of rounding down never was a problem because by the time we got into that user space function that compared to now and when I wanted to wake up time it was always after. It was always after the latency was big enough that we never had this problem in Xenomai it was very fast so yay for Xenomai for being fast. So let's see we have our entire embedded control code base which is actually all hosted with in Git on our local Git server. It will now run on Linux and Xenomai that embedded control all runs and we can we can do everything we can print and do everything we need to do the whole code base is running. The old controller was a 32 bit big endian machine the new one is a 64 bit little endian machine commercial RTOS and they're actually compatible so that's very nice so we can we can build that we can actually maintain the two code bases. It's one code base we can maintain the code base and the two systems together. So the key thing was to bring in the slicer so we brought in the slicer as this was running on Windows we built it on just as a standard Linux app and we let it rip and it takes over the whole processor. I mean it is loading that sucker down this slicing away as fast as it can and there was never a problem with the real time I mean Xenomai worked extremely well I've run jobs for two day solid huge print jobs there's never a glitch now granted that as my hardware guy tells me who's a little bit nervous about you know an unsupported operating system. It's like well we have three printers running now in the lab and it's like three printers a very small you know sample and you've only run it for a few months and it's like the fact that you had no problems isn't good enough you know but I think it's pretty good it's been an extremely successful. Investigation now we have to get past the oh what are the legal ramifications of open source software what what are the support ramifications what are the how do we fix problems when we're in production all these things but anyway any any questions. Yes that question is that we analyze the cost difference and we have the Windows license I'm not exactly sure but I think it's like around $150 per machine I'm not sure don't quote me on that well actually I think I'm being recorded so I do know I do know that our our commercial art toss costs us more than $100 per printer that we ship more than $100 per printer so I think our total cost just for licensing. Is like $250 or more and this. As if you were at the open source. Sort of licensing talk that was here a little bit before me. It's I don't ever want to say that Linux is free you know we there's a cost to it we want to contribute back to the community but. The bean counters will like it. Yes so the question is why are we running slicing on the same process or they were doing real time control and hey why do we even need real time control because. There's lots of smart motors and other things that don't really need. You know hard latencies and those are good points. The reason to run the slicer on the printer is when it's slicing it's slicing in a way that is very particular for that device so it doesn't just slice up the model. It has to slice it into the exact resolution and pixel dimensions and everything that are for that printer or that printer class. It has to understand how the supports fill in and things like that now we could do that on the host computer that is like that that's driving the printer could do all the slicing on that. But what this provides for our customers is. They have we have a thing called 3D sprint which is our our application that runs on your desktop PC or your laptop.  And if you want to start another job the exact same job on another printer you just select another printer you say go print. And so they may from one little laptop be controlling 20 printers that are on their shop floor. That they spent millions of dollars for these printers. You know they're now they could have a separate laptop for every printer. And so each printer has a dedicated laptop but this this is something our customers have found to be very useful. And for us it's less of a maintenance headache when we release the 3D sprint application. It doesn't have to know anything about how the slicing works. It just has to send the send the job to the printer and the printer figures out how the slicing works. So that's why we do that. And again in the current application we have a windows slicer and then we have a separate thing that's running the real time control separately so they don't have to interact. And we could keep doing that. Our hardware guy really wants us to keep doing that. So he wants to develop the next real time board because that's fun to develop. But we'll see how it goes. And then as far as do we really need hard real time. That is actually a really good question. And I went to a couple of talks at this conference that made me kind of question how how would this work if we didn't have Xenomai. I suspect that we would get the occasional glitch now. So part of this is that we're running at an existing code base like I said that has a huge legacy history. This code has been developed over a long time with lots of developers. It's very well hardened. And so we like to keep this code base active and it does depend on hard real time. We've tried we actually tried running it just as an application and it didn't even come close to working. So yes. Yeah, go ahead. Yeah, so the question is how many slices ahead do we need to be processing as we're printing. And there's two answers to that one is the slicer generally on most parts will run quite a bit faster than the mechanism can print. So we're getting slicers maybe we're getting two or three slices per printed slice. And so the slicing gets completed well before the prints done in turn on the embedded side. We have chosen because because this kind of gives us a good amount of buffering 16 layers. So we we bring in and we start processing and we have 16 different threads running the processing we need for the data on 16 layers. The throughput is way fast enough to handle a layer per printed layer. But there may be some amount of buffering that's required so that we never get starved. We can't allow to be starved. The question is do we do any kind of on the fly correction for you know errors in the positioning of the of the platform things like that. No, actually we do not. We require our platform to be exceedingly high resolution on its motion. So we have a platform that we can move like one two thousandths of an inch accurately. You know it's very very high resolution on that motion. Yes. Yeah. So basically our generally our deadline or our window biggest latency that we can accept is on the order of 50 microseconds. And generally speaking the preempt our key patch will give us that but it's not 100% guaranteed. And so we could miss a deadline occasionally. There's ways that we can recover from that and a lot of the things like the thermal stuff if we miss a heater control operation. The heater control operation are happening at half line cycles so that's happening at 120 Hertz so that's you know pretty slow anyway. So we don't really need hard real time for that. But the problem is that we we've had the experience where even on the hard real time system on the commercial RTOS if we're not coding the priorities right things like that. We'll still miss a deadline. And when that happens it's not good. We when we were originally a lot of this code was developed for printing solid ink for color printing. So it's a similar print head and technology that's for that. And if you print a bad page. Well you can kind of reprint it or something you know kind of get around that but this this is not I mean if you if you miss print a layer it can leave a bad feature on it and it's not acceptable to the customers so. Yes. So the question is how many engineers how many firmware developers essentially are on our team. And do we require or does you know my require a multi core processor on our team. Well originally it was Philippe was working as a kind of an consultant. It was just me and then eventually we brought in a second guy and the two of us were able to take this extremely large legacy code base and get the whole thing running and making these beautiful prints. So we were very it was really some of the most fun I've had. I mean it was a great little small team and we got an amazing amount of work done in a small amount of time. And it was a ton of fun. And by the way I was going to say that we all are super lucky to do what we do. This is really a great job to have is to work in embedded systems and it's so much fun. As far as the Xenomai Xenomai would work on one core. In fact we thought it was using all the cores of our four core processor. But then when we started looking at the at the affinity it turns out that by default Xenomai has sets the affinity to a randomly selected core. And then as you as you spawn your threads all the spawned threads are on that same core. So we looked and there are 30 threads we're all running on one core. And it was still working fine. So we thought well that's you know maybe we could spread it out but there was no reason to. And then the as far as the slicing application yeah we could set the the slicer to run on the same core as the real time is and there would be no problem. I mean it is running on this it is running on that core as well as others but we could run that slicer on a single core and Xenomai. That's the really great thing about Xenomai it's I mean I'm writing user space code and it looks just like user space code it builds like user space code. I lots of the threads I'm doing are true Linux stuff manipulating files and talking to the network and doing all the things I need to do. And this is a quote unquote a Xenomai thread but it never goes into real time space but it's communicating with my real time threads in a very easy way. It's just a beautiful I loved it. The art we're using Intel i3 and it was running at like three and a half gigahertz didn't need to be fast. We were doing some measurements comparing you know what it was before and after and it was like 10x faster on some of the stuff we were compute intensive stuff even though it was running on only one core. It was still 10x faster than the commercial art so it was like wow we have way too much power here so we could dumb it down a lot. Yes the question is are we looking to move to like an arm processor cheaper hardware etc. And the answer is I'm not allowed to talk about that so you can guess. So personally I really I'm I don't have any problem with running on an I3 I think it's fantastic I love it and these printers are selling for 40k pop so why not you know why why do we need to. But. It's. I can't talk about what's going on in the labs. Another question yeah so no we don't have the question is do we have any other sensors to kind of see how the prints going along and kind of make sure that everything's. You know correct without having a I think this is I'm I think I'm getting the hook. We have an end game to go to which I don't I've never been to but I think it's going to be fun. Thank you all for coming today thank you for your attention and also thank you for your contributions to open source because it's fantastic I love it and I want to keep using it thanks.