Hi. Hi. I'm Timothy O'Malley. I'm with EEG, where I'm EVP for US Business. And this is beautiful. It looks like print, right? It does look like print. I would like to show you our latest Spectra 6. It was announced earlier this year. This is the most advanced color e-paper product line that we have, and you can see it looks like high quality print. It looks amazing. It's high contrast, high saturation. We expect great things from this. E-Ink Spectra 6. Does that mean there's a 6th generation? In this case, it's the set that there's 6 primaries that we're using. So these images are composed of 6 different primaries that are grouped together in order to reproduce full color. Is there what you call that layer that's sometimes in front to make it black and white in the back? And then you have the layer that makes all the colors. Do you still do that part? We still do that part. Our product is this electronic ink film. This film contains the different particles that when moved to the right location or mixed in the right way produce those different colors, those 6 primaries that we talked about. And when arranged appropriately, generate images of full color. And so that allows us to create not only this high saturation, but also these beautiful images. And how long does this take to refresh? These are refreshing in about 12 seconds, which you may remember some of our earlier product lines for high color. We're closer to 30 seconds. We continue to push the advancement there. So while they are much more saturated and with a wider set of colors than before, they're also faster. This is so awesome. Is there any chance this is going to be affordable and mass produced and all of the world and changing the whole world? Of course it is. As you know, everything starts with volume-dry price. So as we introduce this, as early adopters take it, and then it gets more popular and then even broader, price will come down over time. That looks awesome. Is it possible to have a smaller bezel in between? So this is our larger format, which does have a larger bezel slightly. As we introduce different sizes, each of the sizes will have bezels that's appropriate with our reliability testing. So over time, we continue to push those down. We find that in most of the installations, they want something that's sturdy and structural. And so this is a great first step product for us. Of course it looks completely amazing outdoors. It looks completely amazing outdoors. The more light you have, the better an e-ink display looks. It's basically the opposite of every other display that you have experience with. We shine outdoors. That's what I love about e-ink. You are getting people away from these lights, indoor lights, getting them to go outside, get some air in the fresh air, and look at these beautiful photos on the street. In both cases, if you were to see this on the street, you might not know it's an e-ink display and if you didn't happen to be there when it was switching. That looks like printing that much and it will just shine outdoors. There needs to be a calculation where you say, okay, that's the cost, but because of this functionality, it's worth, with the advertising income, if it's an ad banner, in an X amount of months, it paid for itself, or an X amount of weeks even. There's a number of things that go into that and you may know that in different parts of Europe, they're putting restrictions on what signs can operate in certain outdoor locations, primarily for energy use. So an e-PD and e-ink display at very low power doesn't use any power when it updates, helps bring that capability to more locations than you could have with a high energy consuming display. Combined with, we can run off solar power in most cases, which means that you don't have to dig up concrete, lay out wine, power, in order to run the display. More locations, better total economics of use over time. Nice. All right. That's awesome. Yes. That's what's wrong. What are you wearing? Your t-shirt? My t-shirt, there's an image from CES this year, where we did... You were the best of the show. Oh, of course we were. Color changing car with BMW. BMW might tell you they were the best of the show. We're thrilled that we were able to work with such a partner. And this t-shirt commemorates that as does this on the wall over here behind me. So we've created an image that is referencing, referencing that car that we did in CES. It wasn't practical to mount the car on the wall, so this was the next best thing. You couldn't bring the car? No, we didn't bring the car. There's no lift over there. It's only an escalator for people, not for cars. The car is a superstar model. Its dance card is full. We couldn't get it for the show. Nice. So this particular Prism Street product line is taking that type of color that you saw we talked about before, and now putting it into a different format. It's bringing personalization and innovation to design. We want to enable designers to take color changing surfaces and build that into interesting ways to improve product design and improve that human element of interaction. I think the trick is how heavy is it? How expensive is it? Is it possible that it's not so heavy and it's not so expensive? So electronic paper is a very thin film. It's basically thin pigments, so it's extremely light. There's no heaviness to it. We do need to work on engineering in order to apply it to different surfaces, and we're working with partners in order to be able to do that. The CES show, BMW did a lot of work to apply it to the car. There are experts in that, and that's the type of integration I mean, where we're experts in the color film. We work with people who are experts in product design, and together we create something better than the parts. It's just a few kilos. Maybe not even that. This is very lightweight. I would not focus on the weight. Focus on the cool factor, the emotions it generates. When you see the color changing car, it creates a lot of emotion. I love watching videos on YouTube of architects. I like the Danish architect called Björke Engels, for example, but it'd be awesome to just be an architect and have this tool and be like, I want to do this crazy building, and I know there's a few projects already, right? There was a San Diego rental car center where they copied an old design from World War II called Dazzle. In World War II, the Dazzle design was to hide battleships on the horizon by creating different angles that broke up the surface. Because they have a base out in San Diego, they wanted to lean into that type of aesthetic, and so they put these panels on the building to recreate that kind of feel. It was very nice. They're all independently controlled, separate panels. It was a fun project to do. Can you say a little bit about E-Ink trying to get the cost down for all these displays? It's all about volume. You just make more and then it gets cheaper, or how does it work? So for the most part, we have over a billion displays in the world installed today. So cost isn't one of the things that we get necessarily as preventing adoption with a billion displays installed. As with anything, the more that we can do, we continue to try and bring cost down. But we've been really pushing on functionality, feature set, and performance because we mostly hear the quality is what people want. Awesome. Yes. I love this show. Let's walk around your booth. Which way do you want to go? This way. We can go this way. I can briefly introduce you to some of our transportation, which is over here on this wall. So we have a number of signs installed in Europe, in Australia, in the US, so globally, where we have different aspects of transportation signage. So again, we're talking about either information or advertising presented to people where they need it in a low-power, sun-friendly, paper-like way. You can see how beautiful and high-fidelity these signs look and even better in sunlight. Nice. Yes. We've introduced recently our Kaleido 3 Outdoor product line, which is these two 42-inch large format displays are. So Kaleido 3 Outdoor extends our temperature range in order to support outdoor installation formats. That's why we call it Outdoor. And we've also, in this particular case, increased the size of the product offering up to 42-inch. Primarily these will be used for information type signs, as you can see here, and are perfectly suitable for transportation environment. Am I seeing front light? These both have a front light on them. In many cases, we think front light extends the experience. People need to see it in the dark and they need to see it in the light. With the Kaleido product line, it's a good fit with front light. So we tend to integrate it that way. But of course, a customer can make their own choice. What's the main thing about Kaleido?  And what we've been doing lately with Kaleido is continuing to optimize the light stack so to keep every photon possible so that we can improve the color experience. We've probably seen our displays over time with color filters, and they keep getting brighter and higher resolution, and that's what we've been doing, and this is what Kaleido 3 Outdoors is. Yeah, that's the word I was looking for, with a Spectra color filter. There's no color filter with Spectra. So that is... Spectra uses pigment particles, similar to what you would have with your inkjet printer, but we move them dynamically using our magic. And so you get a high saturated color from particles that are intrinsic colors, not a color filter. Each pixel can show any one of those primaries, and you have all the optical benefits of the full saturation of that pigment on the surface. Nice. And there's some more spectra, right? This is also the same. We have some of the smaller sizes and some of the larger sizes, but yes, this is the same. And where do these go, the small ones? Most of Spectra 6 is initially targeted at indoor. We think retail will be very popular, but also any place that people want to do some high saturated advertising, especially if we can have power savings and... So, yeah. Oh, you've moved over to the e-note and section. Is that okay? Should we finish that one? I'm okay jumping to e-note for now, if you want. Yeah, we can go back and forth. I love these devices. Yes. So, as you know, E-NIC is really looking at a paper replacement strategy. We look at places where people use paper today and say, how can we make that better? Of course, writing. I'd have to turn now and stand by. So, of course, writing is a big way we use paper today. And so, we've leaned heavily into improving our latency in order to give awesome writing experiences. And we've seen that by increasing adoption of e-note globally. So, I run into more people at conferences with e-notes that tell me how much they love their latest version of it, whether that's the Kobo Ellipse or the Remarkable 2, or we have a Kindle Scribe over here. Which one is the best? They're all the best. You cannot say, you work at E-NIC, right? I love all my children. Yeah, I like these guys. Books, I've been filming with books for ten years. I also love books. They make a great device. It has a nice Android integration, which is a strong feature that they particularly bring. That's not an e-NIC, is it? No, but the computer is connected to this monitor. So, one of the things that you may have read recently is E-NIC did a study with the Harvard School of Public Health where we looked at the health effects or the less healthy effects of screen time.  And so we looked at what kind of stress does that do on your retinal cells, on your eyes. And an E-NIC display with a Comfort Gaze Front Light, which is a blue light minimizing front light, is up to three times healthier compared to other screens. And so we're trying to make people aware that in some of your 9, 10, 11 hours a day in front of a screen, you can make a healthier choice for certain reading, certain content, whether it's writing or reading, or even if you are working on documents on a monitor, you can use it. I think that if people used 1% more E-NIC in the world, the GDP would go up 1% because people, the Netflix is not very good for GDP. What's not very good for GDP? Netflix. Oh, that's funny. Because people just watching TV shows... They make a lot of content, though, so they must... I love Netflix. The last three years I've been doing a lot of Netflix. But it's awesome to work in a company where you see people that use your technology to be more productive. Yes. For productivity. Yes. And we're very fortunate. I've always counted myself to be fortunate. We have a loyal fan base globally of people who love E-NIC technology and they follow everything we do. They post on different boards. They're about new products and they're loyal, excited, technical enthusiasts of the differentiation that we bring to the world. Fujitsu. Right. Yes, I think so. And the phone. So this is a light phone. In some cases there's a throwback initiative to getting back to some phones that are simpler that don't have us necessarily staring at them all day long. So this is a simple phone. That looks awesome. It's an Amazon, I think. People can just buy it. I think, yes. I think people can just buy it. Very affordable. It's a very simplified kind of like Android or something. That's phones and taxes. Yep. Just to make things super simple and hopefully very long battery life. Potentially. It should be. Oh, fossil. This is a fossil hybrid smart watch. So people are familiar with their digital watches, with their digital screens that do cool things and their fashion watches typically are analog and not digital. Fossil took the best of both worlds. They've made a fashion watch. It's beautiful. It looks nice on your wrist. But it still has some of those advanced features from the digital side. And that's why it's called the hybrid watch and people love it. And these guys, Norwegians. Yeah. Oh, it's such a nice feeling. Such a nice writing. Yes. And they have some great cloud integration on this one. The whole feeling. How it interacts with the material. Yes. They very much work on the surface feel and the pen tip feel in order to give you the paper writing experience, the feel of writing on paper and you still get the entire digital experience of all your notebooks are with you at all times. And Amazon. And Amazon. Yep. These guys have been shipping. Sorry. Shipping millions and millions of kindles. Yeah. And now they have. As of Q4 last year. Describe. They added describe. It still has all of the Kindle functionality so you can get to your library and read your books. If you make your fonts bigger and I have to admit now that I'm older I sometimes make my fonts bigger. You can get more words on the page. But this also integrates all that writing notebook capability that we've started to see catching on in other places. So it almost validates that this is going to be a significant category. And with more and more entrance in this space I feel like it's potentially helping to bring the price down to hopefully less than half an iPad, you know. And then it starts to become a no-brainer especially if there's a right cloud functionality. Yeah. I see people in different places on the plane or whatever to say they love paper and they really want to stick with it. And then they have seven books. Three books to read and three books to write in and you can have all that in such a tiny amount of space and you still have the paper-like look and the paper-like feel. It's a compelling story. And yeah, we are working constantly to bring prices down but we're also constantly working with partners to improve that digital experience whether it's cloud integration whether it's user interface trying to make the value more and the future set what the customers expect. Nice. Once I was on Copenhagen and I saw an E-8 bus stop. It was amazing. We have quite a few bus shelters globally. Europe, America, Taiwan over in Asia. This one's done by ConnectPoint. In particular for bus shelters, they're good because we don't necessarily want to run power to each of them and we want the visibility effect. So that's a perfect fit for E-8. And whether it's this size which is a little bit larger, 31 or whether it's this 13.3 and there's actually a double 13.3 so there's two of these side-by-side. This one by GDS is particularly interesting because it can be solar powered.  Boom! Self-sustaining signage for your bus stop. Nice. And you just need to plug it on the floor or something. Yes. They'd probably drill a hole. They'd drill a hole? They'd probably drill a hole. Yes. And this one's done by Mercury Innovations. If you look you can find some interesting signs in Australia where they have maybe 15 different signs sort of tacked on to each other and this confusing smorgasbord of you can't tell when you want to park. And so they've looked to improve the user experience of can I park here or not? It seems like a simple question but there's the street cleaning days and the holidays and so they actually integrated some EPDs, some e-signs into this application of parking so that you can see them. Nice. Yeah. Alright. That's awesome. Yeah. Any other questions about this? I think we've walked through most of the booth at this point. What's the conversations you have here at the display week? People coming? Yeah. People are... We've had a lot of conversations. Most of the comments on the Spectra 6 are, Wow! How did you do that? We usually tell them secret sauce. I also... I could show you... So we had a lot of comments on the Gallery 3 as well. So that's these two up here. As of last December, we announced the Gallery 3 product and that eight partners, customers would be launching products with it. The first two to launch are Big Me and Pocket Book. And so this is interesting because it brings that native color experience, no color filter. It's using the same particles inside to generate the different colors but into the very demanding consumer reading experience. So the transitions have to be fast. Our first version of Gallery was about a 30 second transition, kind of as I mentioned before. And this is down to less than one second. That's a huge advancement that we were able to make. And we still are able to keep all this color information. So my father was collecting comic books from the 60s, 50s, the old intents and everything. Everybody wants to do manga when they talk to me. The question is how do you get one subscription that gives you everything? The Netflix of comic books. Yeah, so at some point if somebody wants to make a device like that we'd be happy to support them. In most cases, if we're working with a Kobo or working with an Amazon they have a catalog which may include manga and they could deliver that to you in a digital service. In some cases it's a subscription. In some cases it's a by content individually. And we understand that graphic novel and all of that is an excellent category for it. We're also working a lot in healthcare. This small device down here is called Badger. It's particularly cool because it's basically closed caption for while you're talking. You wear it maybe as a doctor or a nurse and then you go to your patient and maybe that patient is hard of hearing. Maybe that patient speaks a different language. This badge in conjunction with your phone needs to connect with the phone. Will translate in real time what you are saying and place it in the native language that you choose so that somebody can read it. It's a huge value. If you've ever been in one of those settings it can be frustrating if you can't communicate and you need assistance. This is potentially huge. We're also working for some larger things. These are digital whiteboards. You may have been in the hospital and you notice there's a whiteboard. In many cases it's never used. We want to connect those whiteboards. Still have the experience of getting information, but connect it to the back end medical records. It's relevant to the patient at all times. When the patient leaves the room all the information is immediately cleared so there's no information leak. It supports compliance. We do see interest in value in this application as well. We continue to work on IoT. This particular one is an insulin pen. It's made by Bigfoot. It's a display in the cap. Don't mind that it says device art. It's a demo unit. It helps you with your dosage information and tells you when it's been dispensed. We're continuing to push into the smart space of Internet of Things. I'm guessing one of the biggest markets now for eInk is what you call the shelf labels. Shelf labels are a very strong market. There are hundreds of millions now. I think we have over 900 million displays worldwide installed. It's huge. Many people will not be aware of it because you go to your store. In the US you might go to a Whole Foods or a Kohl's or a Best Buy. It looks like a shelf label. You don't even know that it's actually electronic paper. It's like a no-brainer for all these supermarkets and everything. It quickly understands how much savings it enables. It's five to seven years so once you install it it lasts for a long time. It gives them labor savings. It's very expensive to send somebody out and change all the labels in a store. If you have 30,000 products in your store it's a lot of labels to change and it takes a lot of labor. Plus we have shortages of people to hire so if you are hiring you might want them to do something else in the store instead. These stores are always competing with in an instant. The retailer at a disadvantage if they don't have similar flexibility and there's always compliance. In many cases there's a requirement that your price on your shelf matches the price that you intend to or have advertised and so this helps with compliance as well. In many ways it's a strong ROI proposition right now. Absolutely. How did we do? That was awesome. I even see some prism here. A lot of stuff happening. Absolutely. This is our first prism line. There's here four different colors. Each one is white and a different color and they'll slowly turn on here as we wait. They highlight different ways that EEC is a sustainable company and makes a sustainable product that helps downstream. Our customers support sustainability. Dark Sky approved. We don't give off any light. No light pollution. One of the only dissuades certified is the Climate Pledge. That's the Amazon initiative in order to reduce greenhouse emissions. Science-based targets is a similar thing. We walk the walk and our products help our customers do it as well. And thanks for the walk. Absolutely. Anytime. I hope this was informative. I hope it was fun and maybe even educational. Check out the WISE.  I think I'm saving over a thousand euros every year because I use it everywhere when I pay for stuff. You want to pay for stuff in the world. You can check out my longer video where I explain some more. I think this is amazing and it's free. You can just sign up with my link down here and you can use it on your Android Pay your Apple Pay. It's like a prepaid Visa debit card. You need to put a little bit of money on it and you can just put money on it from your local bank account. The fees are so tiny. It's like five to ten times cheaper on your bank. Don't use PayPal anymore. Don't use Western Union when you send money to a whole bunch of countries around the world. Just use WISE. It's way faster and it's five to ten times cheaper. So really check it out. It's cool and I appreciate if you use my link below. Thanks for watching.


we're on the right side of it. So let's look at Metta. It's the best thing today. We also could do Baba. So let's see what we get. Free 15 is the target in this. And let's make sure it's open today. There we go. Here we go. Okay. All right, staff's gonna be 60. Whoa, here this is going. Could this pull the market down? Yes. I don't know the numbers yet. Just wait. This is gonna go today. It's really gonna go. There's no rush to get in it, basically. Okay, staff's gonna be 60. Let's do 75 by 60. There, get it right now. Get the stop in at 60. Boom, Metta. Here we go. Beginning if you wanted. 315, I said, and we're almost at the target. So let's look at 313, 314, 313, 312. Dream target today for Metta. 300, it's not insane. 305, that's not insane. That's definitely not insane. Here we go. Under 315, Metta, Metta, Metta. Here we go. First target, boom. It's gonna break 315. Can we do it? Can we do it? I think we can. Actually, I think we did, didn't we? Yep, we did. 314, 85. We're gonna have a good week. It's all setting up. The wheels are in motion. What's that one sign for that episode? The wheels are in motion. Remember that one? He's trying to get the green card for the restaurateur. Oh, God, I can't, I forget his name. It's all the tip of my tongue. Oh my God, what's that guy's name? That was his neighbor. Babu. Babu, he was trying to get his green card. Here we go. The wheels are in motion. And you know the by the dippers are gonna come in. Will they make it? Will they last? Will they survive? No. Here we go. Wow, today could be, I don't even, I just wanna zip it. Let's just wait and see. Here we are. If you're in Metta, you're in it. You have to stop at the right place. Let's look for 314, 313 and change. Oh, what a nice guy. 310 today for sure in Metta. I still think we should be out of this quick and soon like we normally do. The by the dippers. Okay, here we go, Metta. The wheels are in motion for us to win and have a big week. We go Metta under 85. You can even add to the Metta here. If you want to get crazy today, I don't. Normally on a Monday, we don't even have that many things to look at. We're lucky. We had a lot today. We had a ton. We could have done this too, but we didn't. But you could have done this too, and you still could do it here if you want to. So again, the spy's been a lot stronger than the cues lately. Part of the reason the cues were falling is NVIDIA. But again, if you look at Apple, you say, gosh, the cues are falling. Apple had the rally. Not really going anywhere. NVIDIA. NVIDIA. Everybody looks good. Tricky market if you don't hold the conviction because it's been back and forth. And that's the bugaboo. Here we go, Metta under 85 and a push and a flush. And I'm probably going to be out. Call it a day. Look at what's out for tonight. If anything, here we go. New low on Metta under 85. Here we go, 80, 77, there. Here, 314. We're going to get there. Here. We're almost there. Here we go. Here, Metta, Metta, Metta, Metta, Metta. That's going to have a really big bar today. But again, we're going to get out of the day trade quick. You know what else I want to look at when we're done? Oil. Someone remind me of that. 31. You could be out of this right here if you want. In fact, let's just look at oil right now. Now we got to keep an eye on this. This has to be on our radar every day from now on. Okay. Who's doing what? Who got out of this? Who's still in this? This looks good. Again, I warned everyone. The buy the dippers will come in. It's happening right now. Time of the day is 937. That's why why this is pushing up. Make sure you have the stop at the right place. If you get out, you get out. If you're in it, you're holding. At least for one more move down. Let's double triple check our stop. We could give it more room. I don't really want to do that, but we could. Let's give it over 318. The only reason I'm doing this is because of the market. Backing up right now, I want this to hang on. If you get out of it, you get out of it. You're done. 31810. I'm moving it to 31810. I'm giving it 50 more cents. Who's doing what? Who got out of it? Who's still in it? Actually, you know what? I didn't look if there was data this morning. Let me look at that too. Market looks fine though. This is good. If you got out of it, you could get right back into the sucker actually. You can get right back into it. You could do an ad here. You could do whatever you want to do. If you got out, you got out. You could be done for the day. If you're still in it, you're still in it. You move the stop. Original stops holding. This is a good gap. I don't know what else to say, but because of the market, it backed up. We've discussed this so many times that people aren't going to give it up in this market. Again, the Fed did not say they're not going to raise rates and the Fed didn't say that they're going to lower rates in 2024, but that's what every one is speculating. We'll know a lot more next week on Wednesday. Either the market isn't really going to have any traction until Wednesday or it will. I think we're going to drop today. I think this is going to come out of nowhere because it kind of did come out of nowhere, but there was some negative news overseas that came over on Sunday into today. I think we're going to drop today. I think today is the day. You've got to watch your trades if you're in stuff. If you're in stuff and you suffered through the rally on Friday, you believe in the trades. If you killed them, I don't know what to say. I mean, again, don't risk more than you can afford to lose in the trades and you'll believe in the gap and you'll have a chance for the trades to work and for you to make money. This is not an easy market for people to train because it's not just rallying, rallying, rallying, rallying and it's not dropping, dropping, dropping, dropping, dropping. I mean, literally if you look at the market at 10 a.m. and look at it at 12 and look at it at 1 and 2 and then at the close, you could say five different things all day. You say, oh, the market's higher. Oh, no, it's lower. Oh, no, it's higher. Oh, no, it's lower. That's what you could do all day and you could drive yourself nuts. So don't do it. Rate the gap. Believe in the gap. Take the trades. Set your risk accordingly and let it play out.  Don't forget that too. Don't forget that one piece of the puzzle. This is tempting to do an ad here, but this is tempting. Let's add it 99 if you want to do it. Let's add it 99 if you want to do it. Metta with your new stop. There. Take more if you want it. And again, we moved the stop. Size it accordingly. Here we go. Metta. Here, here. I knew yesterday when I saw that news come out that we were going to be down in the futures. And then we were, and actually the Dow was up, but the spy was down and the cues were down. And then we slowly started to drift lower and lower and lower. We just started to slowly drift lower until this morning. And again, I'd love us to be down more, but whatever. Here we go. Metta. Free 1431 was a new low. One flush through the, if you did the ad, one flush through this and a break and we're going to be out and we'll have a great start to the week. And if you're in puts, you got to watch them. You got to watch them today. You got to see your desk all day. I'm sorry, but that's what you have to do. And if you can't put orders out to sell you out of stuff, if it hits out, every day between now and the holidays, you got to pay attention to what's going on because you want to make sure to make everything count now because around the holidays is going to be slow. Actually, when does Hanukkah start? Hanukkah is this week, isn't it? I have to watch for that too with volume days. Could have low volume around that holiday. Here we go. Metta. This was news. Again, we'll go over it when we're done. Let's mark it just again. NVIDIA. Here we go. Shelley, are you in the new NVIDIA? We have a lot going on today. I'm not going to get anything done today because we're going to have to watch what's going on here. It's going to be a long day. Shelley, are you in the new NVIDIA? Who is in NVIDIA? I'm talking about the put, that I called last week in NVIDIA. It looks so great. This is going to pull the market down to 450, 445, 440, some crazy number at NVIDIA. This will pull down the cues. This is going to rock the market, but this is the market stock too when we're in it. One more push in Metta. If you did the ad, you could be out of this right here. You could be out of it if you did the ad. You could just take it. No one says you have to hold anything.    I think we could get some traction today. Don't get out of everything. Here we go, Metta. Today feels like a Thursday or a Friday, but it's a Monday. That's a good sign for the week. I mean, it's hard to believe this market's holding on with NVIDIA. It's not going to last, but you know what else I wanted to look at. So I stayed away from this test on. I'm glad that I did. This is really a range bound. It's completely ridiculous actually. Again, this is down today with the market, but it looks like it's trying to rally. The market falls just won't make it, but this is really in a very tight range. Didn't do anything with this because of the truck launch last week. Bulletproof truck. That's testing it out with cyber truck or whatever. So stayed away from this, but this did fall with everything else, but it, it's really just range bound. Here we go, Metta. Under the low, one big push, under three, four, 14. Here, we're going to get out of this soon. 9.49 in the morning, video end of video end of video end of video. Spy rally, strange. The cues on the spy look so different right now. Let's look at the banks. Here, Metta. Just quickly look at this. The cue in the spy looks so different. I don't know, maybe it is a banks. That could be. Here we go. Metta, hand in the button here. Metta, new low, Metta under three, 14. Here we go. Nice morning. Great way to start the week. We even did the ad. We were nuts today. We could get some traction in the market. There we go, under three, 14. Did anybody get out? It's moving so fast. I was gonna, I was gonna, wait, I could still get out here. 20 some cents off the low. I'm really going to get out soon. Here we go. 88, 80 here. I wanted to just break 80. Here we go. We're going to do it. We're going to do it. I wasn't going to bounce at all. One more push here. Here we go. Metta, Metta, Metta, Metta, Metta. Here that is. We squeezed a little more. Wow. Good morning. Again, 310 for sure in this today. This will pull the market down. This is doing it. Great job. Wonderful.


Welcome to sorting things up in the DevOps world. I'm here with Juan Manuel Silva, a salty guy. Give him some claps. Right. Thank you all for coming. For those of you who've been here last year, I gave a similar talk. This is going to be a little more in-depth, so let's get started. Okay, let's get the boring stuff out of the way. My name is Juan Manuel Santos. I work as a team leader and a board engineer at Red Hat. I'm also one of the organizers of CISarmy, which is the Argentinian system administrator's community, and Arderla, which is a local co-working slash tech conference event that we do every year in Buenos Aires since 2014. I've been using Salt for a couple of years now, mainly with no regrets or with all regrets, whichever you want to choose. Let me get this disclaimer first, so let's get this out of the way, too. I am only a humble user of Salt. I have tinkered a bit with the code.  I only have three days to prepare this, so who doesn't like pressure, right? My thanks go to the EuroPython team for managing to squeeze this talking. So, why Salt? As you may or may not know, Salt is a configuration management system. In case you don't know what that is, think puppet chef Ansible, but only better. And why do I say better? It's because it's written in Python, and it leverages YAML and Jinja. Now, I know some people in the room might not like YAML, but you can also use JSON if you want. It is relatively easy to understand, and I said relatively, because it has some complex things, but what it lacks in simplicity of reading and understanding, it makes it up in being extremely powerful and giving you a huge amount of control over what you can do with it. Some of this we'll be seeing in the next few minutes. One more detail that frequently gets lost in translation. Salt can work without an agent, in the case you don't have root access or you're not allowed to run the agent on your machines. VSSH, much like Ansible does. So, previously in EuroPython, as I said, last year I gave a talk. This was mainly an introduction, speaking of the basic mechanics in terms and concepts behind Salt. As a quick recap, Salt has a master minion architecture, where the master is the one that gives out the orders, and the minions are ordered to do minion stuff. It does so by defining states and high states. The states are representing the state a system should be in, and the whole collection of states that should be applied to a system is called a high state. Another cool concept is that of matching, which means targeting your minions and to determine which states apply to which minions. And finally, there's the concepts of grains and pillar, grains being information sent from the minions to the master, and pillar being information sent from the master to the minions. Sadly, and I have to say this, still no Python 3 support. Salt is still in Python 2. It's getting there, though. There's a big issue, hopefully we'll get there. As usual, it's not because of Salt, it's because of the Salt's dependencies. But anyway, moving on. Two more concepts that didn't make it to last year presentation are those of the mine and the syndic. Now, the mine essentially gets data from the minions sent to the master on a regular interval. Now, even though this is done on a regular interval, this is not useful for metrics because only the most recent data that you collect is maintained. Another thing that might confuse you is that all the data is made available to all the minions. So when you query it, you might get the answer of the same data from all the minions at the same time, which can be quite confusing. In fact, you might be wondering, isn't this like grains? Isn't what grains are supposed to do? Get data from the minions sent into the master? Kind of. The thing is, the mine data is updated more often. The grains are mainly static. They're only updated if you purposefully update them, which it's not something that you would usually do. Also, if minions need data from other slower minions, the mine acts as a kind of cache. So there's that, too. And, okay. There are two ways of defining which mine functions you want to collect from the minions. In the case of normal operation of salts, you would do so either in the pillar or on the minions' configuration file. In the special case of not using the agent, as I mentioned before, you have three ways. Since you don't have the minions' configuration file, you can either use the roster, the pillar, and the master configuration file. And so, a quick example of what the salt mine would be so that you don't get much to confused, but promise you, you'll get confused. Looks like this. So let's say we want to first target all the minions in our web servers group. We are going to be applying a mine function to gather the IP addresses of the first network card every five minutes. So this we can later use, for example, in an HA proxy configuration to populate the server list. Now, I know that you might be getting baffled by all the ginger here. Try not to think of it. The important thing to understand here is that should we add a new host to the web server group within five minutes, we can have its IP address up in the HA proxy configuration file. This is all thanks to the mine, which we can configure the interval of updating. Now, before we continue, since we already mentioned that salt has a master minion architecture, there's an inherent topology to it. So let's talk a bit about that. The most common one would be one too many, meaning one master many minions. But of course, this is boring. This might not scale. This also kills cat during lunar eclipse. So what are the alternatives? How much can we toy around with this? Could we have, say, more masters? Could we have a multi-master topology? And I don't know if there's any information security guys in the room, but if you are, you're going to love this question. Could we implement segregation? Meaning, could we segregate part of the infrastructure split it so they don't communicate with each other? But, you know, there's still a functioning salt infrastructure. And coincidentally, now I'm wearing the right hat apparel, let's answer those questions with another question. So what if we try more power? What's all this? This is something called the syndic node. The syndic node is an intermediate node type, which acts as a pass-through. The aim of it is to control a given set of lower level minions, which means that in the case of the syndic node, we're going to be having two demons, the syndic and the master. Optionally, you can run a minion too. So the way it works is something like this. The main master, which now we're going to call the master of masters, you're going to see why, even though it's already a funny name, sends an order to the minions and to the syndic node. The syndic node relays those orders to the local master that is running in the same machine, and then that master gets the orders and relays them to the lower minions. So now our syndic node is actually called the master of minions. This, of course, works the other way around when some of the lower level minions reply to any orders, they go first to the lower level master, then to the syndic, and then up to the main master. So if we have the master, which now is our master of masters, it can have as many minions as we would like connected to it, then we can have a syndic node, for example, a syndic master of minions node, which can also have any given number of minions connected to it. But the good thing about this is that we can even nest levels of syndic, one over the other, and have as many minions as we like. So the topology here, it's kind of up to you. So the only places where you're going to have to ensure connectivity is where the lines are. So how do we actually do this? The configuration is quite simple. On the syndic node, we set in the syndic master directive. This should point to our main master. We also have to define an ID here because the syndic node takes the ID from here. Then on the master node, of course, we have to tell it that we are now ordering other masters. We are now in control of syndic nodes. In the case of the minions, they should have the lower level minions. They should have the IP address of the syndic node in their configuration file. Just a few more steps. We run the syndic node, of course. On the main master, we're going to have to accept the key because, essentially, there's a new key that gets generated. So now you might be getting the idea that behind this talk is to make you think of the possibilities. You could have different syndics per environment, development, QA, production. You could also have different syndics to comply with the security standards that you might have, that you might want to come up with. Just to mention it, we can even do multi-master with this. We can have syndics and many masters, main masters. We will not cover it here, but just know that this is possible. So that's it for mine and the syndic node. Now we're on to more heavy metal stuff. Our first stop here is going to be the event system. So what do you think an event system does? Of course, it keeps track of events. But that's not the only thing it does. The important thing is that events can be acted upon. And this system is also the base of the rest of the systems that we're going to see in this talk. In essence, this is mainly a Serium-Q pub interface. The important things to understand here is that every event has a tag which allows for quick filtering and identifying an event, and also has an amount of arbitrary data inside of it which tells us information about the event. So with just a simple command, run in the master, we can already start watching for events. Start watching what's going on. We can also use this other command to send a random event that we are just making it up. You can see that this would be the tag. Those would be the data of the event. The data is mainly a JSON string. In Python, it would be a dictionary because, in fact, you can also send events from Python code, from pure Python code. And if we did things right after sending the event, this should show up if we were watching attentively to the event bus. We can see that there's our tag and there's our data. Okay. Now, another interesting bit that I didn't get to make the distinction last year. There are two kinds of modules, state modules, actually. The first one is the execution modules, and the other one is the runner modules. So the execution modules is the main kind of state that you see in salt. It means something that is going to be run on the minions, whereas the runner modules are going to be run on the master. And these runner modules can be either synchronous or asynchronous. They are added via the runner directory's configuration in the master file. And that's the best part. What do we put inside that directory? Pure Python code. So runner modules are essentially, is essentially Python code. And an addendum to this is we just talked about events. Any print statements that we put inside our runner modules will be converted to events. So if we do this inside a runner module, we will get something like this. See that, okay, the tag is not quite nice, but there's a data. So even though you can write runner modules, and you're certainly welcome to do so, it is tempting, but there's actually no need. I mean, there's already a full list of runner modules available in salt in the documentation. So feel free to check those out. Now, wouldn't it be nice to live in a place like that? Sadly, we're not talking about those kind of beacons, but kind of. Salt beacons are like those concrete towers with the light bulb on top. They're also a kind of single. Or something like that. I mean, they use the event system to monitor things that are happening outside of salt. And when something happens to those things and notification is sent, which is actually an event. So those are configured via the Minions configuration file because we're actually interested in the Minions at this point. And any system administrators in the room, anyone? Okay. Does anything of this ring a bell, something that is, you know, notifications? I notify maybe? Okay. Yeah. I mean, I notify, which is a file system monitoring API to track changes on files and directories. Kind of looks like this. So in fact, there is an I notify beacon, which you use to monitor changes to a certain kind of file, to a certain file, in a given time interval, and then you have it. Anytime the resolve.conf file changes, we now get an event. There's also other types of beacons, for example, a process. We can be monitoring whether or not a certain process with a process name we specify is running or not. If it's not running and it starts to run, we get an event. If it's running and it stops, we get an event. So kind of nice, right? There are actually several beacon types. Memory, disk, system load, network settings. It works. There are really a lot and they're growing. It's just going to leave you, you can also write your own, of course. It's just going to leave you the documentation here so you can check it out later. Now, this is where things are going to get a little bit more interesting. Yeah, like that. It would be nice if actually the reactor was like this. Believe me, it's actually closed. So, what is the salt reactor? As its name implies, the main job of the salt reactor is to react, but not react in a javascript way, thankfully, to react in a salt way or salty way. In other words, the reactor is the component that is responsible for triggering actions in response to events. So now you see why we saw the event bus earlier. Of course, we need the event system first, but what is an action? Since we're in salt, an action is essentially a state that we define. And what is actually going to happen in reality is something goes something like this. Something is going to happen if with the thing right, there's going to be an event, maybe because something was being monitored by a beacon or something else, and the event is going to be picked up by the reactor and the reactor is going to translate that event into an action or actually a state. Reactors are defined in the master's configuration file. It's a component of the salt master engine. As we said, the reactor will be making these associations. The associations, if you remember what an event was, you remember that it had a tag. So the association is made via the tag. So we put a tag in the configuration file and we define which states are going to cover that action. Sync tags here is quite clear. Do note that there's an asterisk there. We can use wildcards because some events are fired by more than just one minion and have the minion ID in the tag. So for example, this first one here is the event of a minion starting up. Two if you want to match all the minions starting up, you just put the wildcard in the right place. So this whole slide is actually the main reason I'm here. It's the one thing I spend the most time while working with salt. So I ask you to please pair with me. There are a few caveats. The state system that we just saw here, those are states living inside the reactor. The state system is actually rather limited and you can easily skip this while you're reading the documentation and trying out your reactor states. Trying to run things that would normally work in the rest of salt, in the rest of the states that you have might not work here. You will find the things are missing and for starters, forget about grains and pillar. Grains and pillar are not available in the reactor. If you choose to use those, you get unexpected results. Also, reactor states are actually processed sequentially. They're first rendered and the data is then sent to a worker pool. But since they're first processed sequentially, you're going to want to make your states as simple and as small and as fast as possible. So after long hours of fighting over the reactor and tearing the little hair I have left in my head, this is the short version. Do not handle logic in your reactor states. This might be a bit too confusing because what's the point then? But I'm going to explain it in a bit more detail. You should use your reactor states for matching and then decide which minions to which states based on an event.  Do not try to add some logic here. You're going to spend a very long time and you won't be happy about it. So I don't know if this is actually true. It's what it looks like from the outside. It appears there's a disconnect because we're talking about two different engines, even if it's under the same demon. I like to think it's because of Python namespaces, but I could be wrong. But so too long didn't read. Do not handle logic there. So as we said, with the reactor we are associating events to states. So if we have our custom event and we have our custom reactor state file, the idea is to keep it as simple as this. And if you really have to do complex things and ensure that many, many things are done when a given event is fired, just put those inside the long running and complex state. So once the reactor parses this and sends it to the worker pool, this will be running on the main salt namespace, so to speak. So what can we use a reactor for? One good example is out of accepting all the keys of all the minions in our environment. Since it's quite a hassle every time you start a minion, you have to go to the master to accept the key and so on and so forth. So as you might have guessed, whenever a minion tries to authenticate an event is fired and whenever a minion finished starting up, there's another event. So for the purposes of this example we are going to assume that all minions whose names start with nice are going to have their keys auto-accepted. So first of all in the state that's going to be dealing with authentication we'll first want to remove the keys coming from the minions that have failed to authenticate. The next step is going to be to trigger a minion restart and now I know this is ugly, this is just for the purposes of examples. Every time I read SSH in the middle of another language another configuration management system I kind of creep out a bit but this is just an example. What we want to do is have the minion re-authenticate generate a new key, so to speak. So reaching the end of our big state if we are in pending status of authentication pending status and the name starts with nice we accept the key. And as for the last state when the minion finishes starting up this is actually a good practice that you can implement. Whenever a minion finishes starting up we apply a high state to that minion. This is something nice to ensure that all your minions are consistent at least when starting up. Now note here that we've been hard coding the nice and maybe some other things around it it's because as we said before we don't have the grains, we don't have the pillar we don't have a safe way to store information make it available to the reactor. So keep that in mind and use the reactor. And our last component today is going to be the API. Of course, Solst has a REST API. The main idea behind it is to send commands to our running master. The API supports both encryption and authentication. The authentication which is something that you might not see very usually in Solst well Solst has an external authentication system it allows for authenticate against LDAP, against PAM it also has access control in it so it's really outside the scope of this talk it's a very big thing to talk about but it is worth mentioning that it actually exists and the entire things that are managed by the API are controlled by another demon the Solst API demon. So if it's a REST API we can of course use anything that can make HTTP requests and get information from it or send information to it. In this very short example we are making a request to a certain URI for minions and if we pass the correct minion ID we're going to start getting data about that minion. In this case we are for the sake of simplicity we're not using authentication here. Now there are several API endpoints available already bundled with this Solst API they're pretty much self-explanatory but let me draw your attention to one in particular the slash hook this is a special endpoint it's a generic webhook entry point and the whole reason for existing is that any post requests that are done here will be generated events on the master side, on the event bus and the post data that we send to it is going to become the data of our generated event. Another important thing because this is a special endpoint it's the only one that Solst allows you to explicitly disable authentication in this particular part. Another thing is if you're disabled authentication it does not mean that you can do whatever you like you're expected to implement some kind of security. Why would you be disabled authentication? Well, I like to think of apps that can barely perform an HTTP request that can barely understand a URL so they can only do a request with a special hard-coded token that you specify so that's why we have that there. Now how about from all the rush that we just been in it how about we put them all together be nice and friends now I know you might be better here you've seen a lot of information and I think that you might be a little bit confused but I assure you we can do pretty interesting stuff with all that we saw the events, the beacons, the reactor and the API now more graphical understanding of how all this connects together we first have the beacons and the API the main interesting point about these two is that they're related to elements outside salt the beacons monitor things outside salt and the API it's an API so anybody can make a request to it so they're both related to elements from the outside you know both of these two will be generating events in our event system in our event bus those events can be later picked up by the reactor given what we define inside the reactor which then can be translated into salt states now with the great responsibility of having to manage your entire DevOps, Lops, Workflow infrastructure comes a great power there's a deliberate reordering of the phrase here because if you configure salt properly you're going to have full control of everything in your infrastructure, in your workflow everything and from within salt so as such you're expected to know what you're doing and you should always rely on a sensible way of doing things for example, beware of the security risks you might be tempted to you know take way too much power to salt and that's a good thing but beware of somebody trying to do an ugly thing with it so to finish this off, let's take a minute talk about what all you can do with all of this I'll just be naming you a couple of examples from the top of my head leave you to think the rest that's because that's what salt is salt is kind of like a batteries included approach to give you the space to create your own solutions much like Python is, which is why I love salt so just to name an example let's talk about self-healing anybody knows what self-healing is, what it consists of anybody heard the term? okay, so in more humane words self-healing is the ability that we give our applications or systems the ability to repair themselves whenever something bad has happened whenever they encounter an adverse situation on their own that's the thing, that's why it's called self-healing now all this might just be a REST API call away because if in your application you can identify the bad thing that has happened can be corrected by something that can be automated you can do it with an API call because salt can have control of that or another example and I think many of you have encountered this let's say half your team refuses to use Jenkins or the CI tool that you're using if you're not because you can leave them be whatever they are using and integrate the rest of the push build test deploy endless CI cycle with salt you can manage it with salt too another example if you were talking about scaling both up or down or sideways growing shrinking you can prepare for it with salt and you can also trust in salt to do some provisioning we haven't covered it here but salt also has a salt cloud demon to provision cloud instances and last but not least with a good beacon setup you can make sure that your environments are consistent if you have things that aren't supposed to change and you suspect that somebody tends to do nasty things with the beacons you can react immediately upon any changes that you deem unwanted and so these are mainly all the examples that I could think of with the short time that I was given as I said before I really do hope that you can leverage what you saw here to come up with your own solutions because I'm sure that your problems might be worse than what I simply presented him so as for the docs and as for last year all the documentation is in the official salt stack documentation I really encourage you to take a read if you have any particular questions there's also the possibility of bothering the guys at the salt free note channel in IRC I do that a lot and we're reaching the end so now we have time for some questions so feel free to shoot away Can you compare salt with Ansible? I'm going to be honest with you I haven't used Ansible I know that it's my maybe has a more basic approach what I've been told from the people that have tried both is that Ansible lacks some components that salt has like the reactor for example so it goes around those lines I was wondering how one could use salt as a deployment tool is it feasible to deploy a complete web application with it or is it just well fit to set up the system and then you need to revert to a proper deployment tool to deploy your application for application deployments right? yeah web application so set up a database put some basic data in it deploy your jungle application web server and things like that yeah I mean maybe that was covered in the previous talk which is more basic but no no no that's a very good question and yeah you can do it you might have to handle a slightly more manual approach in order to tailor it for your environment but you can certainly do it and if you're thinking of doing some bare metal provisioning you can also do that not exactly with salt on its own but salt has form and integration form and it's provisioning software that was mainly written for puppet but now has salt integration so you can do the whole cycle from it hello I understood that the communication channel is SSH is it correct salt has a way of working with SSH it's not the main way that it works and what I want to ask is how do you handle minions at running windows that's a very good question actually it is possible but I'm sorry I've never had to do it I have been a little bit playing around with that and found a way to insert an SSH daemon in windows using suckwind that has an SSH it seems to be working I'm just curious if there's other options and as long as you are aware of any limitations that you might have it should work the rest of the system is shared and it's the same thanks for the great talk I'm using salt for three months it's really cool there's also one stuff engines getting some additional events and currently I'm looking for the way to make pillars dynamic to get information from console or from at city during deployment to get some information from key value storage external is it possible I would have to look that up not entirely sure but I mean everything appears to be extensible in salt so I don't see why not yeah maybe it is hi how do you install without SSH or is there any good approach to do that upgrading salt without SSH you mean the master I don't think I understood where you are going with the question how do you you need to have access to the system you need to have access to install the new version and you also have to restart the asian that's one thing that is still not handled very well in salt is restarting minions whenever there's an upgrade because you can't do it from inside the master because you're going to be losing communication for a bit so yeah it's kind of a tricky spot still regarding the question about dynamic pillar I think it's possible salt has a mechanism to get a pillar from external systems you can implement Python model to there is a plugin called Glass which uses that to make the pillar more usable in fact and the question is how do you test your states we have several breakages in production due to human error I know I don't have a system I use salt for personal use I don't have the luxury of working with the salt environment but I know where you're going with it it would be nice if you could have a development environment or QA to try things out because yeah once you've made a change to a state and salt doesn't like it it will blow up it's kind of tricky you have to keep looking at the logs careful what you changed you're bound to have the last change that you may cause a problem if you see a problem it's working how do you handle provisioning new servers and how do you handle your inventory of servers well from the let's answer the first question in two parts if we're talking about bare metal provisioning you have to use something like formant that allows you to boot a system and then apply salt states to it so salt is not it's like puppet in that way and it's not like ansibot that way it doesn't have the ability to provision a system from bare metal from the ground up when the system is already installed and has a minion running you can do whatever you want with it if you log the inventory from the perspective of master all the master sees are minions so it is up to you to group them using node groups or grains or whatever you deem to be necessary you would be basically setting your categories on your own building node groups setting grains on certain minions to identify them from the perspective of the master but essentially there is no no way of distinction in fact when we talked about the syndic node the main master the master of masters will see all minions connected to it even those from lower level syndic nodes so this is in response to the question about testing the salt states we do this we'll use vagrant on your local machines with a masterless minion setup and then spin up in number of VMs and actually test the states at least to some minimal so that we can catch human error like that because we have the same problem how we deploy across hundreds of machines simultaneously in one error can really mess up your day so I've tried with vagrant locally it works pretty well because you can spin up different kinds of VMs so we use FreeBSD or Ubuntu or Sintos and you can simulate a lot of those environments easily Interesting, thank you Hi, thanks for the look Thank you I'm seeing that most of the questions we are asking about salt we think that salt can't do can be done perfectly using Ansible like initial system configuration or maybe someone asked about service management I know you already said you haven't used Ansible but have you heard about someone using Ansible and salt together? No, pretty much when somebody chooses configuration management system they like to stick to it it has to do with the learning curve and all that stuff so it would be harder from the very few things that I've seen from Ansible it's quite different from salt even though both are written in Python even though both use YAML they're quite different so every organization wants to choose one technology that's interesting Ansible can be used for the bare metal provisioning parts and salt maybe for the rest or salt for the reactors you could certainly mix those two to create a richer machine to install salt master on it and to configure it and then to apply high state are we talking about the fabric python module? Yes, right it comes from paramico it uses SSH to Yes, right it uses SSH to make a basic provisioning to start a richer machine to put salt master on it and then use salt master and full power of salt Yeah, you can also do that We have time for one more question or not Thank you guys