horse with a speed of light, a cloud of dust, and a hearty high old silver, the Lone Ranger. The first men to penetrate the mountainous regions of the western United States were prospectors in search of gold. They found great wealth, but the precious yellow metal seemed to bring out the worst in many of them. Robbery and murder were everyday occurrences in the mining camps until the Lone Ranger carried his fight for justice into the hill country. Without the force of his courage and resourcefulness, law and order might never have been established on that lawless frontier. Return with us now those thrilling days when the West was young. Out of the past come the thundering hoofbeats of the great horse Silver. The Lone Ranger rides again. The Gold King mine was one of the largest ore workings in the vicinity of fine bluffs. It implored an extensive crew and the mine itself was composed of a dozen tunnels radiating from a central shaft. At the time our story opens, young Dave Muncie the foreman was directing operations within the mine. Suddenly, from the direction of number three tunnel, there came the sound of a powerful blast. You there, Bart. Come here. Did you set off that blast, Bart? What if I did? What if you did? Well, you brought down the whole tunnel. It'll take a week to clear up the damage you've done. That ain't nothing to me. What's that you said? You heard me, I reckon. You're through. Get your time and clear out. You're getting high and mighty, ain't you? I said clear out. And Jeff, who do you think you are? Why you? Just hold on a second, mister. I was hired for Wilkins. Seen as how he's manager while you're just foreman. A record will take him to fire me. I've taken about enough from you, Bart. Yeah? You've been a troublemaker ever since you come here. You've talked others in the crew into disobeying orders. You've done your work just as carelessly as you could. If you and the others in the crew like you, let's put us behind on a production schedule. That's just too bad, ain't it? Now that I've fired, Jeff, I'm going to do something I've been aching to do for quite a spin. Yeah? What's that? This. You'll hit me. That was just the first one. Here's the mate to it. I'll get you for that. Come on. Take this. You missed, but this'll do for you, I reckon. You can't get away with that. Come on, fellas. We'll teach the whipper staffer a lesson. Get back. Get back to work. Come on, fellas. Rush him.  We have done taken orders from your regular picks, man. Take that, Jeff. That's one for you, Ted. And you, Mike. Let me out of this pick. Drop that pick. Try and make me. I said, drop it. Oh, my arm. Let me go. I'm sure, like you say, let me go. Get back. If you wasn't our man. I haven't grown. Come after me, if you wish. I don't know who you are or why you came here, but you have her along at just about the right time for me. It was likely to beat me up good. I heard the blast from outside and was afraid there might be trouble. Sounded too powerful for regular blasts. It was powerful enough to do all the damage you see over there. All right, men, you're through. Every last one of you. Report to the office. Get your pay and keep moving. Maybe we will, mister, and maybe we won't. Here comes Wilkins. I guess he's got more say around here than you have. Hey, Mr. Wilkins. What's all the trouble here? This fella here. What are you doing here, mister? You're Wilkins. You're the manager. I am. What's that to you? I was trying to place you. Seems to me I've seen you before. You haven't answered my question. Well, I came because of the blast. I've already explained to your foreman. All right, Bart. What were you going to say? It's skunk here. It says we're all fired. Mr. Wilkins, just take a look at the tunnel there. Some more of Bart's work. And the crew just now tried to mob me, but hadn't been for the mask, man. They would have, too. I said it was fired in nine minutes. I'm sorry, Dave. I've been watching your work carefully for six months you've been here. What are you gifting that? I told Mr. Potter when he sent you here that I was afraid you hadn't had the experience to handle mining men. Being the only, he was privileged to overrule me, of course. Don't you see? But I think he'd agree with me now that you're scarcely the man for the job. Mr. Wilkins, that ain't fair. It ain't right. I worked hard, done everything I could. If I could just get cooperation from these fellows. Exactly. If you could. The right kind of foreman would. No, Dave, as I say, I'm sorry, but I'll have to give you a notice. You can't do that. Of course, you're a personal friend of the owner. But under the circumstances, I'm sure he'll back me up. This is the rottenest deal I ever got in my life. And no use being bitter. Top lasers with being bitter. That ain't it at all. You blame me for not having authority over the men.  How can I have authority when you do that? We'll not discuss it further.  That's a good one. You can consider yourself released from this moment, Dave. You'll pay a week or two in advance, however. Why, you mealy mouth, two-time and pole-cat. I already... Come on, Dave. I can hold it. This man is your boss. He has the right to fire you if he wishes. Not like that he ain't. You start anything here. You're in the wrong. A lowdown. You'd better take the advice of your outlaw friend, Dave. And when you go to Mr. Potter with your story, just remember this. We didn't fall down on production until you came here. And thanks to you, Mr. Potter may have to sell, for which I don't think he'll thank you. Now, clear out. You're coming along, Dave. You want to make cover me another better time for us. Well, there goes our foreman, Mr. William. Man enough to bust. And nothing he can do about it. Outside the mine, Dave parted from the masked man, and with a heavy heart, drew the money that was due him, gathered his equipment together and left the property. That evening at home, his wife tried to console him with... Now, Dave, don't you take on so. Nothing happens but what is for the best. The best. You know what this means, Mary. Well, it means I'm through as a mining man. It means there ain't another outfit in this part of the company that'll hire me. I'm a failure. A failure. If it goes someplace else. The story'd soon get around. Mr. Potter was a good friend of your father's, Dave. Sure. That's why I give me the job in the first place. You can tell him your side of it. Shucks. I ain't no hand at writing. He was here so I could talk to him personally. But he's coming here. He might even drop in to see us tonight. Is that a fact? Well, how do you know? I got a letter from him this morning. I was saving the news for you. Gosh, that's swell. You see, I told you things weren't so bad. Now you'll be able to tell him how Mr. Wilkins kept siding with the man and how he blamed you for things that weren't so.  Just see if he don't. Yeah, but I'd hate to have to work with Wilkins again. That'll be all right, too. After Mr. Potter tells him how to treat you. Where is that letter he wrote? When do you say he'd get the pine bluffs? You needn't get the letter. I can tell you what was in it. He said it'd be in by the afternoon stage. But that was hours ago. He'd go out to the mine first, wouldn't he? Yeah, that's right. Dave, I'll bet that's him now. Mr. Potter, come in. I knew it was you. I just told Dave so. Good evening, sir. Good evening, Dave. You're looking good. Here.  Thank you. Well, I, I ain't feeling so good though, Mr. Potter. Mm-hmm. I was at the mine, you know. Then you know how awful that Mr. Wilkins treated Dave and Potter. Please, honey. I'm sorry, Dave. You know that, of course. Wait. You mean you're taking his word for all that's happened? Dave, I was your father's best friend. And I'd like to talk to you like a father. But still don't you see that I... One moment. You know, Dave, I think the world of you and Mary both... You've been awful good to it. Only done what I could. But what I wish to say is this, Dave, don't be too ready to blame others for your own mistakes. Then you do believe Wilkins. He's been with me a good many years. But you ain't worked with him like I have. You don't have to put... Every young fellow beginning life has to take some mighty hard bumps. If he's the right kind, he takes them, marks them up to experience and tries over again. But I didn't fail. That's just what I mean. Frankly, Dave, you did. The record show it. I think you would be wiser to go after something else. That doesn't require so much practical knowledge. That isn't fair. Dave did his work as good as anyone. I gave the job in the first place against Wilkins Advice. But even so, don't you see that... Hear me out, Dave. I don't think you quite realize the position I'm in. If things hadn't gone so badly these six months that you were a foreman, I could probably have afforded to keep you on until you really learned the ropes. Well, if you'd give me another chance, I'd make good on it. I'd even try to get along with Mr. Wilkins. Perhaps the next owner will give you the chance. The next owner? Well, Mr. Potty, you're not going to sell. I think I'll have to. But why? The Gold King's one of the richest minds around here. And in the last six months, there wasn't enough ore taken out to pay expenses. Things happened to hold us up. But even so, I figured you must be making plenty. Scarcely that. As I said, I didn't make expenses. Over and beyond that, I contracted certain obligations that must be satisfied at once. If production was stepped up to... It still wouldn't help. I must have the money at once. So I'm afraid there's no alternative but to sell. You got a buyer? Colby's been interested. Colby? Ain't he the fella that opened up the old lost nugget mind and built his own smelter? That's the one. Gosh, I'd rather almost anybody but him got the place. It's got to be sold. I never did cotton to that fella. I don't like him much myself. He's been doing well, however, and seems to be the only man able to pay the ready cash I need. Oh, that's awful tough. Don't worry about it, Dave. But, Mr. Potter, I feel as though it was mostly my fault. If I'd handled my job better, then maybe you wouldn't have gotten to this fix. I'm so sorry, Mr. Potter. Forget about it, both of you. All of us must have our worries, I guess. You have to leave already? I'm afraid I will. Had a long trip. I'm tired. I have an appointment with Colby and his lawyer first thing in the morning.  Oh, of course.  If Colby and I come to terms, I'd like you to be there's witness to the sale. You mind? I'll have to be at the office by nine. Mr. Potter, I don't mind at all. I'll be there, but I sure wish you didn't have to sell. Amos Potter, accompanied by Dave, kept his appointment with Colby the following morning in the office of Colby's lawyer, Hanford Blake. Colby and Potter found it difficult to come to terms and. Colby, the Gold King's worth two hundred thousand dollars if it's worth a penny. You're a mining man. You should know that as well as I do. You're quite right, Potter. As a matter of fact, I'd say your mind is worth considerably more. Well then. But my offer remains the same. One hundred thousand. Don't sell to him, Mr. Potter. That's the same as robbery. Young man, I understand you was here simply to witness the sale if one was made. That's all right, but I say. I'll handle this, Dave. Colby, you can afford to offer more. You'll have the money. And something in this instance more valuable than money. Yes. Information, Potter. Information. Meaning? That I happen to know you must sell. That you haven't found anyone else willing to make the first payment you're asking. I see. I suspected something of the sort. One hundred thousand with thirty thousand down there as the thirty thousand in currency. Just as you said it would have to be. Take it all leave it. Don't, Mr. Potter. I must, Dave. And as long as Colby knows I must, there's no use holding out longer. I have the papers here. You'll find them all in order. Now, if you just put your signature where I've made that. What's the meaning of this? I'll take that money. Oh, hold it. Careful. I never figured for a crook that I'd pull a stunt like this, Mr. Potter. You're coming with me. I don't get it. Come on. Let go of me. Hold on. Wait. Potter, I'll see you again. Wait. Thirty thousand more. After him. Get the sheriff. Report this at once. I'll see you. There he goes. And if we don't get that money back, I'm ruined. The curtain falls on the first act of our Lone Ranger drama. Before the next exciting scenes, please permit us to pause for just a few moments. Now to continue our story. The masked man riding double with Dave Muncie and with Colby's thirty thousand dollars in his pocket, then Silver racing toward the well-hidden camp where Tahto waited. There he reigned the powerful stallion to a halt. Oh, Silver. Oh, you see, if you... Get down. You'll get caught. You won't get away with this. After I've taught you, you may not be so anxious for me to be caught. You'll get cash. Here it is, Kimosami. Put it where it'll be safe. I just hope them bills are marked. I hope you're jailed when you try to spell Fendham. I'm neither spending that money nor keeping it. And what you steal it for? For just one reason. To keep that sale from going through. Huh? I've reported the theft of course and the sheriff will be looking for us. We'll wait until the search has died down and we'll ride to town again. What for? To give the sheriff that money. What? I don't savvy. Dave, did it ever occur to you that Potter might have been the victim of a crooked scheme? A crooked scheme? A scheme to force him to sell? He's had bad luck, but that don't mean there's anything crooked behind it. In the mine yesterday, I said I thought I'd seen Wilkins before. The manager of the mine? Yes. Yeah, I recollect you saying something of the sort. What about her? Were Wilkins and Colby good friends? Friends? Well, I should say not. That's about the only good thing I can say for Wilkins. That he didn't say anything good about Colby. I thought as much. What do you mean? Before Tyler and I made camp here, we were camp near Colby's lost nugget mine. Yeah? And Wilkins called on Colby at least half a dozen times while we were there. More than that, they were very friendly. Are you sure of that? I am. I have no proof, of course. I'll have to ask you to take my word for it. Well, it could be all right. Yes? It'd fit in. If Wilkins wanted to slow up production, so has Mr. Potter to have to sell. Well, he was going about it the right way. Can you give me some further information about that? Wait. Well? The crew at the mine. I was just thinking. Stranger, did you know that almost every one of them fellows has been hired in the last six months? That Wilkins got rid of all the men that had been there before? I hadn't known it. Well, that fits in to you. You just bet it does. That would explain why those men weren't afraid to go against your orders. They were in on the plot with Wilkins. And I knew they were safe. That's just it. Uh, you think a lot of aim is Potter, don't you? He's been one of the best friends a fellow could have. And together we may save his mind. Oh, gosh, no. There's something else I just happened to think of. Yes? Saving the mine won't help him at all. It's cash he's got to have in real quick. Gosh, I was near forgetting all about that. You get the mine, Dave, and the money he needs. Yeah, well, how? I'll tell you when I can show you the proof at the same time. You can take my word for one thing. Slowing up production at the Gold King with just half the scheme against Potter. When we uncover the whole scheme, Potter's worries will be over. The sheriff scoured the countryside with a posse in his search for the mysterious masked man who had disappeared with Colby's money. One evening came he was forced to return to Pine Bluffs empty handed. That night, however, the masked man was not idle. Procuring a horse for Dave, he sent the young man into town. Then he and Tonto raced across country to make the first of several lightning swift calls. Their great horses plundered across the prairie, never pausing until they reached a small cottage near the Gold King mine where Wilkins lived.  Open up. The masked man. Where's your horse? In the lean to ride. As badly as once, you're riding. From the mine towards town they raced, Wilkins closely guarded by Tonto. In town, the masked man led the way to the hotel where he and Potter were staying. They approached from the back, circled to the side and halted. The masked man, leaving from the saddle and calling through Potter's half-raised window. Potter! Who's there? Get dressed. Who are you? What's the meaning? The masked man. Right. Wilkins. Wilkins, what are you doing there? Never mind that. If you want your mind and the money you say you must have, get dressed and don't waste time. Colby, staying overnight at his lawyer's home, found himself rudely awakened, forced to dress, then trust only half-awake into the saddle. What's this all about? Potter, is this your dude?  Well, I wouldn't... We're heading for your mind, Colby. The law snuggled. What's that? I meant just what I said. Now get going. Get him up, scum!  In the meantime, Dave followed the orders, given him by the Lone Ranger, and hurried to the home of the sheriff. When the sheriff admitted him, he stood to a table and made an open envelope before the law man's astonished eyes. Take a look at that, Sheriff. Cash? Hey, let me look at that. You recognize it? Why? Well, this must be the cash that was stole from Colby. It was taken from Colby, all right, Sheriff, but it wasn't stole. Of course, too. The mask man got it. Say, how'd you come across this? When the mask man took the money, he took me along also. I know that. You bust loose from him, did you? He sent me here with this. You must be local. I'm telling you the truth, Sheriff. He gave me this and told me to give it to you. But what for? Did you figure we was getting too close to his trail and he didn't want the posse after him no more? No. Then what was the...  Now I know you're local. That's just half of it, Sheriff. You're gonna help the mask man prove it. Maybe you'd better do some explaining. This is getting beyond me for fair. Just this, Sheriff, you and I are gonna get the man from the assay office. Frank Seym? Right. Then the three of us are riding out to Colby's mine. Where's the mask man now? He'll be there waiting for us. I don't know. This sounds mighty funny to me. Bye, Thunder Sheriff.  Now let's go. We gotta make tracks. Father, Colby and Wilkins all rode to the gold-megged mine under protest. But the mask man and his faithful Indian companion, Tonto, gave them no opportunity to escape. The mask man called the hall at the entrance to the mine shop. Let's go. Colby, if any of your men are waking, you ought to tell them everything's all right. Understand? This is a no-treat. Maybe. But you'll do as I say. Tonto. We'll need a lantern. There's one over there marking the powder shed. Get it. Tonto, get it. Look here, stranger. You've made all three of us your prisoners. You've given no explanation. I, for one, demand an explanation at once. You'll get one very soon, Potter. Mr. When the law catches up to you, you're going to stretch hemp just as it shows you're born. The sheriff will hear about this, won't you? Well, you won't have to go into town to tell him. The sheriff is coming here. I'm here for. I think you can guess, Colby. What I am, police. A lantern, thanks, Tonto. Colby, I see you have a door across the entrance to your mine. Of course I have. With a strong lock on it. Do you want to unlock this door or have us break it in? You can't do that. I keep quiet if I were you, Wilkins. You'll need your breath to explain to the law later on. That's private property. You have no right to enter that shaft. Hand me that barge of your feet, Tonto. Keep these fellas covered. Tonto, do it. We'll get inside this shaft. Oh, wait. I don't lock the door. That's better. Go right ahead. I have my keys right here. There. Now open the door. I don't see one. Open the door. Yes. All right. Walk on in. You too, Wilkins. And you, Potter. It's dark in here. I have a lantern right here. You're going to lock us in here? No. I'm just going to take a look around. Colby, I see you have quite a bit of ore piled here near the entrance. What if I have? This ore came from the lost nugget? Of course it did. One time, this mine was supposed to have been played out. You've been doing quite well with it, haven't you? The vein wasn't played out. There was just a fork in it. We drilled another 50 yards and found it again. I see. That's more than I do. What's the sense of this? Right now, we're waiting for the sheriff. And that's probably him now. There's more than just the sheriff. Dave Munsey is along. Dave? Yes. And the third man, I think Colby, is going to be interested to meet. If that's really the sheriff, it won't matter who's with him. You'll go to jail. At least one of us will. One of us? Well, what do you mean? You'll soon learn what I mean. Sheriff, this way. Inside the shaft. What's going on here? Sheriff arrest this man. What for? He stole Colby's cash, didn't he? Shucks, maybe he did, but he ain't got it now. What? Found the cash? Mr. Potter, the masked man never stole the cash. Not to keep that, is it? But just to keep you from selling your mine. Now, where's it now? The sheriff's got it safe to home. Where are the... Dave? Dave, is this the man I asked you to bring along? Uh-huh. All right. Who are you? Oh, my name is Frank Savins, Mr. Potter. I never heard of you. Well, you should have. I've signed more than one assay report for ore that came from your gold king mine. He's headed the assay office, Mr. Potter. Well, what if he is? Frank, I've been told that an expert familiar with the ore from various veins can always tell from what mines the ore came. Oh, I should say so. I'll do better than that if I have to. You just show me gold dust. Nine times out of ten, I can tell from that alone. In other words, it would be impossible for you to make a mistake about ore. Well, I've never made one yet. Why? I'll hold the lantern up so you can get a good look. Now, examine this ore I've just taken from my pocket. It's a small sample, but perhaps you can tell me where I got it. If you got it from anywhere around this district, I'll tell you. Sheriff, I protest this. This mask man forced his way in here. This is private property, my property. I want him arrested for trespass, and I insist that we leave here. This is all foolishness, and I refuse to have anything more to do with it. Kobe's right, Sheriff. I'm forgetting now. Don't move. I won't draw unless I have to. Try to leave, however, and I will. Are you going to stand for this, Sheriff? I tell you, Mr. Kobe, I'm probably interested in some of the things the mask fella's been saying. Then I'll order you out, all of you. You, too, Sheriff. You can enter my property against my will without a warrant. Getting awful legal, isn't you? Kobe's forgetting that his law won't stand up against six guns. That's enough from you. Frank, what have you found out about that ore? Well, this comes from the Gold King. I couldn't mistake this. Right. I helped myself to that when I was there this evening. Now look at the ore on the shaft here. Where did that come from? From here, didn't it? Examined it. Come on, Wilkins! They're making a break for it. You get Kobe. I'm going to get him for my hand. Come back here, you two. You're covered.  Just step back your easy life. Another break and I'll drill you. You're arresting Kobe and my poor man, but why? Mr. Potter, here's the answer right here. What? I don't know how it got here, but this ore on the shaft came from your mind. I'll stake my reputation on it. If there's any doubt about it, Kobe can call in anyone he wishes to check with me. Or from a Gold King here in Kobe's mind? Father, Kobe's been stealing from you for months. This mine here was just a blind. He gave out the story. He found a continuation of the vein that it played out to explain where this ore came from. But Kobe couldn't have stolen from the Gold King without Wilkins' knowledge. It would have been impossible. That's just it, Mr. Potter. They were in cahoots.  You need cash. You have a legal right to everything Kobe has banked in town. It came from your gold. But the whole thing seems incredible. The risk they took... They took practically no risk. Why do you suppose Kobe built a smelter of his own? He did it because if he'd sent the ore to the smelter in town, as you do, his trick would have been discovered. Here he could reduce the ore to pure gold. And no one but his own men would know what mine that gold came from. Kobe, if I were ten years younger, I'd give you the beating of your life. Let me do it, Mr. Potter. Just let me teach this concoct lesson. When I get through with him, I'll take on Wilkins. Don't come near me, Blasher. That's all right, Dave. I reckon the spare leg gonna get in the jailhouse and teach him lesson enough. Besides, Dave, I won't have a man who works for me getting in the brawl. Who works for you? Mr. Potter, you mean I got my job of farming back again? You have not. Well, I didn't expect quite that much to start, but I... You won't be foreman, Dave, because you'll be too busy managing the mine. Huh? I'll be manager? Oh, golly, wait till I tell Mary about this. And the mask man, I gotta thank him. Well, he ain't here. Now, why'd he leave like that? He always does, Dave. You know him? Not by name. But you heard what he's called, I reckon. That mask fella was the Lone Ranger. They're waiting for us on the train ahead! You have just heard of a copyrighted feature of the Lone Ranger Incorporated.


Hello everyone. We're doing the first talk of today's Observability Day. We're going to talk about the overload in observability data, how to deal with it, and about our experience doing that. So we work in Red Hat on Kubernetes observability, mostly finding relationships between different signals, insights, intelligence, and so on. So first, check how many of you raise your hand have worked with time-serious data, Prometheus, Thanos, programmatically? Okay, well that's probably a redundant question to ask on Observability Day, but now how many of you have worked with any data science tools like probably in Python, pandas, numpy, tensorflow, less. So you're the target audience. So we're going to talk about tips and tricks how to work with, how to do data analysis, data science, stuff for observability data specifically, for time-series. So we are, we work in general with all of the observability signal logs, metrics, alerts, traces, and so on. And we talk with the customers, admins of Kubernetes clusters, but we also talk with our internal SRE teams. So site reliability engineers who take care of Kubernetes clusters for our customers. And over the years, we have accumulated some feedback from them that they say it's difficult for them to operate our observability stack as difficult to find things. So the theme is, so they say that there is a lot of data, so they have to deal with alerts, metrics, logs, events, configuration changes, day-to-day, traces, and so on. So it's good that we have a lot of observability signal, but it's just too much for them to figure out quickly what to do with it. And when they finally get paged in the middle of the night, they have an alert, they go check the associated metrics, they check the logs, but what they usually see is the current symptoms after the problem has happened. But sometimes the real issue was the day before, a couple of weeks ago, so they have to go and check the history that is not always evident. And finally, they manage thousands of Kubernetes clusters, and at any given moment there is one engineer responsible for up to 500 clusters, so if there is an issue, they're expected to solve it efficiently and fast. So as the fleet grows, as the amount of clusters grows, we cannot expect to scale the SRE team as well. So it has to be more efficient. And this is how our SREs look when they wake up at night and have to solve something. There's just so much things they have to take a look at. So what can we do to address these problems? Over the years, we've tried different approaches and you can probably guess the most popular solution right now. So, but that doesn't always work. So lessons we've learned over the years is sometimes a person who really knows the data can do a simpler analysis of the data simple baseline model that is, that performs and gives better insights than throwing terabytes of data, of unstructured data at an AI model. We've also learned that while the dashboards, graphing, metrics and so on is useful, it's without proper down-sampling recording rules to make it easier to look back months. It's going to be really slow and you usually have to look at it metric per metric. And we found ourselves doing the same thing over and over. So getting raw Prometheus metrics, transforming them, down-sampling, combining and so on. So what we decided to do is artificial intelligence is good and it has its own applications, but it's not always feasible and you don't always have the time. So we are trying to do what we say, what we call observability intelligence. Some of it is addressed by AI models, but some of it can be done by a simpler analysis by someone, an engineer who knows the data well. So imagine a typical data scientist, which probably none of us are. They spend months working on a specific problem, they get the data, they get acquainted with the data, they reshape it, do some data engineering, then they start trying out like five, ten different algorithms, then they do accuracy estimations. So they have all of those tools that they apply to the specific problem that is not native to them. And it usually takes months. So what we try to do is give engineers in the field a series of admins who know the platform, who know the data and can code some simpler tools to distill the data in some manageable insights. So after all those years working with the telemetry data, we have accumulated some snippets of code to transform it, to work with it. So we decided to gather it all together and publish it for other people to use. So as of now, it's mostly focused on the time series data, but Ivan will show what the current state of the project is and share a bit on where we plan to take it. So right now it's a Prometheus client that handles authentication, fetching the data. It has caching because the if you want to get something longer period, it will take longer time. So it also converts from native format to more data science, native formats like pandas, data frames. It also has, we had to add some domain specific features like for handling alerts. It has visual visualizations and for testing some analysis. We also added simulations so you don't always have to rely on the live data. And now Ivan will show the demo for what it can do. Let me switch to some live demo thing who doesn't like live demos and all the things that can go wrong. So the time seems to be good. Perfect. So I will go through what we have right now in a library called Opsynth which we just released and it's basically taking all these lessons and tools and approaches that we've learned and happened to copy over and over over the last couple of years and put it into one library. And we hope that this would be able to give you more tools that you could actually get hands on the data science and see how it actually can help you with working with the observability data and it's not that complicated that it might sound at the beginning. So you can see that I'm doing data science because I have a Jupyter notebook opened and you know when somebody has a Jupyter notebook opened he's doing data science. So this is the notebook that you actually get when you get to the repository that we have there. You will get the links at the end. It's in Opsynth slash Opsynth GitHub. So don't worry you don't need to take photos of this and then trying to rewrite it at home. You will get everything as I do and you should be able to run it. So I would act as I would be talking to Prometheus, real Prometheus, but I you know I like adventures, but not that much. So I actually have used one of the features of the Opsynth which is being able to simulate the data as it were real. So actually I prepared synthetic data that act as some others that were happening in the fleet of clusters and I will be doing the presentation on top of that. But don't worry we don't do it just because you know we formatted the data nicely so it fits well. You should be able to replicate the same thing and we do the same thing with real data. It's just more convenient for presentation as well as for testing and other stuff to have tools for synthetic data as well. So just keep that in mind. So the first thing that you need to do is load the data. So I actually chosen to load the dates or the data between two dates and the first thing is basically just preparing the client code. So we have the client that talks to Prometheus and we have a wrapper class on top of that. If you are not familiar with Python too much, don't worry. It's not really that complicated. It's just some loading class that handles some additional functionality such as splitting the queries over a longer period of time. If you want to load weeks or months or even a year worth of data from Prometheus or tunnels that we are using, you might find that it's not that efficient to do it at once. You just have so much data that you don't want to spend too much or you usually can overload your cluster and you can either turn the data into data like I do this way, but you can also just chunk the queries over per days and then load the data one at a time. And that's what we are doing here as well. I could enable caching. So if I need to repeat that or I want to somebody else to replicate what I've done, I can enable the caching and the next round of the notebook actually would be much faster because instead of reaching to the live cluster or live Prometheus instance, I would have the data locally already. So that's just the preparation part and then I would load the alerts data from Prometheus itself. You can see it around quickly because this time I'm actually using the synthetic or obsynthetic data here. And you would get something that you would kind of expect. We have the columns that represent the labels from the metric, from alerts in this case, and we have the values which look very similar to what the vector ranges look like in Prometheus. We have the timestamp and the value in this case is just one as the alert is firing. So nothing so fancy yet. But this is not the best format to start working with getting some insights. And what we'll be doing here is trying to find relations between alerts. What you might observe, especially in Kubernetes world, but I guess it's pretty common, whatever you are operating, it's usually not that just one alert fires and it represents the problem. Many times the alerts are more symptomatic and the root cause actually might cause more alerts to happen at the same time. So what we try to do is actually identify the alerts that happened around the same time and then finding relations between them. And data science turns out to be pretty efficient on actually doing this. But it's not for free. You need to do some work first. So what we will do first is turning this time series into something more convenient to work with. So we'll do some data transformation. So what obsynth does is providing some functionality, some methods that help to switch the alerts ranges collection or the other ranges into the intervals. So out of the box, you turn this, which is just, you know, this timestamp is firing, this type of thing, you start firing, you turn it into each timestamp or each time series and knowing when it started and when it ended, which is pretty convenient, especially for the alerts. We always need to do something like that. For our use case, it's much more useful way of working with the data set. Once we have this, we can combine this kind of daily data together. We still like, you might remember that we loaded the data one day at a time. So we have actually the data ranges in collection and we want to concut them together. So in the next cell, we'll actually do that. Again, another function from the library and we'll also add the resolution time to the end of the alert, just for cases where the alert happens there just at one timestamp. So we'll do just some corrections against something that we've learned over the time. So now, like, similar thing, but we have over the whole period of time, you can now start doing some data science on top of that. So in order to do that, we need to somehow identify each alert, give it some, like, what are the most important parts of the alert that represent the issue, because sometimes you might have just, you know, the instance or the pot or container name or pot name or some random value that doesn't help you much when doing the aggregations. So we picked, like, what are the most important parts of the alert and we assigned the alert ID based on that. So in this case, which is the instance ID, which is supposed to be, you are running the same service in multiple instances, and then the alert name that represents the issue. And once we have that, we can actually visualize that. That's something very powerful, like, we've, another thing that we've learned that sometimes just doing the right visualization and being able to do it quickly helps you a lot of time, like, not in this case, because you just, you might not even see the noise, but I can just zoom it in to show you, like, how it looks like. So basically, we have different alerts that are happening. For some reason, Cursor is not showing up here for me, but I can do it anyways. So we can see that some alert happened. It started at May 20, March 20, which is tomorrow. And it lasts for 30 days. Again, I'm working with the synthetic data. It will be more up to date tomorrow than it was today. We are looking into the future. Anyways, this is one particular instance. And you can see the example of some problem happening, multiple alerts firing. This is the case that we would be identifying as one group. So in the next cell, we'll actually do that. And we'll do something that data scientists call one hot encoding, which is a very fancy name, but which basically just tells for each alert that we have here to assign one or zero depending whether it was happening in particular situation or not. So, for example, in this particular group, there was one instance that had the cube deployment replica mismatch and cube note not ready at the same time. So we have two ones there and nothing else instead. And you can think about it as the coordinates in multi-dimensional space. I like talking about this stuff. So you can imagine how much, like, 30 dimensional space and each alert having some dot there in this nice 30 dimensional vector space. So it's not that easy to think about these things, but again, options for the rescue. So another thing that we implemented there is applying dimensionality reduction and clustering algorithms, basically using those tools that are available in Python data ecosystem and using it for our benefits. So once we do that, we turn this 30 dimensional vector space into three dimensional vector space here, which is nicely visualized here. Another thing that I like about Python and Plotly is this interactive things where you can see that each dot represents some alerts, like the Elasticseer JVM heap, huge high, whatever it means. And we see here some three dots that are actually distinct from each other. And those where, you know, that we actually prepared to be, we expect it to be in this group. That's why, how we synthesize the data. But in the real case, you would start seeing these clusters of different issues. I can actually show you the similar or same graph here, which, with some more real world data, you can see it's much richer, but you can see the clusters there as well. So they actually are able to represent these kind of issues. But getting to our nice little demo, we have three things, you know, made it to make sense. So we have the cube note, not ready. It's one alert, and we have target down as another one for people working with Prometheus. They've seen this kind of combination before. I'm pretty sure. But we have other things that we have there in this data set that are still kind of hidden. For example, the Elasticseer JVM heap use hi, which is another example that actually chosen to, that should be part of this group. And for some reason, it's not there yet. Like, this is kind of noise of other alerts that were there, and there was not enough relation between each other. So let's see what caused that. So if we actually visualize some of the situation with this particle example, we see one thing that sometimes happens with your alerts, which is the alert being flapping. For one reason or another, the alert can go up or down or up and down depending on the metric. Sometimes you can't avoid that. It's just not part of how things are. And when you're doing the data science on top of that, it makes some troubles like this. It adds some additional groups that are actually not really a group because in this particular time, let me see here, like the issue was already there. The fact that the alert was hitting or started at 4.13 was just because of the flappiness, but it really started much sooner. So another thing that we have there in Opsynth is some mechanisms to combine this together. So after we do this flapping reduction, we, even for this flapping thing, we know when it started, and we can treat it as any other thing that we have there. So we just re-apply the same mechanism as we had before. This is the previous visualization of the clustering. And after we do the flapping reduction, we have a bit different picture. It's similar, but, and we have, you know, these things that the target down, down, it was there before. But now the new key to the block is here, where the elastic search related alerts are, you know, there are a lot of things that happen to be in a separate cluster as well. So this shows, or this is an example of how doing some additional processing at the beginning can lead you to more insights, regardless of how fancy algorithm you put on top of that. So sometimes the data cleaning and visualizations are much more important than using some very sophisticated data science, machine learning approaches to do the stuff. Otherwise, you can also have a look at some of the things that we have done. So the end of the demo, again, you can go to our repository and round the same thing. As we are actually using the synthetic data, you should be able to do it really as we did. But hopefully you would also be able to plug it into your infrastructure and see how your alerts look like. So that's for the demo part. And let me just finish with the slides here as well. So what we have done is we are going to do the demo part now. Just a beginning. And the vision, and we are really curious whether it's something that would be interesting for many people here, or some people that might be doing similar stuff, is to build this comf2 that would be able to plug data science ecosystem. We already are using pandas, plotly, even scikit-learn. And again, much more tools are available there. And use that to synthesize and distill the insights from those related data. And the Opsynth is quite a nice name for that. So again, we are at the beginning of this thing. I actually paid the plain internet just for the sake of this picture so that I pushed the first version from the cloud. So this is really kind of cloud native. You can't get more cloud native than this. So we released the first version yesterday. And we invite you to think about this, talk to us, give us feedback, think about your use cases, think what could be possible, and let's see if we could put this into something that could be reused across the community. And we are getting to the end of the talk and now it's place for your questions. Are there any questions? If there are, there are two microphones that you can use to ask them. In, otherwise, we are still here, the whole conference, so feel free to reach out to us and we can talk much more about like what are or what would be your thoughts. Thank you very much.


Hello and welcome to MPS Goes to Casa Pelusa. I am delighted to say I'm here with Sleep Thieves. Guys, how are we getting on? Great, great. Really good. Guys, can I ask you, it's just a curious question of mine. Sleep Thieves is such a fantastic name, but where did it come about? Well, maybe it's not a very romantic answer, but we were looking for something that would describe the music as well as kind of being lasting. So at the time, we were looking through loads of footnotes in the book I was reading and it came up as the name of the title of an essay. And it just seemed to fish and work, and even though our sound has changed over time, it still kind of fits with the sound. That kind of electronic, that idea of being kept up late and wanting to, it's not in a nasty way. It's like, I want to stay awake and listen to the end of this album or listen to the end of the song. So that's the idea. That's really cool. I'm glad you cleared that up too for us as well. There's a lot of fights about this this morning. And your music, as you were talking about it there, it's very disco noir, very cinematic. But how did you bring that to a live experience? Synthesizers, just loads of synthesizers. 80s one. Pretty much 80 synthesizers. That brings the disco and the noir part to it, you know. Yeah, moody light and that also brings the darkness. Yeah, no red. Yeah. Blue, no red. Yeah, blue, no red. Okay, that's cool. Blue, no red, no red, we'll forget about that. And guys, your music is very cinematic. But if you have the chance to put one of your songs to one of your favourite films, what would it be? I'm thinking Michelle Gondry film. So, you know, something in the, because that kind of weird genre of crazy things happening and all the art and I love the characters he picks and the actress he works with. So I think, I don't know if you guys agree, I'd go with Michelle Gondry. Yeah, that's a good choice. We're right here Michelle, we're ready to write a whole soundtrack. Hopefully Michelle now will pick this up and this will be a start. You can thank us for this question. Great, well guys, just as a final thing, if people want to check out Sleep Thieves, where can they check them out? Yeah, we're online, iTunes, Spotify, all your usual places, sleepthieves.net. On Facebook, we're on Twitter, we're quite active on Twitter, so have a chat with us on there. We usually answer every tweet, that's one of our things, we will answer every tweet. Or your money back. Or your money back. Brilliant guys, so this has been MPS Goes to a Castle Belieuser and thanks very much to Sleep Thieves for this interview.


All right, cool. I'm gonna go ahead and get started. This is sort of a rehash of what I talked about yesterday, but not really because I emailed Derek and got him to give me his list of complaints. So, some of the stuff on here, it's stuff that we've talked about. For instance, like the Continual Integration Testing is kind of a giant pain. We're all sort of setting it up ourselves. We can talk about this later in the FS tests, sorry, FS tests section. One of the things that he brought up is the stable thing. Are we actually expected to backport to all six releases of stable? We've talked about this in another session, so I'm not really gonna kind of rehash that. One thing that he did bring up that which hasn't been talked about is he's seeing an increasing number of people fuzzing file systems and then filing CVEs for fuzzed file systems, which triggers our security thing, which means then we have a time limit and to post a fix, you can't pull in the entire community because it's supposed to be only like the maintainer and maybe one or two other people. So I suppose this is a pain spot for him. I haven't experienced this, but you essentially get a CVE, now you're on the security process thing, which means that all development and review and everything is done very quickly without review and without the open community or whatever. Mike, can we please do something to address this? I think this is probably a wider kernel community thing. I have always been of the opinion that fuzzing file systems does not constitute a security problem. I know I'm probably a heretic in saying that because you have to be root to mount the file system. And every time I say that, somebody says, well, what if you plug in a USB drive? Turn off auto mounts. But anyway, I think that it is kind of terrible that like when we have these sort of issues that it kind of goes through the whole security apparatus. Have you experienced this at all, Ted? Yeah, so the main issue is if your company would like to sell into certain markets, like say the US government. The US government has this little interesting thing called FedRAMP requirements. And P1 bugs have to be mitigated within 30 days, I believe it is. And then they have like these different priorities that are based on the CVE severity score. And lower severity scores, you might have 180 days to remediate it. And that's a lot more time. Note that that's time to actually roll out the fix into production. Or you have to answer to the FedRAMP auditors about why in your environment it may not matter. And that's where you can make the argument about, well, we don't enable auto mount and we're not crazy enough to let random container people mount untrusted file systems on critical security systems, whatever. But it involves a communication with a government bureaucrat. Which the security teams don't appreciate doing. Can't imagine why. And so it's a real thing. I think the good news is most of the time a lot of the fuzz images aren't necessarily going to be a super high severity score because it actually still requires, if not local root, at least local login. The ones that are super, super scary as far as the CVE folks are concerned are the ones where you can actually remotely exploit the problem. And so folks who are dealing with network file systems actually have a lot more to worry about that product. But again, it only matters if your company happens to be trying to worry about FedRAMP requirements or you're selling into the financial world and they care about that for PCI compliance and whatnot. And that's really more of a commercial problem. But the good news is when that happens, it's a lot easier for me to justify headcount. Whereas getting headcount for, this XFS test fails one in 800 times. Yeah, good luck getting headcount for that. So it's a double-edged sword. Right, so this kind of sounds like a derrick. Yeah, I didn't think that Security at Kernel.org generally opened CVEs. I thought, yeah, Greg and company were fairly against that. So if the CVE is getting opened, it's from some other security? Yeah, it's by the research lab. And the problem is some of the research labs have an economic incentive to say we've opened so many CVEs that are high severity because if you then subscribe to that lab, you get early access to their security reports. And so they have an economic incentive to file lots of CVEs so that people feel motivated to pay lots of money to get early access to their reports. So I think that's the cycle that we need to get into in terms of communicating better what a CVE is and how to define them. And I just wanna say that FedRAMP doesn't really play into Security at Kernel.org so much. I mean, they're not caring about that. So it's more of the subsystem maintainer's decision, discretion about whether they wanna fix an issue before it goes public or not. So the net dev developers, for example, they're kind of known as not wanting to fix issues privately in Security at Kernel.org. They wanna fix it out in the public and things like that. The downside there is that it allows for zero days to exist for a small amount of time in distros and downstream products. So really, it's up to the maintainer. Do you trust your small circle of maintainers to fix something correctly, privately? Or do you wanna push it publicly and do it that way? So yeah. And I think a lot of it also depends on who your employer is, right? So if your employer happens to work for a large cloud company and you're not fixing the problem means that you need to explain to a VP why you're not dealing with this high security problem. Then that puts stress on that person. But to be fair, that's not a stress put on the maintainer. That's the stress put on an engineer who happens to work for Oracle or Google or Microsoft, right? Okay, cool means. All right, I think, like everything else, we kind of resolved in the conference that he mentioned, which I guess he'll be happy to see. So I kind of wanna talk about like the, like more expand on the things that I talked about yesterday. And Chuck and I had a good conversation about this is that like what do we expect from our maintainers, right? Traditionally, it has been, we expect them to merge patches, write code, test, review, all of these things. And as, what? Backport. Backport, right. And this is a lot of work for one person. And like a lot of us have kind of ad hoc come up with different ways to address this, making sure that our developers are reviewing the patches, not just the maintainer. Make sure we have good testing setups, this kind of thing. One of the other things that I wanted to call out and that I called out yesterday is like making sure that our maintainers are kind of actively involved in maintaining the community itself. Like the, we're all very driven and very passionate about the work that we do. And that passion sometimes leads us to get a little short of each other, right? And this, that's not really great for like good, maintaining good working, working relationships. And that's why we have stuff like this. Like it helps to be in the same room, to remember the other person as a real person and to kind of like resolve that. But I would love to see us move more towards taking care of these problems before they happen and not having to resolve them after they've blown up or having to wait till a conference to resolve them. You know, it's not possible for a maintainer to keep track of every email thread, of course, but for big things and big stuff that may be contentious, it is kind of easy to spot. And if we can encourage our maintainers to either themselves or find something in the community that's good for this play mediator to make sure that we don't have our developers picking fights, not picking fights, but like getting overly passionate and getting overly invested in the code. Because in the end, this is just code, right? We need to accomplish our tasks. We need to do the things that we're trying to do, but we shouldn't be getting upset with each other. And if maintainers can kind of be a little bit more proactive in keeping track of things that might be contentious and then making sure that people that are butting heads do it in a respectful way and in a way that they can continue to work together as time goes on, right? Sorry. So I had sort of two thoughts on that. One is that one of the things that I've actually found to be really helpful is we have a weekly EXT for developers meeting for which Derek also shows up. And that has actually allowed us to have discussions that will ultimately need to be resolved on the Linux file system developers list. But we have a chance to chat about it. Just Derek and I, maintainer to maintainer and the other EXT for folks are around. But it allows us to sort of work through various issues. Folks like Yankara is there and he actually adds a really, really useful perspective at times and then ultimately, of course, it has to go to the Linux file systems developers list because not everybody is on that particular call. But that is really, really useful. And I wonder if some kind of, I don't know, monthly, bi-monthly, quarterly sort of a chance for the developers to actually have a chance to chat might be helpful. I know we all need another meeting like we need another hole in the head, but it's helpful. It really, really is helpful that Derek and I have that chance to exchange things. And he just happens to attend the EXT for developers meeting because his work at Oracle, he has to worry about EXT for bugs too. But maybe that's something we couldn't consider expanding. I absolutely agree. I think some form of this would be, if this is in general useful for file system and be a best people alike, that'd be really useful. I think I've ran into this issue where I've had royal conflicts on the mailing list and my observation is that in very, very rare circumstances actually a third person comes in and is like, maybe you should all come down a little bit and that's a big problem in the long run. Yeah, and I think that the other, in your particular case, it's hard, right? Because I think for, I can think of one conflict specifically, like for that, I don't feel like I, like it's, how do I say this? We need to make sure that it's okay for random people to pop into a conversation and be like, all right, everybody calm down, right? Like, okay, I'm not the VFS maintainer, right? And I'd have no skin in the game or whatever, but it's like, if I pop in and say, okay, guys, sorry, let's all calm down, let's bring it together and kind of work out the issues, right? Like I wanna make sure that we're kind of encouraging that, like encouraging all of us to do this because it's easy for us to go and say, well, Linus should do it, right? Because I mean, you're probably closest to Linus, right? But I think expecting one person to do that is not awesome. But on the other hand, again, I'm not always paying attention, right? Like your stuff I saw well after the fact, right? And I think it, I wonder if it's a thing where we can kind of encourage people to say, okay, if they're in this, call for the third person. Call for a, you know, a maintainer or somebody to like, hey, we're having some trouble here. Can you help us work it out? And I think this also goes with the technical thing, right? Cause there's kind of two problems that I see that maintainers really should be stepping in on. One is the conflict resolution, but it's also like technical direction, right? Sometimes like you have two developers that like don't agree on what the best way forward is. And that's where I really want my maintainers to step up more and say, okay, we're gonna, this is the path forward. Don't care what it is, but like the maintainer has enough context on the whole thing and they can come in and say, okay, instead of letting these two developers or three developers or whatever continue to butt heads, the maintainer steps in and says, okay, this is the direction we're gonna go and gets agreement and we all move forward, right? Yeah. The other thought that I wanted to bring up is I do agree that in the ideal world, the primary goal of the maintainer should be conflict resolution. But I guess the model that I've always had, and I can definitely see how this can lead to burnout, is one of servant leadership. So the maintainer does everything that the rest of the community isn't willing to do, right? In the ideal world, I would have a test engineer working on tests for me. I don't, so I do test engineering work, right? And that's because I know it needs to happen and no one else is going to do it if I don't, so I do it. But, and the other thing that I do is I spend an awful lot of time recruiting people to my development community and encouraging other companies to staff people for that particular file system. So I'm working with an unnamed cloud company, not my own, to try to get some cloud engineers to work on EXD4 and hopefully join our weekly call. And that is probably something that other maintainers should think about doing is the cross company sort of inculcation and sort of encouraging other companies to try to bring people to the table and to the extent that other members of the community can help the maintainer in those two activities. I think it's actually a great way to try to reduce burnout, right? Because half of the problem is we just simply don't have enough people on some of the file system teams. Yeah, and what I really want to do is reduce the amount of things that the maintainers are doing so that they can focus on these things that are arguably what they're good at, right? Like Ted, you've been here forever and you are not argumentative, right? So conflict resolution and technical direction for EXD4 is probably what you should be doing the most. And the more we can develop tools in the community to do things like CI and automated testing that offloads the testing bit from you. And you already encourage your developers to review that offloads the review thing. And so like the more things we can, because we can automate a lot of these things, what we can't automate is the human parts. We can't automate conflict resolution. We can't automate technical direction. I don't know if you guys ever saw the mission statement of LTP project or in general how their maintainers interact, but their spirit is in general and specifically in the statement. If you want, just give us a demo of a test and we'll make a proper test out of it. And they keep constantly cleaning their tests and really if I submit the test, I submit tests that are already pretty clean, but they're doing the extra work afterwards and if there was something like that going on, if you could get headcount to do that work for FS test, because if you look at the people that clean up FS test right now, you will find several high level maintainers, high level burned out maintainers that are the top contributors and cleaners of FS tests, right? Yeah, I think that the better we get about running XFS tests in a consistent manner, the sloppier we can be with accepting XFS tests. So, and that would make it faster and easier and then people can come along behind and if there's things that need to be like, I really, you know, I kind of keep harping on this. I want to go faster. I want to merge many more patches per current release and in order to do that, I need to make sure I'm not breaking anything. And okay, is my FS test perfect? No, I probably missed some helper or whatever, but it's testing a real thing. Let's get it into the tree, so it's getting.


Next question is from Johnny Humber. Are artificial sweeteners as bad for you as sugar and how do they affect blood sugar, et cetera? Ah, the big, this one's always a debate, right? Always a big deal. I know Lane's head just exploded. Yeah, I know. Lane Norton and I have gone back and forth over this many, many times. Okay, so as far as blood sugar is concerned, artificial sweeteners don't seem to affect blood sugar and there are some studies that show that it may affect the microbiome and how it responses to glucose and all that stuff, but those studies are not very good. So I think we can safely say with current research, which is a lot of it, that artificial sweeteners won't affect blood sugar. Okay, so that being said, if all things being equal, sugar versus artificial sweeteners, you're eating an appropriate amount of calories, you're, everything else is healthy, doesn't make a big difference. Now here's why I like sugar better than artificial sweeteners for most people. It's because of the behavioral effects that we can get from, that we can develop from using artificial sweeteners. So let me explain what I mean, okay? When you eat something with sugar, if you're somebody that's working on your nutrition and you're becoming self-aware about the things you're eating and you're trying to eat in a healthier way, sugar has this natural barrier known as calories, right? So if I'm watching my calories and my macros and I'm trying to lose weight and I'm like, I wanna have that soda.  I think I can only have one. When I drink things or eat things with artificial sweeteners, I've eliminated that barrier and it feels like I can have as much as I want because there's zero calories, there's no risk, there's no problems whatsoever. I can drink 15 cans of this soda because it doesn't affect my macros or my calories at all. The problem with that is, yes, from a calorie standpoint, that's true, but from a behavioral standpoint, it's terrible because these things are very, very sweet. They change how, stuff still happens in your body when you drink these things. You're still perceiving sweetness. So your palate changes, other foods start to taste more bland, you start to develop this relationship of feeding whatever feeling you have, whether it's stress, anxiety or whatever with this sweet flavor and it's not a long-term approach. And I'm gonna say this right now, I've worked with clients for a long time. I've never had anybody lose weight long-term successfully by having artificially sweetened substances over sugar. It's never been a strategy I've seen to be successful. The only time I've seen it be a successful strategy was with competitors who- That's where I was gonna interrupt you and go because I do see value with it on that. And by the way, you were the one that got me to, stop drinking diet sodas and the artificial sweetener stuff and replace it with drinks like that had sugar in it. Like Hanson Rupier, for example, was like my switch from Diet Coke. And exactly what happened is what you said is I knew that drink had like 135 or 150 calories and I can't remember how many grams of sugar in it. And because I know it had that, it kept that at bay where I would be like, okay, I already had one today, that's all I get. Where what I've always noticed about my behaviors with Diet Cokes is once I allow it into the diet, oh, I have one every other day, and then it's one a day, then it's two a day, I'm gonna get up to three, four in a day really, really quickly. If I'm not paying attention, I can do that because you justified zero calorie and it promotes those behaviors. I do see though somebody who's competing and who is weighing and measuring and tracking everything that to me, that was something that I love to, that was like a treat. I was like, oh my God, I can't have any calories. I'm out of calories today, but boy, I'll go sip on a Diet Coke right now. And that will give me that kind of sweet feeling and feel like I'm getting a treat. I used it like that a lot and I liked it for that. But for the average person who is not tracking, it just promotes the bad behaviors. And it ends up, you end up craving something else to go with that diet drink or that zero calorie food. Well, it's interesting because the artificial sweeteners tend to really ramp up that sweet reward. It's like even more powerfully sweet, which throughout the day, you're constantly thinking about where can I get my next sweet? Like it's something that your body starts to really seek for that same feeling. And so behaviorally, I'm a lot more drawn towards providing that through maybe I'll have some cookies or maybe some things will sneak in my diet that previously wouldn't have otherwise. Well, so to that point, Justin, that was another thing that I noticed during when I was competing. So I've talked on the show before about, I did a show, a couple shows like this where I did all whole foods and I did no artificial sweeteners, nothing like that. And then I did another show where I allowed like the quest bars and diet drinks and stuff in there. And one of the things that I noticed about when I was eating whole foods versus the artificial sweeteners was the taste of fruit. And when I'm allowing the diet cokes and all these drinks and quest bars, things that are all that utilize a lot of the artificial sweeteners, the taste of a strawberry or an apple is extremely bland. When I eat nothing but whole foods, now all of a sudden a strawberry, a grape, an apple, it tastes like candy. It tastes sweet, it tastes incredibly sweet. But when you're always pumping in those artificial sweeteners, then you go have natural sugar and fruit, it ruins fruit. They're sweeter than sugar. In fact, and look, I guarantee you, someone watching right now, if you have lots of artificially sweetened sodas, let's say you have a lot of diet cokes, you probably find the taste of a regular coke to be bland. And I've talked to many people. No, I prefer the taste of Coke Zero because the sugar one doesn't taste as sweet to me. They're extremely sweet, they're still in effect in the body, changes how you perceive foods. And then look, let's look at the studies. The studies that are controlled, and what I mean by controlled is, they put people in a lab and they give them everything measured out. So here's your food and that's it. Does replacing sugar with artificial sweeteners cause them to lose weight? Yes, their calories drop. But that never happens in real life. In real life, when they do studies on people living their lives, when people cut out sugar and replace it with artificial sweeteners, they never lose weight. Almost no studies show people lose weight except for when things are super controlled. Well, don't they actually show they eat 500 more calories? That's what happens. They replace it with other calories because it causes those behaviors to happen. It is a terrible long-term approach to weight loss. I've never, in my life, have I had success? Have you guys ever had success with a client? Hey, replace all your sodas with artificial sweeteners and it just doesn't work. So that's the thing about it. The promise with artificial sweeteners was people are gonna lose weight, they get to get the sweetness and they're gonna lose weight. It's never worked, it's never worked. And so that's the problem. So yeah, if everything's measured and you're measuring everything in your pre-contest, you already have dysfunctional eating anyway. Fine, go, throw some artificial sweeteners in the fire. Well, you know, because we're talking about behavior stuff right now, right? So if you're tracking, there's your accountability.  Versus the person who's like, oh, well, I had zero calorie drinks today so I can go have this dessert, I should be okay. Or I can go ahead and have two, 300 calories here. They're guessing, they're going based on how I feel. Sweetness or a change in flavor, it overcomes palate fatigue and allows you to eat more. So in fact, I've talked about this before, what's that show called, Man vs. Food? Yeah, the ice cream thing. Yeah, it was a contest where I'd eat a kitchen sink full of ice cream and he got like, I don't know how far, 70% of the way there. Potato chips in between. Yeah, he started- French fries. French fries, he started gagging, he couldn't eat anymore. So how did he get himself to eat more ice cream? He ate a big plate- Yeah, he ordered like 3,000 calories of French fries. Yeah, super salty French fries.  And what that did is it made his palate fatigue, it overcame it, was able to eat the ice cream. We all know this, right?   Then they bring out something with a completely different flavor. Pumpkin pie. Yeah, I can eat more all of a sudden.  Artificial sweeteners are far sweeter than sugar.  This is what happens, it actually, it hijacks the body. What is that, Doug, is it 10X? What is it? Thousands. Is it? No. It's incredibly sweet and comparison. Yeah, how much sweeter is artificial sweeteners than regular sugar? I just go aspartame to sugar, you could just look that up. I know it's high. Very high, and they put a tiny amount in there to try to make up for it, but it's still far sweeter.  Like so if you ever messed around with, I don't know if you guys ever did sugar in your coffee or not, but if you do like regular sugar. You gotta add way more. Oh yeah, then versus like sweet and low, it only takes like one or two packets and a sweetener. And even then, it's too much. Right, it's sweetened your coffee up like crazy. 200 times sweeter. Wow. It's sucrose. Yeah. Wow, that's a lot. There you go.