Suppose we have a bipartite graph x, y, and want to match every vertex in x with a unique vertex in y. For our matching to exist, the obvious necessary condition is for the neighborhood of any subset to be bigger than the subset itself. But is it sufficient? Since we can count the number of vertices and edges in a graph, induction proofs are very common in graph theory. So let's consider. Now, remember, you can assume anything you want as long as you make it explicit. So let's keep track of them. We'll assume we have a bipartite graph with partite sets x and y. Let's try to prove the obvious necessary condition also suffices that if every subset has a neighborhood that is at least as big as it is, then we can find a matching. So clearly the theorem is true if x has cardinality 1. So for our induction step, let's assume the theorem is true for all graphs where x has cardinality less than or equal to k. In other words, we're assuming that if the cardinality of x is less than or equal to k, then if for all subsets of x the neighborhood is larger than the subset, a matching exists that saturates x. So now consider a graph G where x has k plus 1 vertices, where for all subsets of x the neighborhood is larger than the subset itself. We'll write that down as our next assumption. So let's take any vertex in our set x. Our assumption implies that the neighborhood of this lone vertex has cardinality greater than or equal to 1. So it has at least one neighbor in y. So let's put the edge x, y in our matching and consider a new graph G' which is G minus both vertices x and y, where x' is x minus x and y' is y minus y. And since x' has k vertices, our result follows provided every subset has its neighborhood larger than the subset itself. The important thing to keep in mind is that while this was true for subsets of G, we've removed some vertices so it might not be true for subsets of G'. And a useful idea in proof, if you can get to your destination despite any roadblocks, you can get to your destination. So clearly if this road is blocked and we can still get there, we can prove our result. So suppose there is some subset T where the neighborhood is smaller than T. We'll note that's a new assumption. Since the neighbors for any subset of x' must be in y' and the only difference between y and y' is the removal of a single vertex, that means the neighborhood of T in G' must be at least as large as the neighborhood of G and T minus 1. That is minus that single vertex. We assume the neighborhood of G' of T was smaller than T and so this means... but again, since T is a subset of x and we know the neighborhood of any subset of x is at least as large as the set itself, this tells us that... so the neighborhood of T in G must be the same size as T itself. Now, since T itself is a subset of x, then any subset T' of T is also a subset of x and so we know the neighborhood of T' has to be larger than T' itself but the neighborhood of T' has to be a subset of the neighborhood of T but notice that this means that T and its neighborhood meet the requirements of our induction hypothesis. In particular, there's fewer than K vertices in T and every subset of T has a neighborhood that is at least as big so by our induction assumption, there is a matching from T to the neighborhood of T. Let's draw a picture to help organize our thoughts. This means that we can partition x into two sets, T and the rest of it, where the neighborhood of T has the same cartonality as T. Meanwhile, y can be partitioned into two sets. The neighborhood of T and the rest of it. And remember, there is a matching between T and its neighborhood. Now, consider a subset of x minus T. Since r and T are disjoint, the cartonality of the union is just the sum of the individual cartonalities. Since the union is still a subset of x, our induction assumption guarantees that the neighborhood of the union has a cartonality greater than the union itself, which is the sum of the cartonalities. But since all neighbors of T are in the neighborhood of T, which has cartonality T elements, then there must be at least cartonality r neighbors of r in y minus the neighborhood of T. If that's not clear, then consider this. We have to fit r plus T neighbors someplace at most T of them can go here, so at least r of them have to go here. And so the neighborhood of r has to be at least as large as r itself. Now, notice that this means that x minus T and y minus the neighborhood of T also meet the requirements of our induction hypothesis. x minus T has fewer than k vertices, and for every subset, the neighborhood is larger than the set itself, and so there is a matching between our two sets. And so we can form a matching by putting together the matching that exists between T and the neighborhood of T and the matching that exists between x minus T and y minus the neighborhood of T, which completes the proof of our theorem. So putting everything together, we've obviously proved that if g is a bipartite graph with partite sets x and y, where a matching saturates x, then for every subset of x, the neighborhood is at least as large. Informally, every subset of x has a target set that is at least as big. We just proved the converse that if we have a bipartite set where the neighborhood of any subset is at least as big, then the matching exists. Together, these form Hall's theorem, which could state as an if and only if theorem, but we won't. Remember, if and only if theorems make for good poetry but bad mathematics, always restate them as two conditionals. We note that the obvious necessary condition for a matching to exist is that the neighborhood be at least as big, and as we proved it is also sufficient. This happens frequently enough in graph theory and other areas of mathematics that such results are known as Tonka's theorems. The obvious necessary condition also suffices. The phrase seems to have originated with the British mathematician Crispin Nash-Williams.


We just made this privacy wall or backdrop for a craft fair or a farmer's market someplace to hang your signs. We used some pickets and some 2x3's and we'll show you how we did it right now. What is up? A welcome back. Do you like to do a builder to make it? So do we and we have a new video each week. This week it's the final countdown. Yes I think I finally got Garrett there. He's he's clicked into gear. I mean pretty much pretty much clicked into gear. Five weeks, 18 projects, 75 items and a booth set up to boot. It's gonna be tight. It's gonna be tight. It is the whole month of getting ready for the market. Yes April is market month. Market month. That's what it's been. That's it. I have one month. I've been trying since February to really get this show started and Garrett had to redo the garage but now we're here. Garage is ready. We've got a workspace and we're ready to get going. I had to feel like spring. It was hard getting ready for a spring-summer farmer's market when it was cold out. I just didn't feel it. I didn't feel it. So this week we're starting with three panels that we're gonna use as part of our farmer's market background or our tent display. What is it called? Booth display? Yes part of our display set up for our booth. We are gonna make these panels is what I'm envisioning. I haven't made them yet but I know what I'm looking for. I want a place to display our door rounds. Last year they pretty much just hung from the side of the tent which was fine. I mean you can really see them coming but I think we could make the aesthetic a little bit better. Kind of like wooden wind chimes too.  They kind of did sway in the wind and the breeze there. But this this year I want to create some display panels. So these kind of look a little more professional. Yes. It doesn't look so hanging from a tent we got at Dix. Step one we're gonna gather all of our supplies. We really just needed like four of those six foot dog ear fence pickets. Two two by threes. We want two by threes cost and weight. Some three inch screws and some one and five eighth inch screws and some glue. Stand and paint. Yes. So let me give you a little low down on these fence pickets. You wouldn't believe how long we sat there. Ninety minutes. Debated. Probably not that long but we came in to lumber saw these pickets decided I'm not sure these are the ones we want. We went all the way out to the garden center. We debated and looked at all the pickets. We came back into the lumber pickets. We didn't like those still. We went over to another section within lumber and saw some other pickets. It was a whole process. So let me tell you what we chosen. Scanning matching skews. We did. Scuse. Looking at other stores. That might have been me but some of them didn't seem thicker than the others. So it was a thing. But anyway let me tell you what the deal is. Normally we get the premium dog ear fence pickets when we're making our front porch welcome signs because they're nice and smooth. I don't really do any sanding. They are pressure treated. I can just stain and paint them. These are not those. So those are 350 each right now. They were like 270 but they've really gone up in price. So but still affordable to do a front porch welcome sign. But when you're doing what we're doing for these display panels they don't need to be that high end. They don't need to be super smooth. I don't want to take away from the sign themselves. I don't want the display to be so beautiful. It's not. They don't notice the sign. So we're going to use these just standard dog ear fence pickets. These right now are $1.98 so they were like $2 each. But I am going to stain and paint these. So I think I might hit them with the sander just to knock some of this roughness off because I think they're gonna be difficult to paint and stain. It's just gonna catch the rag when I try and stain it. Oh that's true. So anyway because we're gonna cut these up and they're just gonna be displays. I think that these little the rough version here will work. So that's the difference between two dollars and 350 and which ones we use for which. Well do you want to run a sander down them before we start? Yeah probably we should do that. What do you want to do after we chop them all up? I think it'll be easier to do them as a whole board. I agree. Step two we're gonna make all of our cuts. We're really just gonna cut these planks down to 24 inches and these two by threes we're just cutting down to six foot and then we should have about two feet left over. Maybe we have to use it as the frame or a foot. Not sure yet. But what I am sure about is that we're going to be using the pocket saw to make all of our cuts. Every bit of sawdust in the bag just came pouring out the front. That way it makes it look like we've been working. Now we're gonna stain. We're using some briar smoke. We're gonna lay down a base stain. Then we're gonna come back in and paint. Yeah we're gonna give it that distressed painting technique. Yeah right now we're just giving that that underlaying color something that's not raw wood. Yes and I'm trying to go with this briar smoke because I think the rounds are gonna be more of a darker color stain like what is it dark walnut and this will be a lighter color stain to give it some contrast. I'm gonna paint. You know like that is not a paint brush. Oh I know we're gonna scrape the paint across. Just give it like a kiss of the paint with a scraper. Yeah we tried this technique a couple of projects ago and I really like the way it looked and I think it's perfect for this back drop behind us because it's really a rustic look. It's a paint saver. So you're really just gonna pour a little bit on your board and then run your scraper tool down it. Here's your technique. Just making like a ladder on both sides. Some rungs will be missing and we're gonna pin it all together with some one and five eighths inch screws. I think we're gonna put some pilot holes in there though to make sure we don't split the boards and then we're just doing two screws per side per board. That's a lot of holes a lot of screws.  Front back front back and because we want to make it look distressed sometimes one board will be missing like it belonged there it just was taken down and used somewhere else and some other project. Except we decided on no missing boards. New news. Yes so I want to have greenery peeking out so it looked like there's little planters halfway down so we were gonna take a board out so I could have greenery there but since I want it to be displayed on both sides what I think I'm going to do is one on the back. Are you with me? Front one on the back one on the front and then the one that's in the back I'm just going to staple some greenery too.  Yes so it will just alternate. All right I get it now. Really just adding some flowers some greenery something to give it a little pizzazz something to brighten up the dark empty spaces on either side. Well they're not dark and they don't really need anything but I thought this would be a nice it just gives it some interest and it'll be okay if we cover it up with a door round for display but if it's not covered up then it's got some interest and you know flair. A little something. I did do a little research on some greenery. I went to Hobby Lobby yesterday to get what I knew I wanted. Well I went first. Yeah I wasn't gonna go into all that. Yes I sent Garrett first during the day while I was working and he brings back a bag full of greenery. It was seven variations of Ivy because he felt like that was the most affordable thing that he could find and and it was and maybe it would have worked but it wasn't exactly what I wanted so I was I went marching back up there after dinner and I was like nope that's not what I want we're gonna pick that I got I already know what I want but they were eight dollars each and I was gonna need at least four or five per rung and I wanted to do three per board and I was looking at like a hundred and fifty dollars to add the screenery so that was not gonna fly so I returned everything that Garrett purchased and then I didn't purchase anything and then I said we're going to the dollar store so I had one of the youngest with me and I said we're going to the dollar store we're gonna go see what they have so they had these hydrangeas not the most the best quality silk flowers out there but I think that we can add a few of them here and they'll give us the interest and you know we lose flowers over time we can easily well flowers happen we're just gonna staple them in like in a couple of the gaps all right what's it look like yeah yeah yeah you can make a defeat we're gonna give us some feet so that we can move a bolt right we got to keep this thing portable all right it's got to be able to fit in and out of a car so we're just going to use we had another picket that we finished staying the same way and so we've cut that into thirds just like we did the rest of them there's still 24 inches they're gonna be the feet we're gonna take this one and chop it up so it's five and a half inches wide we're gonna get four of these out of this one I hope we'll see and then we're gonna use part of the two by three that's the back brace we're gonna cut this to use it as a 45 degree brace so again it's all part of what we've already stained and this death for the rest of the project just using our leftovers making the most oh look who's making a second appearance tonight it's like a cameo the night that's gonna cost us brace it's seven from this piece here pin it together with Brad okay so we're just marking the center of the board and then one picket plus the two by three is about a two two and a 16 two and an eight two and an eight so we're gonna mark the center and then mark an inch on each side a half inch on each side right maybe one inch on each side one and a 16th yeah so you only want to leave and leave enough space to put the panel right down in there between the feet it'll hold them and they'll stand up line this up back here and I'll pin it yes it's temporary held and now he's gonna screw it in the picket I mean are you gonna screw the picket down so that it doesn't oh I guess it's screwed to this piece here okay so we just screwed the bottom to the brace so we take them outside and see if it works let's do it I'll take this big picket some two by threes ourselves a privacy wall you couldn't even see me or a backdrop for the farmers market right isn't this great this is gonna be the perfect display and we're using these little shower hooks so that we can move them around we could adjust them they don't have to be in one place I can just take them off and hang them anywhere and then when we're all out of signs the flowers make it look like it's not so blank supposed to be that way look into it good big shout out to the patrons we love you guys think we're bad out of time so if you're not joining us for the patron after show we will see you next week where we'll do it build it make it again these things are way too heavy to balance yeah we'll balance some hello peeps with the wind


Live from Las Vegas, it's theCUBE, covering AWS re-invent 2017, presented by AWS, Intel, and our ecosystem of partners. Hey, welcome back everyone. Live here in Las Vegas, theCUBE is covering exclusively the AWS re-invent. Got two sets, this is set one, set two behind me. We're here in a startup called ZetaBytes, Rishi Yadav, CUBE alumni CEO, and suit here, Jen Gerr, CTO, hot new startup, ZetaBytes. Normally you're entrepreneurial, your other company's still going, InfoObjects. Welcome back. Thanks for having us here. I don't know, it's the seventh time, it's time. I mean, we love CUBE guys. Yeah, so InfoObjects is the mother ship and doing really, really great. And today we are launching ZetaBytes, which is our hybrid cloud integration platform. We are starting with AWS and then it's going to have integration for other clouds. So startups are impacted, and then we were talking yesterday about kind of a demarcation line between point in time. I say 2012, maybe you can say 2014. If you were born before 2012 or 2014, you probably didn't factor the cloud as large scale as it is. But after that date, your newborn startup, you get the cloud as a resource and opportunity.  So actually 2014 was an inflection point for two things. Number one is that the big data, big data, it started with the hyperscale companies. And at that time, you're talking about Facebook and Yahoo and other places, but it was not enterprise ready. And we suddenly saw the adoption, John, you have been following the big data journey from I think the cloud era basement days, right? So in 2014, it got enterprise adoption and the things like security and governance which were of not much concern earlier, they became a front and center. Another thing which happened was that around 2014, 2015 timeframe, the public cloud, which for eight, nine years, essentially AWS, that was about SMBs, startups, about saving money for them, right? That also started getting adoption in the enterprise. And when you're talking about enterprise, there you cannot tell them that, you know, if you deploy 10 servers on AWS, it's going to save you $200,000. They would say we already have $500 million spent. We have these huge data centers. So they needed some more value than that. But... Talk about your company, Zeta Bytes. So you're launching a new company. What is it? What does it do? Why are you starting it? Take a minute to explain what you're doing. Yes, absolutely. So the Zeta Bytes idea came from this convergence of the big data, public cloud and IoT. And market is ripe for it. And the challenge was that, we talked to a lot of customers. A lot of them have already started working in the cloud and some of them were planning to start the journey in the cloud. And the challenge was that, at the same time, they also wanted to build the big data lake. And he talked about it a lot today, right? S3 being the largest big data lake. So now the question was that, do you really want to go the old school route in which you are using Hadoop and other services around it? And then you do lift and shift to AWS and then you transform to PaaS. So you spend one and a half, two years in doing Hadoop. Then you spend another one and a half, two years doing the PaaS, the cloud-native transformation, or is there a better way? And then we realized that, whether the clients are on AWS today or they are going to be in one year, they need the same experience, the same cloud experience, the same AWS experience which they have on the AWS, they want on-prem. Now that includes the cloud-native APIs, but also the agility and everything else. Okay, so let me ask, should hear a question. So you're the CTO. I know you're technical too, so I'll have both of you. So the old days, I'm a developer. I have my local host, I'm banging away code, and then I go, okay, I'm done. And I say, ship to the server for QA and whatever. And even the cloud. Businesses want that same kind of functionality on-premise. They want to go to the cloud, so all the developers are changing. They want that local host-like feel. They don't want to have to write code, ship it to a server, put it to the cloud. They just want instant integration to Amazon. Is that what you're doing? Is that something? Get it right, because that seems what I think you're doing. Yeah, so Zeta provides that seamless experience. So you have the same set of APIs which you normally would do on AWS. So you still use the same AWS SDK. You still use the same as the AWS CLI. Use all the AWS APIs. We have upset out those APIs on this platform. Build your code base based on those APIs. Now, using Kubernetes, you decide where this workload will go. Okay, so one of the challenges with AWS though is that they release services like constantly. I think we had the announcement at the keynote today. It's like another hundred or so services that they're releasing. So how do you choose which ones do you support? All of them or do you focus on specific ones? First, we are focusing on a few specific ones which are mostly being used. We are starting with, for example, S3, Lambda, Kinesis, Kafka, and Spark and SDFS are there from day one also. Right, okay. Then all of these are what we are doing. Today, if you see the announcement, they have launched Kubernetes now, the Container Management Service. We have that flexibility from day one only. So we have that in our appliance. And using that, even, for example, your workload says some of the pieces should run on the, let's say, on-prem appliance, some of the pieces should go to the cloud. That is also possible. Okay, so you're selling an appliance. Yeah, yeah, one Minion, let's say Kubernetes Minion might run on the AWS. Few of the Minions might run on your appliance. You can easily access, do all the control management. It's a business model. They pay for the box or is it a service? Oh, they get the box as part of a service. What's the business model? So we do both. So it's a software platform as well as an appliance. So the beauty of appliance is that everything is already optimized for you, right? So that makes it very easy. But if a customer has a chosen hardware platform, we can definitely deploy it on that also. And adding to the 100 services thing, and I think that's a great point, that AWS has so many services now that can you really go and figure out which service is most optimized for your needs. That's where you need a partner on-prem side, right, and that's what we are going to be. And another thing, as Sudhir mentioned, the EKS, which they announced today, the Kubernetes. So you have Kubernetes on-prem, AWS is supporting Kubernetes, and we are also supporting Kubernetes. So if you want to work your workloads at that level, it's completely seamless. And you were saying before that your target is enterprises, so the appliance delivery model and the simplicity of being able to manage a lot of different services, I mean, clearly, being able to manage things at scale is something that enterprises are crying out for because otherwise I have to, AWS is great if you want to hand-build everything yourself. It has all of these components that you can assemble like Lego. But if I'm an enterprise, I want to be able to do that at scale. Humans don't scale very well, so I need some technology to help with that. So it sounds like you're actually providing the leverage to get enterprise humans to be able to manage AWS. Is that a fake characterization? Absolutely. That is definitely a very important aspect of it. And another aspect of it is that if you do not want to have some workloads on AWS for one region or another, IoT workloads by definition cannot be on AWS. Low latency workloads, that they cannot be on AWS. In the same way, the workloads in which you need some extra level of security. So within your data center, as much as we beat down the data center piece, you have your own security and governance, and you can do that. And that's coming back to your question that are we going to support all 100 services? Yes, but the local execution, we are only going to provide for some services, which by their very nature, make more sense to run on-prem. Yeah, the core services. Core services. All right, so how do you guys going to sell this product? Take us through the startup situation. You're here, are you talking to customers? Why are they by you? What's the conversations like? When do they need you? Take us through your conversations here at re-invent. Yeah, so before that, the AWS has been super successful for the green field applications. The new applications, the applications which are born in the cloud. But when it comes to transforming the existing application, it becomes a big, big challenge. So a lot of customers who are coming to us, they are interested in how I can seamlessly transform this. What's an example workload? An example workload. The example workload for us is going to be the big data workloads, which we have specialized in for last so many years, right? So one of them can be IoT, so you probably can explain more about that. Yeah, so the other example could be, for example, from today's keynote, if you see Expedia case, or like Goldman Sachs case, they spent a lot of time in converting their code to the AWS specific code, right? Millions of lines or billions of lines of code. What we are doing today, if you're developing your application, tomorrow it could be future ready for AWS. Okay. That's your convenience. We are actually merging your experience with AWS. Right. So it's making it easier for enterprise to make that transition from what they're doing today across to cloud. That's a big deal for them.  So your market is hybrid cloud basically? Yeah. Anyone doing hybrid cloud should talk to you guys. Yeah, and code would be future proof what you're developing today. All right, so is the product shipping? Okay. Yes, so we are in the early beta stage. We already have five beta customers and the product is going to be ready in a week's time. Yeah. So beta now? Yeah, yes. You guys are already ready? Early access. Open beta or restricted beta? It is going to be restricted beta for now. Then it's going to be open beta. So yes, we are going to add five more customers in the next two months for the beta. So. I'll take a minute to explain the type of customer you're looking for. Are they all filled spots anymore? You have five more spots, you said? Yeah, we have five more spots for the beta. Who are you looking for? I didn't know. Any large enterprise which is planning to move to AWS but start struggling with all the integrities, looking at the hundred services and how do you integrate your existing applications there? So how do you take those baby steps? So we are going to not just take the baby steps but sprint through it. So that's what our Zeta Bytes plans is for. Rishi, congratulations on the new startup. Launching here, Zeta Bytes open beta. Five more spots left. Check them out. If you're doing hybrid cloud or true private cloud, they have five more spots available. It's theCUBE bringing all the action, startup action here and also the conversations that reinvent. I'm John Furrier, Justin Warren. We'll be back with more after this short break.


Everyone, prepare yourselves for battle! Welcome to Ultimate Epic Battle Simulator, two. Not one, but two. Ultimate Epic Battle Simulator was a game that came out probably like 2017, 2018, something like that.  Let's do this. Let's just do Roman times. Let's do Roman soldier. Let's do a hundred, that's the base one? We're gonna start small. We're gonna start with a hundred, okay? We're gonna start a hundred of those. We're gonna do a hero, action man? Chuck Norris, whoa. Let's do Asian grease. Let's do a Spartan. Let's do a hundred of those. Let's just do just a simple one. A very, very, very simple one to start. We'll see who wins, see who lives, see who dies. We don't need to go insane with this. Right off the bat, a hundred thousand? You're gonna start me off with a hundred thousand that's the base? Just let me chill for a second. Okay, press K to start the battle, all right. Let's see where everybody's at here. They're all right here. Okay, they're just gonna go. P into slow motion. Okay, all right.  This actually seems pretty, pretty even again. Don't get all, oh, either this is boring. Just wait, we're just doing one normal battle to start off. Okay, all right. Oh, team two. Okay, the Spartans are actually doing like way better. Oh, there you go. The Spartans won, see? Will you calm down?   What? I can drown all of them?  Now, we're gonna pick it up a little bit, okay? Now we're gonna pick it up a little bit, all right? What's custom? A full auto Sherman? Let's do a thousand of those. Let's do another team. Let's do a full auto archer. Let's do a team three. Let's make this evil, five thousand of those. Great, let's change this to, oh, the city of Toronto. Let's do the desert dunes, that's kind of fun. Let's do another team. Let's do animals. Free range chicken, 50,000 of those. Now let's do another thing, one hero. It'll be Chuck Norris, just one of him. And that'll be good. What does the rank shape mean? We're gonna find out. Here we go. These must be the biggest tanks. Uh-oh. So we have these full auto tanks. We've got, oh, yep, yeah, that's bad. These are the full auto archers. Okay, cool. We've got the chickens, the 50,000 chickens. We've got Chuck Norris, okay. And we have 5,000 zombies, 10,000. Seems pretty good to me. All right, well, might as well go.   What just happened? Air. Ha ha ha, what happened? Let's get out of slow-mo. Oh my God. I didn't know. Wait, but Chuck Norris, Chuck Norris, he's still alive. He's out there somewhere. Where is he? Chuck, Chuck, where are you, Chuck? Chuck, please. Chuck, please. Is that you? Oh, that's not Chuck. Where is he? Where is he? We gotta find him. We gotta find him. Chuck, Chuck, where are you? It's impossible. It is impossible to find this man. He's gotta be just in the thick of it out there. Down here in the dunes. Somewhere is our boy, Chuck, fighting. Fighting for his life. He's here somewhere. There he is. There he is. We found him.  I can't, we can't even make him out clearly. He is but a God. I don't know if this will ever end, guys. I don't, there's so many of them. There's so many. I don't think it'll ever end. I didn't know what they were.  All right, we have to quit out. I think it's gonna last forever. I think that that's the one battle that would never end. We'll do one of those. 1,000 of those, 5,000 of those, 50,000 of those. One of you, five of the rank shape. Medieval, catapult. Let's do 20 of those. We're gonna do a mantis shrimp. We're gonna do 50,000 of them. Here we go, here we go. I made a new thing. I made a thing that has, I think, 500,000 things in it, or maybe it was 100,000. I can't remember. We've got, okay, so our battle is one of these. Some archers, I think, 5,000. 50,000 chickens and Chuck Norris. We've got some catapults with, I think, 50,000 zombies. We have 12 trolls. This is a mantis shrimp. I think there's 100,000 of those. Or maybe there's 50,000 of those and 100,000 zombies. Yeah, yeah, yeah, I think there's 100,000 zombies. There they go. And immediately the catapults are gone. Oh no. Oh no. Oh, they lay eggs. They lay eggs, there's... Oh my God, my whole computer is destroying itself. Dude, one of these? Oh, it shoots out the back like that? I didn't know that. This thing is unreal. Oh my God, Chuck Norris is still out there somewhere. I think the zombies are doing not much actually. I don't know, can anything take out this thing?  What is, sorry? Sorry, what? That's what the full auto archer does? Oh, but the zombies might flank him. The zombies might be able to flank. And there's still, there's still so many zombies. But what about the mantis shrimp? Are the mantis shrimp on the same team? Oh no, the mantis shrimp and the zombies are on the same team, guys.  Or I'm like, guys, guys, the mantis shrimp and the zombies? Oh my God, they're gonna flank the full auto thing. Are all the trolls dead? Oh my God. Look at them flipping around. Look at them. Oh my God. Oh my God. I wanna see if anything can take this thing out. Cause I truly do not know if anything can take this thing out. It's so powerful. They just need to learn to go around the side. Too many of them are going just straight on. I'm going to try and be the outlier, okay? I've taken control of a zombie. Come on, everybody, this way. This way, this is the only way that you're gonna make it.  So follow me.  It's learning. It's following. Oh no! I don't, guys, I don't think, I don't think that this can win. There's only 3000 troops remaining. And Chuck Norris though. No, and the troops are dwindling. We need something with a little bit more firepower. We need to see what can take this thing out, okay? And the custom. We're gonna have the full auto Sherman, okay? Team one. Team two. We're gonna do World War II. 10,000 of those. We're gonna do another 10,000 of those, okay?  What are you going to do? Two and a half million. Wait, no, that's only 250,000, isn't it? Yeah, we're gonna do 250,000 battle ducks, okay? They have to come over the ridge though.  Okay, so these are the trolls. These are the trolls. These are the battle ducks. Look at these guys. Oh my God. They do have to come over the ridge a little bit. That's fine. Here's the one thing.  Bunch of shoulders on this side. Starting it off in slow motion, okay? Go. Oh, that was too easy. Oh, that was just so easy. Oh, that was just so easy. That wasn't even fun.     Look at these fuckers. They're all facing the wrong, oh, a lot of them are like snapping their necks.  Wait, why did it only load in 445,000? We're missing 50,000 ducks versus all these trolls. All right, trolls versus ducks. Can these guys fly? I don't know if they can. God, look at all of them. Are these guys gonna come? How are they gonna do this? Don't fall off the ledge.  I think they're all gonna be going down this thing. Oh, my God. Here they come. Here they come. This is where we see who wins and who dies. Here we go, ducks versus trolls. The ducks are just being obliterated. Literally not a single, not a single troll. I thought these were battle ducks. They're just not strong enough, guys. They haven't even gotten a single one.  Not a single troll. Okay, hold on, hold on. We're gonna try something. What about instead of 10,000? We do 10. Trolls. You're there. Go for it. Go for it, troll boys. Can half a million ducks defeat?  Some of the ducks are already dying. They're just dying of what? Thirst? Wait, total kills. 445, but they haven't done anything yet. What do you mean total kills? Oh, God, I don't think this is gonna be good, guys. I do not. But it's half a million of them. Oh, but they take them out so quick. Come on, duckies. They just gotta get enough so they're just completely surrounding them, right? Cause I feel like there's too many ducks to where they wouldn't just completely surround them, right? Like this guy, like this guy over here. Some of these guys are getting a little bloody. Is it their own blood? I don't think so. Here they go. They're starting to do it. They're starting to fully engulf them. They take out so many in one fell swoop. They take out like 10 at a time. That's possible, guys, unless we put them in the center of the battle. We put them in dead center, surrounded by half a million ducks. Can't even see them there's so many. Troll boys, I wish you luck. Godspeed, battle ducks. Honestly, I hope you win. This is where we find out if power in numbers is true. 10 versus half a million, and all half a million spawned in. Let's see what happens. What are you doing, duck? You're not even attacking. You're just eating grass. My disappointment is immeasurable and my day is ruined. I once believed in a world where half a million ducks could defeat 10 trolls. But after seeing the lack of effort and the lack of courage from these ducks, I just don't think it's possible. I think in time these 10 trolls would completely obliterate every single one of these ducks. That is so sad to see. That is so sad to see. Will the chicken do any better? Cause they didn't have those eggs that they were throwing around. Let's find out. We're gonna enter it in slow-mo. Oh, they all turn. Oh no. Unbelievable. Oh, it's working. Oh my God, these trolls are allergic to eggs. They won, oh, in a matter of seconds too. Two, they won. Oh my God, that was incredible. Or 10,000 trolls versus 500,000 chickens. Who will win? I wanna figure out where the trolls can prevail because now it's in the chicken's favor.  Oh my God. Oh my God. And they're taking them out. Look at the numbers go down. Just an onslaught of egg. And the chickens are getting less kills, but the sheer volume of them alone, I think will be enough. And look at the ranged shots on these guys. Oh yeah, there's no way that the chickens don't win. Just look at how many of them there are.  Look at this at work. You chicken. Reload. Look at all the egg. And just into a puff of smoke they go. Oh, they're trying to flank them too over here. Oh my God. You guys see this? They're trying to flank them. This is like, this is like Chronicles of Narnia. You guys know that scene from that movie where they're all on the thing? This is like that. This is the exact landscape. All right, guys. Well, I feel like we don't need to stick around and find out what happens. We know what's going to happen. There's far too many chickens. There's far too many chickens. But we have one more question still to answer. Can one full auto Sherman defeat half a million chicken? We're about to find out. Four, there even was a chance. These guys were cooked. They were cooked. They can't even get their eggs out. All right, if the chickens are able to surround it, why did it load in all funky? It's fine, whatever. Okay, so there's only one way. Under the right circumstances, guys, it can happen. All right, we're gonna put it right in nose to nose. All right? Oh man, this is truly nose to nose. Unbelievable. The sheer power of the chicken. But it's fighting back pretty damn strong. The chickens don't have the range that this thing does. The chickens might not be able to get close enough to fire the eggs. Nose to nose, this thing might be able to win. It's already killed. Oh my God, almost all of them. I don't think the chickens have a chance. I don't think the chickens can get close enough. Can't do it, guys. That is so sad. That is so sad. The chickens did not stand a chance. They just did not stand a chance. Obliterated, absolutely obliterated. And they still have so much hope. It's almost sad. It's almost sad seeing how much hope they have. Look at how determined this guy is. Could one chicken be the deciding factor? Could it be because this dude did take quite a bit of damage? You know, quite a bit of damage. And there's still a lot of chickens over here. And they're making their way. You know, there's still quite a bit. It could happen. You know, I believe in a world where these chickens could fight on. Look at him. Oh, God, I spoke too soon. Are there still any over here? There's that one. This one chicken. I think it's done, guys. I think, I don't think that this won't be able to, oh my gosh. The Full Auto Sherman has won the battle against the chickens. And it was one that many thought was lost for the Full Auto Sherman. And I honestly thought that maybe in the end there, the chickens could come back. You know, maybe someday they will. Maybe someday they will. But until then, remember why we do this. You know, remember why we fight every day. It's so that maybe someday one young chicken can learn what it truly takes to get to the other side.


Well, welcome everyone to this session on living the XKCD 97 dream with open telemetry by Greg Matthews. Thank you so much for joining us Greg and yeah, without any further ado, over to you Greg. All right, great. Thank you. Yeah, so welcome to the talk about open telemetry here this morning, my time. And today I want to get you excited about embarking on a journey to start using open telemetry in your organization. It's a really it's a really big deal that as a technology community, open telemetry has managed to unite such a broad section of software observability space under a common banner. And it's really great for everyone involved library authors, practitioners and even vendors. But what's even so exciting. So maybe some people out there don't even know what open telemetry is. Maybe others have already tried it, but they want to learn more. Don't worry, we're going to embark on this journey together from the beginning and take some scenic stops along the way until we reach the end. My wife and I love to travel with our three kids. We believe that if we can get them out there to see the world and they'll have a much more global awareness and more empathy as they grow up. And I've included some pictures from our trips that we've taken to visit the western part of the United States. And perhaps I can inspire you about open telemetry but maybe I can at least inspire you about coming to visit if you get the opportunity. So before we can really talk about open telemetry though we need to talk for a minute about the problem that we're trying to solve. As we build increasingly complex software systems, we have an increasing need to know what's going on inside in live production systems. Observability is defined as the measure of how well the system's internal state can be inferred by knowledge of its external outputs. It's a concept that has its roots in control theory, which is a fascinating topic that was originally applied to physical systems and sensors, but has more recently found a lot of application in computer systems. I work at a company called Stored where we build software to power what we call the cloud supply chain. As you might expect from a modern cloud native startup, we have microservices and Kubernetes and Istio and all the things. It's as complex as the marketing graphics on our website make it seem. In a complex system, we need sensors to tell us how things are working so that we can have insight into what the problem is when something goes wrong. So maybe you have some basic monitor setup like server CPU and memory consumption, or maybe you don't even have that and you're just rebooting things when it starts to slow down. So what kinds of superpowers can better observability give you? The most obvious superpower is the visibility into exactly how your complex distributed systems work in as much detail as you or your library authors care to instrument it. For example, this is a trace being visualized in Datadog APM from a production system that I used to work on. So using some basic instrumentation, we can see that there's three services involved represented by the green, blue, and orange colors. The green service is performing two tasks in parallel modeled by the purple color. One of the parallel tasks is making a call to the blue service followed by a call to the orange service. The other task makes a call to the blue service, which seems to have aired and then it retries successfully. We assume that based on the structure that after both parallel tasks are completed, then the overall result was returned by the green service. If I hadn't redacted the details, you could you could have inferred even more about what operations it was performing. You can also easily see how long each operation took in relation to the others as well as the leverage we got by using concurrency compared to just a status code or some server logs. This is pretty amazing. The next superpower I want to talk about is closely closely related. When something crashes in your application, maybe you have some kind of exception management tool that will let you know about it. Most likely you get a stack trace to show where it happened in the code. With this superpower, you can now see the exception handling within the context of the overall distributed system. You can see that there was an error, even though the actual response from the endpoint was successful, and it was apparently mitigated by a retry from the calling service. That's all stuff that would be hard to see in just a stack trace from one service. If we click into the error details, we can see the error message in the stack trace. But we can also easily click into any of the other related details to see how we got to that point. For example, we could check the endpoint being called or the parameters being passed. One of my favorite things to do when we first set up observability tooling on an existing system is just to browse the traces and look for a pretty obvious low hanging fruit. I'm sure none of you would ever do this, but people are still out there writing code that results in n plus one queries to the database. Can you see it?  If we zoom in, we can see that the blue bars at the bottom of the graph are actually tiny individual queries. I can try this example on purpose, but it's definitely something I've seen in the wild as well. Y'all, it's 2022, and there's something even better than that. In our cloud native world of microservices that nobody actually understands, it's not even that uncommon. It's n plus one API calls. Just because you've built a bulk fetch API in your microservice, it does not prevent people from calling that fetch one thing in a time in a loop. Another thing that can quickly jump out from a color coded visualization is multiple calls to the same service, which can indicate the potential for combining the calls. For example, here we have several requests in parallel to the same downstream service. Unless this is done intentionally to chunk a large request into smaller ones, it may be more efficient to combine these into a single call. That could potentially decrease load on the downstream service as well as decreasing the overall latency of the top level requests. Now flipping that idea around, it's also worth considering whether we could have better latency by not combining calls to the same service. For example, imagine you need to fetch a large data structure from one service and then use some embedded IDs to fetch more data from a different service. It might be faster to just query the IDs initially and then make two concurrent calls to fetch all the details from each service. This one is kind of obvious when looking at a trace, but much more difficult to see when you're looking at logs or metrics. In this trace, you can see that there are gaps in time between when the client made the call and the downstream service processing the call. And if the green section represents the HTTP call being made on the client side, and the blue and orange sections are the downstream services, then probably these gaps can most likely be attributed to network delays. It could also include time spent in untraced sections of your web server or proxy infrastructure. So also keep that in mind. And the final superpower, when I mentioned, is the culture shift that becomes possible when you have this kind of tooling deployed. Rather than troubleshooting based on past experience or gut feelings, our engineering teams can have a common baseline to start from, based on the traces of what really happened. That ends up resulting in much more precise communication about what the problem is by linking to an example trace that can be explored. It's a very powerful way to communicate a lot of context around a problem without having to spend time explaining it. Now that you've seen some of what's possible once you have observability, let's get more into the details of what open telemetry is and how you can use it. Yes, this is a picture of 30 Raspberry Pi camera starter kits. Isn't that typically what you take when you travel? Let's dive more into what open telemetry is and also a brief history of how we've arrived at today. So back in the ancient before times, there was no law in the land.  It was a time of Nagios, SNMP, and incompatible vendor solutions. Many practitioners tried to make ends meet, building lots of different tools and scripts. Ultimately, open census and open tracing emerged as the top contenders in the space of open source observability tooling. Both had achieved some amount of adoption, but they were still not quite compatible, though very similar. In early 2019, it was announced that these two projects would be superseded by the open telemetry project, and work began in earnest. By early 2021, the open telemetry tracing specification was christened with the 1.0 version and declared stable, but what does it mean for the tracing specification to reach a stable status? While open telemetry is architected as a set of independent signals built on top of a platform that provides the mechanisms to propagate and collect the data from those signals. Signals work as cross-cutting concerns that need to be considered at all the software layers, from libraries and frameworks to the server applications built on top, and through to the clients of those services. For each signal, there's higher level concepts that need to be defined, like the semantic conventions that should be encoded in the data to support the standard tooling and developer-facing APIs for manipulating that data. There's also lower level implementation details in the SDK to actually collect, process, and store the data for later use. And finally, there's community contributions related to each signal to integrate various libraries and frameworks in a consistent way. Along with all that structure and long-term thinking come some great supportability promises. For APIs, the project promises to provide support for stable APIs for three years, so that the instrumentation you write today should continue to work without needing frequent maintenance to keep things current. Similarly, the plug-in authors and application owners contributing to the OTEL ecosystem are guaranteed one-year support on those APIs. In addition to the stable APIs for instrumentation, OTEL also specifies the Open Telemetry Protocol, or OTLP, which describes a common transport mechanism for all the data through the stack. It's an efficient binary protocol leveraging protocol buffers in either GRPC or HTTP. To top it all off, OTEL provides what they call semantic conventions, which enable consistent tooling for visualizing and analysis. For example, the semantic conventions for HTTP explain when and how to set the error status on a span specific to HTTP operations, so that different HTTP client and server libraries across different languages can all consistently describe the operations the same way. The semantic conventions also explain what names and formats to use for common attributes for various kinds of observability data. For example, how different parts of the overall URL should be represented, so that tooling can be developed that leverage the data for aggregation, searching, filtering, et cetera. So now that we've talked a bit about what Open Telemetry is, I'd like to dig into what each of the signals are a little bit more. Logs, metrics, and distributed traces have historically been referred to as the pillars of observability. Rather than pillars, I like to see these as three different views of the same system, like the faces on a cube. Each aspect touches the others, but also has orthogonal concerns. For example, distributed traces can tell us a lot about what's happening in a sampling of specific requests. But each trace also represents many operational data points that can have new meaning when aggregated across many requests and visualized as a time series or a histogram. Traces can also tell us about key operations that happened or where in the process errors have occurred. These facts are things you might want to collect in logs and allow searching and auditing later. Each aspect tells a different story about what the system is doing inside. Traces tell us a story about the details of particular requests across services that contain more detail than would be typical in logs. They can have attributes that can contain both low cardinality tags, like metrics have, and high cardinality structure data, like logs. To make that practical in production, often only a small fraction are sampled. Metrics also tell a story, but not about individual things happening. They talk about types of operations rather than particular instances. They describe the system in a particular time window and often have tags to break down the useful context around the metrics. Logs tell a different story about your code, usually in fine detail. Ideally, logs have a structured format with regular fields to make them easier to index and search. Usually, they're retained for a fixed duration based on simple rules like severity or environments where they were generated. I don't have much time to go into all the details about metrics and logs, but I feel like distributed tracing is less familiar to a lot of developers I've talked to. So I want to briefly cover the theory and concepts, so it will make sense when we actually go to use open telemetry. At the core of distributed tracing is the concept of a span. Spans have a name and a span of time during which that activity took place, which is different than logs, which are generally treated as point-in-time events. Spans can also have metadata attributes attached. There are some semantic inventions defined for common use cases, but in general, they could be anything. Spans can also have events attached, which are similar to a span in that they have a name and a set of attributes, but they represent something that happened at a point in time rather than over a span of time. Each span also has a status, which is typically either unset or set to error by instrumentation, according to the various semantic conventions. This is very useful in identifying and visualizing interesting spans as they're collected. Spans also know about their parent span, which allows a hierarchy to be created so that we can visualize the execution of a program using a waterfall graph like this one. Note that unlike CPU profiling flame graphs, this represents a single actual request rather than an aggregation of where the CPU time is being spent overall in a running system. Each span also knows about the overall trace that it belongs to. Note that there's a top level span in the trace that has no parent, and we call that the root span. We can also represent concurrency in these trace visualizations, which tends to be tricky with CPU profiling flame graphs. Here you can see that our Phoenix controller is making two parallel HTTP calls, for example, using tasks.async before proceeding with handling the responses, which includes making a database query, et cetera. Where distributed tracing really starts to shine is when we think about what happens within each of those HTTP calls. In this case, we can imagine that the first call is being made to a different Phoenix-based server, and the second one is being made to a Rails-based service. In the context of the overall trace, we can visualize which services were responsible for the part of the request by looking at the callers. A span context needs to be propagated between services or processes so that the child span knows which trace that they're a part of and what their parent span was. The span context also lets the downstream process know whether the trace is being sampled and can also propagate other state metadata. Until recently, this was a vendor-specific free for all of incompatible headers, but luckily, there's now a W3C standard, which is used by OpenTelemetry. If we consider the tracing process from the standpoint of the originating service, we pass this trace context to them in HTTP request headers, but we actually have no idea what they do with it and we don't wanna bloat their response payload by requiring a trace dump every time from the downstream service. So how do we get a complete trace? Well, each of those downstream services is able to locally set up a tracing context based on the parent traces ID and which span ID that trace was called from. Then each service can send its part of the overall trace as if it were the only thing that existed and the trace collector process can merge them all together. Since the top level span within each downstream service knows what its parent span ID was, even though it doesn't know all the details about the span. The trace collector is also responsible for sampling which traces to keep. A simple way to do this is via probabilistic head sampling. We choose some percentage of the traces to keep or drop. The sampling decision is propagated to downstream services so they know also whether to sample in the same way. The collector still has the option to store or drop completed traces depending on its configuration. It's also possible to perform so-called tail sampling on the completed traces, perhaps based on criteria like the overall latency of the process or error somewhere in the trace. So far we've mostly been talking about synchronous client server operations but open telemetry can also be used to model and visualize asynchronous operations as well. For example, a producer could send a message to multiple consumers or multiple producers could send messages to a single consumer that operates on a batch. These can be represented using the span kind which describes the relationship of the span to its parent or child or children. In addition to parent child relationships between spans, it's also possible to declare causal links between spans. This is particularly useful when you have spans that are part of a disconnected traces but you want to be able to track their relationships. For example, a bulk batch processing operation may happen much later than the process that caused it but it may be useful to retain a link to the original operations that were related to the batch processing. So now that we know a little bit about open telemetry and observability in general, you should consider the local law before proceeding. No, I'm not talking about GDPR though you may also want to think about that. I'm talking about Conway's law. Depending on how your organization is set up, maybe you're on a team who wants a certain section of the code base or certain microservices that you could focus on instrumenting with open telemetry and then expand from there as you're able to demonstrate value. Before setting out, it's also a great idea to take stock of what existing observability tooling you already have in place so that you can make sure you don't unintentionally break some tooling that someone else is depending on while you're experimenting with this new thing. And finally, consider what kind of organizational support you need. Do you need to get permission for yourself or your team to spend some time on this as an innovation project? Are you gonna need help from IT operations, security or SRE to get the data sent to an internal or an external tool for collection and visualization? Are there other engineers in the org who would also be excited to give this thing a try with you? And so equipped with all this great information in this presentation you saw on the internet, you're ready to actually give it a try for yourself. Like we were just talking about, every journey is better when you bring along a friend or a guide who's already been there with you. And so far, I've been trying to keep the presentation more generically about open telemetry and observability as concepts in the broader technology community, but I'm personally focused on the beam community and primarily on the Elixir language. So some of the upcoming content will be specific to that. But don't give up if you're joining from another language community. One of the great things about open telemetry specification is that it tries to keep all the concepts and terms consistent across all the language implementations. So most of the content will apply to you as well. We touched briefly on this earlier, but I wanna dig a little bit more into what it means that Ohtel provides both an API and an SDK as separate packages. So Ohtel describes the API as the interfaces for instrumentation. These are just the abstract things that you could call, but unless you also include an SDK, the implementation of each of those API endpoints is required to be a minimal stub that doesn't actually do anything. When you include an SDK, whether the official one maintained by the Ohtel project for your language or a third party vendor implementation, the NOOP API calls will actually begin to do what's required to collect and forward the data for that signal. The key thing to understand is that if you're writing a library or a framework, you should feel comfortable to depend directly on the Ohtel API package as your means of providing telemetry about your code. If the application developer using your library does not intend to use Ohtel, then they won't include the SDK and thus they won't incur any significant performance overhead since the API implementations will be NOOPs. The API and SDK implementations for each of the officially supported languages are maintained in the OpenTelemetry org on GitHub. There's several directories within the OpenTelemetry Erlang Git repository to maintain the source for each of the packages. From each of these directories in the GitHub org, a package is pushed to the HEX repository for developers to use. In the Beam ecosystem, the API is implemented in Erlang so that it's easy for any Beam language to use the same implementation. For Elixir specifically, there's also a thin wrapper around the API to make it feel more like Elixir. The Elixir shell is truly very thin, mostly just providing Elixir-style modules that delegate to the ones implemented in Erlang. But it also provides some conveniences that Elixir developers will appreciate, like macros to easily create a span around a block of code. That way, whether you're using Erlang or Elixir, you get first-class support for your language without the overhead of maintaining two complete implementations. Before we really dive into how to set up OpenTelemetry, we need to briefly talk about an unfortunately confusing naming collision. The BeamTelemetry project was started approximately in parallel with the OpenTelemetry project, but with much more focused goals. The Telemetry library essentially gives you a simple, standard, and safe means to instrument your library, framework, or application code in a way that others can easily consume. It's maintained by members of the Erlang Ecosystem Foundation Observability Working Group with input from the maintainers of many popular libraries as well as Elixir itself. The way it works is that your library code is able to call this execute function, describing what kind of event you want to emit, along with measurements and metadata about it. Someone else can then attach to the events that they're interested in, specifying handler function. Then when the event fires at runtime, the handler function is called and can do whatever it needs to do. Maybe log something, send metrics to a statsDServer or start or finish a tracing span. It's important to note that these handlers are called synchronously from the library code in order to give maximum flexibility for various use cases. Thus, it's important for telemetry handlers to be lightweight and defer any heavier processing somehow to another Beam process in order to not block the actual operation from proceeding. These events can do anything you want, but conventionally libraries will emit start, stop and execution exception, start, stop and exception events for their key internal operations so that people can create consistent instrumentation libraries that consume those events for things like open telemetry spans. We can also very easily consume metrics that have been prepared for us in advance by the library. So it's a relatively simple synchronous dispatch API. Under the hood, the way it works is that there's an S table that keeps track of the handlers that have been registered for each type of event. They're called synchronously one after another when the library calls the execute function. So if handler one works, but handler two crashes, it's automatically removed from the S table and it's not called when the execute function is called again in the future. So just like baseball, it's one strike and you're out. I think I got that right. So it's relatively safe. I say relatively here because you still do have to be careful to not do crazy things in your telemetry handlers. There's already quite a large number of libraries that emit telemetry in this way, including most of the popular ones that you're likely using today. So at this point, it's essentially the standard that most elixir and Erlang developers are going to expect from their libraries. Okay, but seriously, that's enough talking. Let's do some walking. How do we actually get some stuff instrumented in an app? The obvious place to go as you're getting started is the official getting started guide for Erlang and elixir on the hotel website. That will walk you through some of the basics that we're covering in this talk today. So it'll be a good refresher for later. But for now, let's focus on the goal of getting an example app set up to send trace information to the console so that we can understand how to get the basics working. The fastest way to see how that's done is probably to take a look at the example apps that we have in the hotel Erlang contrib repo. If we look at the Phoenix example, for example, we'll see that there are three hotel related dependencies that have been added. The Phoenix and Ecto ones will hook into the beam telemetry events that those libraries expose and the exporter will be used to actually collect and forward the hotel data initially to the console and later to OTLP vendors. In the application.exe, we need to tell each of those integrations to connect themselves to the telemetry at runtime. In particular, in the case of Ecto, we have to tell it what the telemetry prefix will be because unfortunately it varies the event name instead of just indicating the repo via metadata in the event. Finally, we need to configure the exporter. OTLP gives us a lot of flexibility in how you wanna process and export the data. So the required configuration is unfortunately somewhat complicated as a result. This is how you tell OTLP that you wanna have batches of data sent to the console. Then when we start up the app and hit localhost 4,000 in your browser, you should be able to see lots of OTLP data printed out on the console like this. The format of the data isn't really intended to be human readable, but we can definitely see that there are some recognizable things in here, like the span IDs, various attributes that we saw from before related to the cement conventions for HTTP. Okay, so that's awesome to confirm that we have the instrumentation setup, but in order to truly show some value, we need to send it to a tool that will help us visualize the data in a more consumable form. We could certainly set up an open source option like Zipkin or Yeager, but I also wanted to point out how easy it is these days to set up a free plan on some of the vendors to directly support OTLP. For example, if we wanna send to Honeycomb, we just need to tell it to use the Honeycomb OTLP endpoint and include the headers that will authenticate you and send the data to the right place within Honeycomb. And then when we navigate around in the app, we'll immediately start to see things showing up in Honeycomb, not bad for just a few minutes of effort. Clicking into an example trace, we can see the relatively simple spans that this demo app is generating for Ecto and Phoenix, along with some familiar attributes we saw earlier in the console. That's pretty awesome. Similar to the way we pointed the exporter at Honeycomb, we can also point it directly at LightStep. One issue here though, is that LightStep currently has a non-standard API endpoint, so it has to be configured in this little bit more awkward way. My guess is that at some point, they'll support the default URL structure and this will no longer be necessary. When we run the app with this configuration instead, we can see data coming into LightStep. And if we click into one of the traces, we can see that all the details are there that we'd expect. This is really cool to me that we didn't have to do anything at all in terms of the application instrumentation and we can easily switch vendors just via configuration like that. In the interest of time, I've only showed you how to get a very simple application hooked up with Phoenix and Ecto here, but there's a lot more contributing integrations for other libraries that you might be using in your application. For example, these are the hex packages that depend on OpenChalometry API package. Some, but not all of these are collected into the official hotel org and GitHub in this contrib repository. Also remember that many other libraries emit telemetry using BeamTelemetry that you could potentially integrate on your own using the generic OpenTelemetry telemetry bridge library. As you might expect, there's some trip hazards worth mentioning. Trip hazards because we're on a trip. Yeah, okay. First of all, you should understand how sampling works and figure out how you wanna handle it. With many vendors, you have to pay based on the amount of data you collect and honestly, there are only so many traces that anyone's ever gonna look at. And also the more data you're sending, the more performance overhead your app will end up incurring. The thing to be aware of is that you might go looking for a trace in your tooling and not find it because it was sampled out. Related to that, you'll need to make sure that you correctly propagate the trace context headers between your services. If you don't pass a trace context, then the downstream service won't know how to connect to that trace and won't know whether the parent trace was sampled or not. When that happens, you might get only bits and pieces of the overall trace that were sampled independently. Also, even when you're using NTP to set your clocks on your servers, it's likely that your clocks will drift and be slightly different. There's not much you can do about it, so it's just useful to be aware of it. For example, here's a trace where we make a call to an external service. And as you can see, the downstream service has predicted ahead of time that we're going to call it and it started work ahead of time. No, no, it didn't do that. There's really not any way to be sure where the downstream request falls in the timeline of the caller, but some visualization tools do use some heuristics to make guesses about clock skew between services.  For calls between services, potentially running on different machines, it's pretty obvious that you need to arrange for that context to be sent along, for example, in an HTTP header. The trip hazard here is that particularly when running on the beam, each internal beam process also acts as a boundary across which you need to make sure the context gets propagated. We'll cover more about that in the next section. And the last trip hazard I want you to warn you about is that your instrumentation will only be as good as the telemetry data it's based on. For example, Phoenix will generate this code by default and insert plug.telemetry here in your endpoint module. That means that the starting point of any telemetry data you send will be after any plugs that have already run up to that point, like plug.static, for example. And the way that it eventually sends its stop event is using register before send, which will fire just before sending the response to the caller. This seems fine at first, but if a response is never sent, for example, because the connection timed out and the call are disconnected, unless you do something about that, you will probably never see that request in your metrics and traces because it never completed and also never fired an exception event. So now that we've taken the first few steps and we're aware of some of the things to watch out for, we're ready to really travel the road to our destination. The path you take will depend on what your observability needs are, what technologies you're using and what existing observability tooling you already have in place. For example, maybe you're using a variety of languages to build a complex web of microservices and a service mesh and various API gateways. OTEL can make sense of all that by giving you a standardized view into the services and API calls between them as you begin to instrument more and more of them. Or maybe you have the world's largest job on a list and it's such a black box that it's impossible to understand how it works inside. OTEL can help you there too, tracing the path taken through the code as it runs in production. Maybe you don't have any synchronous API calls but you struggle to reason about and visualize the big picture of how your system works because you're lost in a forest of stream processing stages without clearly defined dependencies between them. OTEL can help propagate the context between these stages so that you can see the path your data is taking and where it's consuming resources or hitting bottlenecks along the way. The off-the-shelf instrumentation will definitely get you started and give you framework that you can start to use to understand and maintain your software system but it can only take you so far. Soon you'll definitely want to find you'll definitely find that you want to manual instrument internal parts of your application. You're in luck because the official OTEL guide for Erlang and Elixir also has an instrumentation section that walks you through how to do this. To show a simple example, let's look at the example Phoenix app we were using before. It has a straightforward controller action for fetching an index of blog posts. If we imagine that it was doing something more complicated that we wanted to see a span as a span or a trace, we can do that as follows. We just wrap the operations with tracer.withspan macro and use the tracer.setattributes to add additional attributes if we want to, whether they're custom ones or according to various semantic conventions. And just like that, we can now see both of our custom spans showing up in light step. And if we click into the series business span, we can see that there's a custom attribute showing up there as well. I've briefly mentioned context propagation before, but now that we've covered more of the details of the OTEL APIs in Elixir, let's look more specifically at how to do that. Probably the most obvious place where context propagation is needed is when making HTTP calls to downstream services. All you need to do is for each downstream call where you want your context to be propagated, inject the context into your headers before making the HTTP call. This will use the standard OTEL headers that should be honored by any OTEL compliant implementation across different languages and tooling. Perhaps less obviously, we need context propagation across beam processes so that child processes will correctly connect to their parent and those HTTP headers will be correctly passed downstream if they're performed in concurrent tasks. The way I've personally solved this problem several times in the past is just to make my own replacement for the task module that calls the standard library implementation but wrapped with a span. You can see here in the async function that we get the span context from the calling process, then we call the real task dot async implementation passing in a function that we'll call the original function but first attaching to the parent context from inside the child process. Now we can rearrange things so that we start the async task at the beginning and awaited at the end. Without our custom task module, we would end up not seeing the serious business span at all in our observability tooling. But since we propagated that context to the child process, we're able to see the async task proceeding in parallel with the database call. We can also see that our custom span is properly connected where it should be. At this point, if you don't see an integration already available, don't be afraid to build one using the beam telemetry events that the library may already be emitting. If you look into how the Phoenix integration works, for example, there really isn't much to it. You just need to attach to the telemetry events according to the library's documentation and pull out whatever attributes you wanna capture in between your start and stop events. Really, the hardest part will be reading through all the semantic conventions and mapping them to the metadata that may or may not be exposed by your library. So something else that you might have noticed in some of my screenshots so far is that the service name isn't set. And also we see a warning that we don't have a version attribute set. I found this pattern to work well for that. If you see the version, if we set the version like this based on an environment variable in config.exe, then all you need to do is arrange for that version to be set at the time when you're compiling your application. Since the versions should be based on the source code used to build the app, we should never need to change this at runtime. Most likely this is whenever you're running the mix release command. For example, if you're building a Docker container from CircleCI, you could pass it in as a build argument like this. And now we can see the service name showing up correctly as well as the version of the code being deployed. So it's really cool that we can switch observability vendors just by updating our configuration files. But what if we could switch it without redeploying the app at all? What if we could simultaneously send to more than one vendor without imposing any additional load on the service itself? That's what the hotel collector service can help us do. Let's see how that works. The collector is composed of three main layers, receivers, processors and exporters. Receivers allow the collector to listen for various protocols on various ports so that you can adapt any existing instrumentation you have today into the hotel ecosystem as you migrate everything towards more native hotel protocols. Exporters similarly allow you to plug your standardized hotel data into existing infrastructure as well as new hotel infrastructure. In between, processors allow the data to be filtered, enriched and otherwise transformed however you need. The first thing to know about the hotel collector is that there are a lot of different deployment options and configurations that are possible. You'll probably want to look through the docs to understand what kind of setup you need for your organization. But to start at zero, this is the current state of what we've seen so far. When you're not using a collector at all, then you can point each of your services directly at one vendor at a time and you don't have much flexibility into what you're sending other than head-based down sampling to control the volume of the data. One step you could take next would be to deploy the hotel collector as an agent on each of your hosts. I'm talking generally about hosts and containers here. It could be a Kubernetes host, an Amazon ECS host or just a plain old Linux server without any containers at all. The basic idea is that you're running just one collector per host and all the services on that host will essentially just send their data via network connection to local host. The additional value you get from this configuration is that you can offload some of the processing and configuration from your application containers to the collector container. It can filter, enrich and fan out your observability data to multiple vendors. If you're using a more complicated container scheduling system like Kubernetes, you can also choose to deploy the collector as a sidecar to each application container. This can be an advantage if you want to configure the collectors differently for different containers. If you're concerned about the resource consumption of a service including all its observability overhead, et cetera. Just like we saw before, these sidecar containers still have all the configuration flexibility to transform and route your data however you want. And if you're operating at the scale of hundreds or thousands of containers where you just have additional flexibility in your configuration, you may want to look into deploying a collector gateway. It works how you might expect just targeting your agents at the gateways and your gateways at your vendor tools. You can also consider more complex routing rules. For example, maybe you want to temporarily route an unsampled stream related to a particular set of services to an internal tool while you troubleshoot a specific problem without racking up a huge bill with your third party vendors. For this presentation though, I'm just going to demonstrate a very simple starter configuration like this. So let's set up an OTLP receiver and a batch processor that collects traces into a batch before forwarding them along. Then we set up OTLP exporters configured for both LightStep and Honeycomb. And finally, we have a pipeline setup to receive and process all the traces and pass them along to both exporters. You can check out the collector documentation to learn more about what's possible, including collecting host metrics alongside the traces I'm showing here. This is just a relatively simple configuration that does what we want. We also need to remember to reconfigure our service so that it's sending hotel data to our local collector instead of directly to a vendor. We just need to configure the exporter to send the data to wherever your collector API is being hosted. Also note that now we no longer require runtime environment variables in this configuration. So we can just put this in config.exx instead of runtime.exx. For folks trying to follow along at home, this is how I'm running the collector in a container on my laptop. We pass through the relevant environment variables that we used in the configuration and map the OTLP port 4318 to my laptop's network. Finally, we mount the config file into the container and tell the collector to use it. My laptop has an M1 processor, so that's why I'm running this ARM version of the container here, so you may not need to do that. And again, this is all just Docker magic. It's not really relevant to OTEL itself, but I'm including it here in case you wanted to follow along yourself. And finally, after going through all those steps, we can actually see that we get the exact same traces in Honeycomb and LightStep at the same time. So now that you have some things set up and you're collecting some data, the next step is to demonstrate the value to your team. My son here is playing the role of the business stakeholder who's so exhausted from going to meetings all day that he can't appreciate the amazing visualizations that I'm trying to show him. One important step is to start building out useful visualizations for your team. Unfortunately, I didn't have time to put together a bunch of impressive demos about what each tool is capable of. But the good news is that regardless of which vendor tooling you're looking at, you're just an email away from having one of their sales engineers show you the product.  Think about that for a second. You could try out a new vendor tool now just in minutes instead of weeks. As we wrap up this presentation, I hope I've been able to inspire you to give OpenTelmentary a try. I also wanna encourage you that what you've seen today is really just the surface of what's possible. If you're a Minecraft player, you'll know that getting to the end dimension is far from the end of the game. Getting your services all integrated with Hotel feels a lot like getting your first Elytra or Shulker box. It's when the real fun starts. So now that you've got things all figured out, you can share these new superpowers with your teammates. Maybe you have ideas for new tooling that everyone could use to look at the same data in a new way. Maybe you wanna get involved in improving and developing new semantic conventions for a specific technology that interests you like web proxies, caches, or firewalls. Maybe you wanna help build integrations to popular libraries in your language ecosystem. Huge shout out here to Brian Nagley, who works at SimpleBet. His contributions to the Hotel contrib integrations are the main reason it's so easy to get started today. And I also wanna encourage you to get involved in the Hotel community in whatever way you can. Ask any contribution helps, even if it's just writing a blog post about your experience or asking for clarifications in our Slack channel or improving the documentation so that everyone else has an easier time. And again, huge shout out here to Tristan Slaughter, who works at Splunk. Tristan has been consistently leading the charge in the Hotel Beam community and has written the vast majority of the code that powers the core of Hotel on the Beam. And anyway, thanks for watching my talk. And I hope you'll come find me later in the Elixir Slack or the Discord. Hi, thank you so much, Greg. That was a very, very interesting presentation. Definitely learned a lot.


Romania's health care system faces several challenges such as insufficient funding, a shortage of medical personnel, and inefficiencies in service delivery. To address these problems, the country must make significant changes to its health care system. These include increasing funding for health care, recruiting more medical professionals, and improving service delivery. If these measures are taken, Romania will be able to provide equitable and accessible health care to all citizens. This article was authored by Ion Petre, Flavia Barna, Daniela Gurgis, and others.