Okay, it is time, so I am very, very proud to introduce two very, very good friends of mine, Sergei and Meredith. Guys, give them a huge round of applause. Thank you so much. So this talk is about the science of insecurity, by which I mean the entire spectrum of exploits and vulnerabilities considered as a systematic, repeatable, and most importantly predictive mathematical model. We're going to examine from first principles what it is about exploits that makes them exploits in the first place, and how we can use this systematic understanding to design and implement software in which, to borrow a turn of phrase from Dan Kaminski, entire classes of bugs simply don't exist. But before I get going, I want to remark on the other talk this Congress that focuses on Turing machines. Corey Doctorow's talk yesterday on the coming war on general computation. You're going to hear a lot in the next hour about certain hazards of Turing complete protocols, and I need to make clear that what you're conveying against is Turing machine computational power in very specific places, namely the communication boundaries between Turing complete systems.  ICMP Echo does not. So that's an important distinction, and please keep it in mind.  It's the oldest game in the book, and it's already underway in the United States. Right now there's an initiative under development called NISTIC, the National Strategy for Trusted Identities in Cyberspace, which is really just the old carrot and stick game aimed at conning citizens into voluntarily giving up any possibility of anonymity online, which is really the same as free speech online when you get right down to it by holding out the carrot of, quote, unquote, safe walled gardens like the iOS app store and social networks like Google+, where the price of admission is your offline identity, backed up with a stick made from the specter of spam and malware and evil cyber criminals on the filthy, nasty internet. And no matter how rotten the carrot really is, the thing about human psychology is once someone has bought into lofty wireless promises about matters such as security, it becomes really hard to convince them that the carrot cake is a lie. So our other option is to break the stick, and that's what we're going to talk about today. Now, I actually owe this observation to my husband, Len Sassiman, who passed away back in July. Len started out in this world as an anonymity and privacy researcher. He was working on that at Kailoven under Bart Pranell. But in 2009 he shifted focus to the language theoretic security work that I had started back in 2005, because he became convinced that the future of an open internet is completely dependent on us smoothing out the attack surface and taking away the ability of repressive governments to hold this thread over our heads. All right, so if all you take away from this talk is this, I'm going to be really, really happy. First of all, the vast majority of vulnerabilities come from protocol and message format designs that require you to solve problems that are really not solvable if you want to process them securely. You know, putting yourself into a catch-22 that you cannot escape from. When you try to set yourself up against the laws of physics, you fail. End of story. But the good news is that from a design perspective, we can route around this. We just have to think about how we design inputs and outputs in terms of formal languages and conduct our implementation accordingly. So we have been living on borrowed time. When the NSA tells you that there's no such thing as secure anymore, they're probably not kidding. About this time last year, Brian Snow was saying there's a security meltdown coming, and I think the last year pretty well bears out what he had to say. And this is not for lack of trying. I mean, there have been trustworthy computing initiatives dating back decades. If you go to any bookstore, you're going to see scads of secure coding books on the shelves. There are dozens of conferences dedicated to secure software engineering and theoretical security.  And don't even get me started on hardware. We have the internet we have, and how did we get here? Is everyone really just that incompetent and has no idea what they're doing, or is something else at work? Well, it's a little column A, a little column B. So what is it that we're doing wrong? Okay, I mentioned the laws of physics earlier, but really our physicists are people like Bertrand Russell, Kurt Girdle, Alan Turing, the guys who tried to universalize mathematics from axioms. Now the problem that we as security researchers want to solve is more focused than the general problem that they wanted to solve, but we start in the same way, which is by formalizing the question, what is insecurity? Is it holes for sneaking in executable code? I mean, that tends to be how people look at it, but that's looking at everything in isolation. Furthermore, you don't necessarily have to have an obvious hole in order to execute, say, a return-oriented programming attack. Memory corruption, buffer overflows, in-band signaling, like Travis Goodspeed's packet-in-packet stuff, capabilities issues, access control issues, all of the above. None of that is really sufficiently general and also sufficiently descriptive. I come from linguistics as a background, and we have this concept in linguistics of elegance, where an elegant description of a language is one that generates exactly strings that belong in the language, sentences that belong in the language, and none of the ones that don't. Now, if you think about those possible causes of insecurity that we just looked at, Wikipedia is not much better. This laundry list here is just all over the map. And if the classification looks arbitrary, what this is really telling us, this is a lesson from the natural philosophers here, if the classification looks arbitrary, this means that we really don't understand the structure and the common origin of the phenomena that we're seeing. So Jorge Luis Borges has this great description of the classification of animals, those that are belonging to the emperor, those that are embalmed, tamed, suckling pigs, sirens, fabulous ones, stray dogs, those that are included in the present classification and bonus points for anybody who gets what paradox that is. You know, again, there's no system to this, there's no rhyme, there's no reason to it. What we need is a way to go from the arbitrary Lamarckian classification where whales and hoofed mammals are sitting there in the same clade under reptiles of all damn things and move to an understanding like Watson and Crick's understanding of DNA that has led us to the science of cladistics where we treat where we treat species as classes based on common descent. It's true that we can and we should classify exploits by similarity but if we only look at the surface similarity and not the underlying structural similarity, we're setting ourselves up for failure. All right, so what does trustworthiness actually mean in a computing system? Well, Tony Hoare, the guy that came up with a little algorithm called Quicksort a while back, remarked in a paper in the 1960s that a program is correct if and only if it handles exactly the inputs it's supposed to handle and produces exactly the outputs it's supposed to produce and nothing else. We're not that good at this. We're not that good at deciding whether an input is valid or malicious and rejecting the malicious ones. We don't have a good idea of how to trust software not to do certain things because how do you predict the unpredictable? Exploitation is simply unexpected computation caused by crafted inputs. So when you're asking the question is it a good input or is it a bad input, well, I mean, this is not a theoretical problem. We've been talking about this in computability theory for, you know, 50-odd years now. We have this concept of undecidable problems, problems that you cannot solve regardless of how much computational power you have and some undecidable problems have to do with recognition of inputs. No general algorithm for solving these problems exists. So the networked world that we live in is a composed world. You have individual components that accept inputs, must accept or reject those inputs and then generate some outputs and these components communicate and it's necessary for these messages to be interpreted identically by the end points that send and receive them because, you know, if Alice thinks she's telling Bob, you need to go to the store and Bob hears you need to go to the bus station, Alice isn't getting whatever Bob asked her to get or, yeah, Alice isn't going to get whatever she asked Bob to get her. So we're going to talk today about, you know, how the halting problem arises in both the single component case and in the distributed case. All right, so how does undecidability apply here? Well, in the single component case, some protocol formats are sufficiently complex that simply being able to discern a good input from a bad input is undecidable and furthermore, some protocols are so complex that determining whether implementation A and implementation B handle them in the same way is also undecidable. All right, so we talk about recognizers as algorithms that examine an input string and determine whether that input string belongs in the language or not. For subturing classes of languages and we'll get into what those are in just a minute, the answer is either yes or no for turing complete languages, the halting problem says the answer is yes, no, or maybe. And if the answer is maybe, you're never going to know because the algorithm's never going to stop executing. So when input recognition fails, the code on the inside of your protocol, the state machine of your protocol, is going to receive something that wasn't expecting. Any primitives that it exposes can be programmed essentially with input from an attacker to trigger memory corruption or implicit data flows, and a weird machine emerges.  So where do they show up? Well, the vast majority of programs that handle inputs scatter their validation checks throughout the entire program. The one exception to this tends to be compilers.  This is a horrible design pattern. Vaguely understood input languages are the mother of O-Day. And a weird machine is born. An adorable little root shell just pops right out. So as Talver said, exploitation is setting up, instantiating, and programming a weird machine. One of those holes that the programmer inserted when he aimed his shotgun gets blitzed by some crafted input, and the state that the internals of the code enter into were not intended by the programmer, but you know what, they're there anyway. So if we're going to understand this systematically, we have to look at this from the basics of computation as described originally by Alan Turing and Alonzo Church, but we have to think about it with regard to the lessons that we've learned from exploit programming. People like me study models of computation. People like Halvar and FX and all of our brilliant exploit guys study actual computational limits of real systems, and the great part is we meet in the middle. So the Turing machine was the mathematical model that Alan Turing came up with to study the limits of what it is possible for a computation engine to perform. And he was able to formalize a class of problems for which no Turing machine can solve them. The initial example problem that he described is known as the halting problem, because it turns on the question of whether a given Turing machine will halt, which is to say return yes or no, or end up in that infinite loop maybe state where you never get an answer back. Because even if the answer is yes or no, you have no idea how long you're going to have to wait. And if the answer is maybe, that answer is forever, but you're not going to know. So if you say, I can take a universal Turing machine and I can execute another Turing machine on it, and that universal Turing machine is going to decide if the input will ever terminate, fail, does not happen.  Unfortunately, some designs, fortunately more of them in the composed distributed system model, where you've got two different implementations, end up reducing to an undecidable problem, which means that there really is no way to fix that problem as it exists. We can avoid it, but fixing it requires a redesign. But there is no 80-20 engineering solution for the halting problem. If somebody is trying to sell this to you, then run away. If someone bothered on your behalf, fire them because you don't want to be around when it breaks. And this is counterintuitive. We're used to seeing more effort improve the results, but at the end of the day, you don't want to be the person who ends up slaving away for the rest of their life, trying to just pour more and more and more and more effort down a bottomless hole. Okay, so a little bit more about the history of halting, just to sort of give you an idea where we're coming from. So back in the 17th century, a guy named Leibniz, you might know him as one of the inventors of calculus, asked the question, is it possible for a machine to determine whether an arbitrary mathematical statement is true? And he kind of worked on that in isolation during his life. People thought he was a bit of a nut. And then Hilbert poses this again in 1928 in a list of problems that he believed if we could answer all of these problems, then we would have a complete understanding of mathematics, and we could call mathematics done and we could move on. So church and turing work on this independently for a while, and both of them come up with the answer. Nope, sorry, it doesn't work. Now what's interesting about... Another interesting thing that falls out of this is a thing called the Curry-Howard correspondence, but I'll talk about that in just a second. Conceptually, it's easier, actually, to think about the halting problem in terms of a thing called Russell's Paradox. So back on November 2nd, there was a general strike in Oakland and elsewhere. And Quinn Norton, who's embedded with... She's embedded with Occupy right now, writing for Wired says, huh, well, you know, I want to go to the general strike, but if I go, I'm going to be working. Oops. This is Russell's Paradox in a nutshell. You know, normally the normal way this is phrased is also called the Barber Paradox, where you've got a village where the barber shaves every man who doesn't shave himself. So who shaves the barber? As a funny side note, by the way, the smart-ass answer to this one is, well, let the barber be a woman. And that's actually not such a wise-ass answer after all. That's actually pretty clever, because it means you're introducing types, and that actually renders it decidable. If you're into that kind of type theory, come talk to me afterwards, because I want to meet you. All right. So I mentioned the Curry Howard correspondents earlier. Programs are proofs, and vice versa.  We're working on establishing a formal duality of this so that we can, like, convince the academics of it, but, you know, part of why I love proofs by construction is that you can basically just look at them and go, oh, that's obvious. That's easy to show people how that works. All right. So as we said earlier, inputs are a language. Some languages are harder to recognize than others, and for some of them, recognition is undecidable. So what kinds of languages do we even have? Well, there's a hierarchy. Noam Chomsky came up with it. So the simplest class of languages are the regular languages. You might also know them as regular expressions or finite state automata. Then there's the context-free languages, the context-sensitive languages, and the recursively innumerable languages, which are, and the recursively innumerable languages are the ones that are equivalent to Turing machines. These categories are hierarchical, and different categories have different properties that we can use to our benefit. So finite state machines just have very simple nesting. They can use delimiters. They can have repetition. So if you look at that graph up there, you start at the start state, and as you accept characters from an input string, you move from one state to another. And if once you've finished all those transitions and consumed all your input, if you're in state seven, then you're good. You've matched the input to your language. If not, then it rejects. Trying to match recursively nested structures with regular expressions fails, which is why when you try to use regular expressions to parse HTML, Zalgo comes out of the walls and eats your soul. If you want to match recursively nested structures, you can do this with what's called a push-down automata, which you obtain by taking a finite state machine and adding a stack to it. And then you can balance parentheses to arbitrary depth and everything's golden. But there are other properties that we see in protocols that don't fall into this category. If you have a protocol with a length field in it, like, oh, I don't know, HTTP, TCP, pretty much every protocol out on the Internet just about. With the exception of ATM, that one's regular, which is kind of awesome, but anyway. Those require a context-sensitive grammar. You know, if there is some metadata that is necessary in order to interpret the rest of the data, then that's when your context-sensitive. And finally, your turn complete if you are saying, is this input some program that produces some given result? That's undecidable. A guy named Rice formulated this in the term in what's called Rice's theorem, which reduces to the halting problem. All right, so our language hierarchy again. What this tells us is we need to stop weird machines. And we can do this by constraining input language strength to context-free or regular. Is this all about parser bugs? Well, no, but that is an awful lot of it. If you are a component of a program and you receive an input from something, you have a recognizer whether it was formally written as a recognizer or not.  And if the language itself is not well-defined or understood or if it's under-specified, then one, the program is broken. All right, and when I say languages, I mean quite literally every formal system that we use in software. I'm talking about network stacks, because the format of a packet is a language. I'm talking about web servers, SQL servers, et cetera, because there are requests for my language, but it gets better than that, because even just on a single system, your memory manager, the heap that assigns memory, that's a language. The call graph is a language, and that one likes to be context-sensitive, it really does. And most of these language recognizers are hand-rolled, and that's bad. Implicit recognizers, bad recognizers, because you cannot efficiently and effectively test or debug. When you're intermingling your recognition and your processing, like if you just recognize a part of it, and then if you just recognize a part of an input string and do something based on that, and then recognize another part of it and do something else based on that, you're gonna end up with data flows and transitions that give you a weird machine. It's running on state that it borrowed from the rest of the program. So, if you haven't recognized it, don't goddamn process it. If you have not confirmed that an input string is actually a member of the input language you want to process and you try to process it as one anyway, you open yourself up to letting an attacker program the weird machine that's lurking inside your implementation. So you need to know and be strict about what your input language is. And this should be easier than people seem to think it is, because, you know, we have our FCs. They have convenient grammars in nice shiny BNF notation in the appendix. We can and should generate these automatically rather than hand coding them, because really, who's got the time for all of that? We're programmers. We're hackers. It's our job to be creatively lazy.  But once you've done that, you can generate it and then hand the resulting parse tree off to the internals of your code and let the rest of you and let your code operate over that parse tree. But please, no more Turing complete input languages. Don't let the Turing beast devour her safe computing future.  Regex syntax is terrible, but people understand it for the most part. It could stand to be better, but people understand it better than they understand everything else. You know, part of the problem there is that we just don't have really good tools yet for handling higher classes of languages. People are still kind of starting to wrap their heads around parser combinators and really nobody likes Bison. But the tools are out there and we're working on making better ones. But where this matters really is in design. We must reduce computing power greed and keep the Turing beast from eating our investments. So most of what I was talking about there previously had to do with the single system case. Now let's talk about systems that are communicating between each other. Is what you're seeing the same as what they're seeing? The story doesn't end if there's no shellcode chestburster. So across interfaces is where we need to minimize the complexity of parsers and recognizers. So like I said in the beginning, it's the communication boundaries that we really care about. It is necessary for parsers that are involved in a multi-system exchange to parse those messages in exactly the same way because if you don't, then it becomes really easy for an attacker to generate something that one system regards in one light and another system regards in another light. Now you might be saying, oh, but this all sounds so theoretical. Nope, sorry. So two years ago, Len and Dan Kaminski and I beat the hell out of X509 in pretty much exactly this way. And we came up with about 20 different ways that you could generate a CSR, send it off to a CA that's running some particular version of X509, get back a certificate, show that to a browser that is using a different X509 implementation. And that browser believes that the certificate you got signed is a valid certificate for a domain you don't own. You can take a look at our talk from Black Hat. I think it was 2010. Yeah, exploiting the forest with trees for more details on this one because it's full of lulls. But really, this is the halting problem again just in a different form. I mentioned that Rice's theorem reduces to the halting problem. They are functionally equivalent. There is another problem that reduces to halting called the context-free equivalence problem. So the context-free languages fall into sort of two subclasses. You have the unambiguous context-free grammars, also known as the deterministic push-down automata, and the ambiguous context-free grammars, which correspond to the non-deterministic push-down automata. So if your language is regular or deterministic push-down, then determining whether two implementations are actually doing the exact same thing, that's decidable and that's fine, and you can actually automate that. But if your ambiguous context-free or stronger, this problem is undecidable. And we run into this in IDSes, too, which is kind of wacky. You know, if your problem is that you have a shotgun parser in your code, and you say, well, fine, we'll just throw a less vulnerable component in front of it, then all you've done is move the problem to another layer because instead of your target code and a possible attacker speaking different dialects of the same protocol, as it were, now it's just your IDS talking to it. So you haven't really helped any. All you've done is just moved it around. And this is research that people have been doing for a while. You know, Potato Chick and Nusham in 1998, Vern Paxton in 1999, they've all observed this. They just weren't thinking about it in terms of the broader computability picture. But that's okay. That's what academics are here for. So once you've created a computational automaton of a particular strength, the genie is out of the bottle and there is no going back from that, the dark energy will resurface elsewhere in your code. You haven't solved the halting problem. You've just moved it into a different instance of the halting problem. So when you are designing protocols, it is vital to choose the simplest possible input language, preferably regular, but certainly no stronger than deterministic context-free. And this sounds kind of intimidating, right? But Meredith, you said that length fields make a protocol context-sensitive and doing that comparison is undecidable. How can we survive without length fields? And my answer to that is we have these nice things called S-expressions. If you can bound a message with delimiters, that's still deterministic context-free. You can get away with this. It requires different approaches than the ones that we have gotten used to in the last 30 years of internet protocol design. But this is okay, because the other tools that I'm talking about have also been around for even longer. But at the end of the day, the crucial thing is we must have computational equivalence at all protocol endpoints. So, looking back at the very, very, very early days of the IETF, there's a thing called Postel's Principle, which simply stated is be conservative in what you send, be liberally in what you accept. Now, I don't mean to bust on John Postel unnecessarily.  And we needed the flexibility that Postel's Principle afforded us.  So, we're proposing that Postel's Principle needs a patch and it looks a little something like this. Instead of be liberal about what you accept, we say be definite about what you accept. Treat inputs as a language, accept them with a matching computational automaton, generate the recognizer from the grammar of the language, treat input handling computational power as privilege and reduce that privilege wherever it is possible to do so. All right. So, the takeaway for today. Do not let your protocols grow up to be turn complete. People will try. People will tell you, oh, well, we need these special cases. We need to handle this weird corner case. We need to keep bolting on other stuff to make other things possible.  Don't mistake complexity for functionality. It's an easy mistake to make, but don't let somebody sell you on it.  If somebody tells you that, oh, well, you know, this system is like totally extensible and updatable because it embeds a scripting language and data, run, run, run, run, run, in band signaling, bad, very bad. And the practical value here is that you're saving money and you're saving effort. And you can also expose vendors who are claiming that they have what amounts to a solution to the perpetual motion problem. This helps you choose the right components in order to have security that you can manage. It also helps you avoid system aggregation and integration nightmare scenarios because if you know that all of your systems have speak composable protocols and all of the systems that you're going to be composing them with have composable protocols. It all fits together like Legos. So our approach helps people to save misinvestment of money and effort, expose vendors that claim security based on solving perpetual motion. You stuck this slide in there twice, Sergey. Sorry about that. All right, I'm going to skip this one. Wow, you put it in there three times. That was kind of great. All right, so again, stop weird machines, no more Turing complete input languages, reduce computing power greed, ambiguity is insecurity, full recognition before processing, computational equivalents for all protocol endpoints, and context for your regular bitches. There are your slogans. Have fun with them. Thank you. Okay, thank you. We have now comfortably time for large Q&A sessions. So if you have questions, please line up at the two microphones in the alleys. And I think we can start maybe here on the left side. Hi, do you have any examples of languages that do this already? Sorry, not languages, but implementations that do this already that use something regular to transfer between two components? Well, okay, like I mentioned earlier, there are not a hell of a lot of existing protocols that are only regular, which is a thing that needs to change. I did mention earlier that ATM happens to be a regular message format because all packets are of a fixed length, which is a handy thing. If all of your strings are exactly 53 bytes and only that, then you can actually specify that with a finite state machine. That said, I don't know of any implementations that use a regular expression to parse ATM, which is sad. Okay, now a question from the mighty Internet. Yeah, Krokodilirian from Urk asks, if there are any tools that generate parsers in different languages than BNF? Right, so the question was, are there any tools that generate parsers in different languages than BNF? And the answer is yes. So I mentioned parser combinators earlier. So these are, well, they're both a mathematical abstraction and a practical library in a butt-ton of different languages that can actually parse some of the context-sensitive languages, which is really neat. So you know how when you're building a regular expression, you're essentially just banging smaller regular expressions together? That's essentially how parser combinators work, but they also admit recursion. So, yeah, so Don Lyon invented... I totally botched that pronunciation, I'm sorry. But yeah, he invented them back in... It was only a couple of years ago, actually. And so they initially showed up in Haskell, but since then they've crept into Scala, I think also Clojure, Python, JavaScript. So the tools are starting to creep into more and more... into more and more commonly used languages. If you're interested in learning about parser combinators, I really recommend checking out the parser combinator standard library in Scala, because it looks a lot like Java, and there's a lot of really good material out there on using Scala to write embedded DSLs using parser combinators, and the tutorials out there are way better than anything I could give you in five minutes here. Over there? Hi. So you talk about the halting problem and chewing complete input languages. Yes. But I think pretty often we don't care about is it going to take finite time to handle this input, but we care is it going to take a reasonable time? And even with regular expressions, there are certainly plenty of regular expressions there, and a regular expression library implementations have cases where they will take exponential time. Sure. Yeah, classical cases, yeah. So... Well, so my bigger concern... How do you avoid that sort of problem? My bigger concern, yeah, tractability obviously a huge problem, but really my bigger concern about Turing complete input languages is the programming the weird machine problem. So there was this hilarious result on GitHub a couple of months ago where somebody proved that HTML5 and CSS together... HTML5 and CSS3 together but without JavaScript is Turing complete. He did it by implementing rule 110 for cellular automata using only HTML5 and CSS. It is brilliant and demented. And the fucked up thing about it is that you could use that construction to do arbitrary computation in what is supposed to be only a markup language and its associated display components.  Hi. I was wondering if you could give an example around these length fields because I think that's not that easy if you're trying to build something that can transport arbitrary user strings. I've run into some interesting CSV files lately. CSV is probably... can probably be implemented with a regular context-sensitive grammar, but you run into interesting things when you have arbitrary input strings in your CSV that can contain, for example, a line break or it can contain commas or whatever. Right, right. And you've really just hit on it right there because a CSV message is comma-separated values. And if those values are also allowed to contain commas, then you've got an in-band signaling problem where how is the interpreter supposed to determine whether that comma is part of a value or whether it's a delimiter between values. And, of course, the off-the-cuff answer to that one is well-escaping, but, yeah, I'll just refer you to the last 20 years of SQL injection with regard to that one. No, so I mentioned desk expressions earlier. Yeah, those come from Lisp. Land of parentheses. And so the idea there is that you just have some set of delimiters that are not part of the sub-language that describes values. Because, like, you know, what... So when I think of a length field, I think of a value that tells me how many bytes to expect. It doesn't really hurt you to say, okay, well, rather than saying, all right, expect 42 bytes and then count the next 42 bytes, if you instead have opening delimiter, consume, consume, consume, consume, consume, until you see closing delimiter and you're done. You know, as long as those opening and closing delimiters are not part of the sub-language that describes what appears between them, you know, you can get away with this. Does that answer your questions? I'm not sure if it does. Because I think that's just my point. It's easier, in some cases, to just have a length delimited field instead of building this parser with all the escaping just to be able to handle arbitrary user input. Simply because I've run across, shall we say, broken and badly defined CSVs in the past. Sure, sure, absolutely. And again, you know, badly defined things. Looks like Surya actually has some input. Come on up here. So, you see, you're right. It's counter-intuitive. Why would I bring a parser if I can skip over so many bytes, right? Why should I implement the proper escaping discipline and the proper scanning and parsing if I can just skip some bytes? But all bytes are really created equal. And when you're seeing these bytes, when you're seeing these bytes, you don't know which ones are data, which ones are metadata. Packet-in-packet is based on that. Packet-in-packet is based essentially on the really simple machine within the digital radio chip. It's kind of sort of giving you so many bytes for your frame. It turns out that despite the simplicity, it doesn't work so well because you mistake your data bytes for your metadata bytes and the whole scheme, the whole encapsulation, goes out of the window, right? So this is basically the science part of it. There is a thing that seems simple but is actually fraught with danger. And you need to do a more complex thing, but that's actually principled and can be proven to be principled, can be proven to require less computational power in the end. Think of parsing an IP packet. Or worse, think of parsing an IPv6 packet. Right? Imagine your pointer pointing into that packet. Do you know what you're looking at? When you're writing the internal code, do you know which properties about the packet have been validated already? And whether your assumptions, when you're operating off of that pointer and happily casting into your structs and doing something with the values, do you know which assumptions are actually truly trustworthy given the previous sanity checking or not? You don't. You start in this land of mixed data and metadata and you get the mess that we're dealing with basically every day. The reason for this is protocols that use the simpler thing, such as quote-unquote simpler thing, such as length fields. You are starved for context when you're in your internal code, when you're supposed to have checked all the sanity, all the various assumptions about whether this is good data that needs to be trusted and not checked at every turn. Because you can't really check data at every turn. Where do you stop? Right? Also just one quick addition to that. The exception to the context-free equivalence problem for, say, context-sensitive languages with length fields in them is as long as you're generating your implementation from the exact same grammar, if two implementations generate their input handling routines from the same grammar, then those are guaranteed to be the same.  Sorry. Yeah, remind me later. All right. How much time do we have left? We have another 10 minutes, so we can take another question from IRC. Yeah. First of all, I apologize. This was a parser fuckup. Apparently the question about BNF I asked before was not other things than BNF, but generating parsers in different languages like C from actual BNF representations. Oh, I'm sorry, I misunderstood you. I misunderstood the question. Your standard C tool for that is Yak, yet another compiler compiler, or the GNU version, which is Bison. And the input language that both of these tools use is literally just straight up BNF. Okay, so somebody mentioned that some work apps require exponential time to execute. This is only true if you either use a backtracking implementation or add features like backwrapper events, so plain work apps don't really have this problem. And the other thing about backtracking regular expressions is that like PCRE regular expressions is that they have a stack, they're actually context-free. So, yeah. Yeah, thanks for pointing that out. But you see, it's not really the running time that should concern you. It's trying to solve an unsolvable problem when validating input. Yeah, I mean, the real problem that we want to solve is let's stop exposing weird machines because that way we cut the thread of malware off at the knees. You must make sure that when data reaches the processing logic, it has been checked and you're fully certain what is it that you are operating on. If you don't, you've got the explosion of state, you've got probably transitions that you did not anticipate, and you've got that computation, that unexpected computation that is otherwise known as being pwned. So, when we say that an exploit is proof, it's proof by construction that a computation is possible in the actual environment that you're exploiting. You know, you can't argue with proofs. This is science. Yeah. When you try to argue with reality, reality wins. Okay, another question from the internet. Well, Yuv Gang Yu asks, what your opinion is on the trend to invent ad hoc proprietary protocols between JavaScript apps and browsers and their servers? I'm sorry, can you repeat the question? What your opinion is on the trend to the trend to invent ad hoc proprietary protocols between JavaScript applications and browsers and their servers? Oh, so you're talking about, like, you know, RESTful APIs and stuff like that? Yeah, so I'm not a huge fan of proprietary protocols in the first instance. Yeah, you know, I can't stop people from shooting themselves in the foot. I can advise them to put the gun down. But I can't stop them. Well, you should realize that before RESTful was invented, CGI was an even bigger mess.  Which things there were objects, which ones were methods or actions or messages, at least RESTful gives you an opportunity to start structuring that. But, yeah, we're not looking forward to the Turing complete web application future. I can't stop them. Okay, so we're doing one more question. Okay, then the more important one. Several people asked how about using, how it is about using length fields for performance, like in Travis Talk and the other one. And then delimiters for certainty. I missed everything after performance, sorry. Okay. I'm half deaf, I'm bad. Okay, using in protocols, using length fields for performance, and then additionally delimiters for certainty. So, I think performance is really a red herring. For that, I want to refer you to Matt Mite and Guy Hoos' first name. I don't remember Darius' paper from 2010 called Yak is Dead, where they blow up the myth about parsing being, like, you know, impractically heavy and so on and so forth. You know, I have not seen, like, real-world runtime statistics, comparing, say, context-free to context-sensitive with length fields, but we could gin that up. And I'd be willing to place a bet right now that context-free will actually win on performance. Also, consider that by including both the length and the delimiter, you've just created a semantic ambiguity, because you see, what do you think the implementers would do? Some of them would go by the length field, others by the delimiter. You know, and you'll have the wonderful world of ideas evasion all over again. By the way, if you were there for Dan Kaminsky's talk, the conclusion of it, the tricks to reveal your ISPs doing computation you don't want them to be doing on your packet, exposing the fact that they're doing this computation. Well, you know, this is the old world of ideas evasion, writing again, showing that if you don't have computational equivalence, it's damn hard to hide it. And it's easy enough to expose it. So don't rely on ambiguity for security, because ambiguity is insecurity. Okay, thank you. Give a huge round of applause for this awesome talk, please. Before you all leave, I heard stories about your lab coat. So could you come here, please? Can we please close all doors and put the house lights down? So because we have something to show here. Alright, so my dad is a chemical engineer. He worked for Exxon his entire career and before he was born, he spent some time in the research lab there. And when my mom, who I visited my parents for Christmas in Houston, and when my mom heard about the Odeo Lab Coats thing, she was like, oh, cool. And when I dug my dad's old lab coat from the research lab out of storage, the one that I used to play dress up in when I was a little girl. And I was like, okay, well, this is CCC. It needs to have art in it. So I decided to go ahead and bling it out a little bit with some UV decorations. And I guess they're working on the light problem there. So yeah, this is not the very best lab coat in the world. This is just a tribute. And I want to thank Dan and Travis again for putting this art in the Bitcoin blockchain because it really does mean the world to me. Thank you. Okay, lights back on, please. Yeah. Okay.


Hey, everybody. Welcome to our panel on observability in the cloud-native era. And I couldn't be more proud to have such an esteemed, you know, awesome panel of superstars here. Charity majors from Honeycomb, ETL Schwartz from Commodore, and Jason from Datadog. And they are going to demystify all of the things you want to know about observability in the cloud-native era. So I'm just going to give a little bit of framing to the discussion. We have prepared some questions, but we are more than happy to receive questions from the audience. You have three people that you really want to tap into and make it challenging, you know, roast them. And yeah, you're definitely, so there's a mic in the center. You can stand up and ask your questions. So we want to talk about how, when systems are now getting more complex and downtime is more painful and costly, it's becoming mission critical to have actionable data about your systems, to be able to make the right decisions in real time, and particularly when managing high-stress situations like production incidents. However, like all things system engineering, this is easier said than done. Not only is there a jungle of tooling out there, from open source to commercial, but understanding what to monitor and even what not to monitor, how to build your monitoring and observability, the building blocks, and really ensure that everything, eventually when you need it, your dashboards will reflect the business critical issues you need to know about. This takes years of experience. It's not easy. That's why I'd like to tap into some of these monitoring and observability experts for our very own community who can share some insights from their hard earned experience, what pains and friction they see and have seen with their user bases today, and particularly with the complex cloud-native systems that people are building. What are some good tips to overcome some of the common challenges? Without further ado, I'm actually going to let you guys introduce yourself and tell a little bit of a fun fact. Get started, Jason. Tell people about yourself. I guess I'm first. I'm Jason. I'm a staff developer advocate at Datadog. I've worked at a number of places that you can read up there. I don't know that they really matter, I guess. I was at Gremlin. I did a bunch of chaos engineering thing because it turns out that it's a good way to test your observability is to just break things and hopefully you can see what happens.  Fun fact, I used to be a chef, so I would work for 10-hour days, Monday through Thursday, and then Friday, Saturday, Sunday, worked in a restaurant. Yeah. I'm still waiting for you to cook for me. All right, ETL. So, hey, it's a bit loud. ETL Schwartz, city of Commodore. What we do, it's less known than Datadog, a bit less known. We're a Kubernetes real-heavility platform. Basically, we try to make Kubernetes simpler.  So that's my background. And like a fun fact, I really like pandas, like the animal, right? Like not the Python package. Okay, with the package. So, yeah, that's like my mascot for a lot of time. And my name is Charity. I'm the co-founder and CTO of Honeycomb.io. And I identify as an operations engineer before starting a company, which was never on my radar. Before doing that, my niche was really being the first info person who would come and join a bunch of software engineers when they thought they had something real that users might actually want to use. My fun fact is, after a lifetime, I grew up on a farm and we ate most of our pets. I was very utilitarian when it came to animals. After a lifetime of this, I just got my first foster cat.  She's disgusting and I love her so much. Amazing. So it's awesome to have three of you here. Sorry. So, based on that and your short bios that you gave us, you all come with both hands-on experience and now you have more of an industry perspective as vendors building tooling, observability tooling for cloud-native systems. I'd like you to share a little bit about what you've experienced as a practitioner that helps you understand what users need today. And you can get started. I mean, literally, this is why Honeycomb was started because, you know, back in the days of P.A.R.S., my last job as an ops engineer, it was kind of, we were doing a lot of, we're doing like microservices before. There were microservices, a lot of like cutting edge stuff that we didn't really have the tooling for. And as a reliability engineer, I was professionally humiliated because every day the thing was going down. It was just like every day a new app would hit the top 10 in the iTunes store and down would go P.A.R.S. And I was just like, what is happening? Like I tried everything. I tried all the logging tools and all the monitoring tools. And like, you know, what they all had in common was once you figured it out, yeah, you could add a metric, you could add a dashboard, then you could find that problem immediately the next time. But these problems weren't happening over and over again. Every time it was something different. And like the last generation of tools just isn't equipped for that. Like you really have to like have a sort of paradigm shift to like away from basic metrics and logs to more of an event-based trace-based fundamental. You see all your thoughts? Yeah, sure. So like, I started my career working for a company that does like a fraud detection. And that basically means that our KPI was finding bad people that are using stolen credit cards. Every time our system was done, we basically lost a lot of money to bad people. And this in turn really forces you to be like, quite good or at least not really bad. Because I was the one writing the mails to the management every time we had a downtime. And the first time is how much money did the company lost? We lost 100K, half a million. So this like in turns really made me, first of all, not be too worried about downtime, because you know, it happens to everyone on one hand. But on the other hand, like being obsessed around what's the best way to prevent downtime and even more importantly, learning from incident and from like past issues. So when we started Commodore, like that was the main thing that I always had in mind, like this guy or girl in the end, that is super sad right now, because all of his production is down, everyone is shouting at him. And all he wants to do is you know, like to find the right switch to turn on in order to make sure everything works like works. And I think to this day, you know, talking here with a lot of like production engineer, platform engineers, I can really relate to the pain because in the end of the day, it's like, it's hard, like it's hard to be like the one in charge. So that's pretty much it. Yeah, my foray actually into this whole space is actually slightly different, because I think most people come at it as a, you know, I have a service or I have a website and is it up or is it down? And way, way back, I worked at consultancy and one of our clients was this church in a small rural town in Midwestern US. And every year they would have this Christmas show. And so it's probably right about this time of the year, someone else is dealing with us now at that company. But they would literally have like everybody in this town would go and try to buy tickets for their Christmas show, which was like 100,000 people in the span of like they would open up at like 10am. And so it was this thing of like, I'm a web developer, I wrote PHP code. And it became the thing of it's not whether it's up or down, it's the performance, right? Can we scale and handle this? Because it's going to be down, right? Like that's a guaranteed if you don't have that scale. And so mine was kind of interesting, because it was mostly that got me into web performance and started to think about like, how do you get things faster? Which is now very common with what we do, right? Everyone's thinking of performance. Slowly the new down. Yeah, that's true. It's interesting. People have zero tolerance anymore to kind of wait around for the website to load or anything else. Okay, so on that note, you guys spoke about the things that kind of brought you to build the tools that you have been building and also the things that you learned on the way. I think one of the questions that often comes up that I've heard when I whenever I've been kind of in the in this area is that is it even possible to have visibility into all the bits and bytes that systems have today from the millions of services communicating with each other, get CICD, shift left, all the engineers touching everything all at once. It's built upon different stacks, tools, clouds, and they all behave differently. How do you find a needle in a haystack when it really matters? I mean, first of all, the answer is no, of course, it's not possible. It's not possible. I mean, I mean, in theory, it might be possible, but in reality, nobody is going to pay for an observability bill, an OLLI bill. That is 20 times that as big as their entire production systems. Nobody's going to do it. There are diminishing returns that kick in very quickly, which is why like all of this is why I get really pissed off whenever people like we can't sample every line is sacred.  There's a lot of this that is visible to us, but most of it isn't because it just works. And people who are as old as, I don't know about you, but Jason immediately, we remember when it didn't just work. Remember, a curl crashes being like just a thing that happened every week? Yeah, doesn't happen anymore. We don't have to monitor that for that for the most part. We can just be resilient to it. I could keep talking, but I'll leave I think, you know, like the the base premise of it's impossible. I think it's right. Like when we see when I see like, modern development, so much moving pieces. It's impossible. And even trying to do it, I think like, it's set up for failure. Like my, my like two cents here is that and then it's hard, right? Is building the application with the mindset that things are going to crash. Like everything you do, you add a feature flag, it's going to mess with your production environment. You are adding like another step over the pipeline, what will happen when this step won't work? And you need to deploy something fast. Like there are so many failure scenarios. But every time we add another layer, it's like just adding more and more like failure scenarios. And we need to have that in mind. I think the thing that is super hard for most people is they are like optimistic in nature. I don't know. Like I was a developer, right? Like, and I wrote code and I hope it worked like most of the time. And it took me a lot of time to understand when I was a junior developer, that like nothing is going to work. And even if it does like it looks like it's going to work, it's going to fail sometime. And I think like even the most trustworthy things which is like obviously S3. Like I remember the S3 like downtime. And you know, even the AWS, right? Like those guys are good, right? And they store the icon of like S3 being down on S3 because they didn't talk about like what's going to happen here. So I think like having like failure mindset is the key thing to have. But on the other hand, I will say it's very hard to achieve that. Like you do need some maturity and some understanding of failure scenarios. Yeah, I mean, to the just underscore the point, right? Like, yeah, you probably can instrument everything. If your app is small, if it's very, very static and you don't plan to change everything, right? Like if I had a time machine went back to myself 20 years ago, building simple PHP apps or Perl scripts, like, yeah, I couldn't monitor that whole thing because I had one Linux server that I compiled the kernel for and had a stupid Perl script and hasn't changed, right? But let's face it, we're here at this conference and it's cloud native con. If you're doing anything in the cloud, you're doing things dynamically, you're probably doing at least one deploy a day or even one deploy a week. It's changing enough that it's just impossible. Everything breaks. And like to the entire, this is the entire point of this is like, why is everything so hard now that we live in the future and everything's great? Well, it's because it's because honestly, like, you can't think about your systems. You can't think about your software is just the code, right? This is why unit tests are not enough, right? Your unit test driven development is great, but all it tells you is that it logically probably works. But that doesn't actually mean it's going to work or do what you think it is because these systems are emergent, they're complex. You don't, this is why I feel like observability driven development. Like you don't know if your code works until you've watched it in production as real users are using it. And then all you know is that it works for now for some people. Okay. I think those were kind of depressing answers. I'll be frank. All right. Provide some tips and truth is depressing. Like, you know, everyone that tells you that is now yeah, you can monitor everything expecting more optimism and more. The optimistic part is it means that you all will have jobs, right? It gives you a chance to, well, there's round of applause. But there's that. But it's also for me, it's you have a chance to learn. You have a chance to improve your skills and get better and develop depth in what you do, right? Because nobody actually, if you're a developer, you don't want to be the person that just writes code and says, I have unit tests, it works for me, right? You want to build something that's solid and reliable and you want to build systems that are more complex and you want to get better at these things and build these really cool things.  Everything's about risk management, you know, and I don't think this is depressing at all. I think it's hilarious. I think that you just have to have the right sense of humor. It depends on what period of time. I think when you're in real time, it's probably depressing, but afterwards you can laugh about it. I disagree. But then I come from operations where, you know, I don't know, being down is funny. It's funny. All right, so following up on the previous question, maybe give us like some practical tips. Like you started to dig in there, so maybe we'll dig in a little bit further, double click on what you were saying. Like what are some good tips that you would give in order to kind of overcome the common challenges and do your best at finding kind of the issues as close as possible to the incident? Yeah, well, I mean, how do you get better? Practice makes perfect, right? I mean, this is, for better or worse, like I took that deviation and went to Gremlin because I still believe in chaos engineering, right? And per charity shirt, right? Like you either test in prod or you live a lie, right? And so, well, you could wait for those actual production outages, but hopefully you're not having those regular enough that that is actually practice, right? So you have to set up scenarios. So chaos engineering, right? Like start testing your apps, try to break them and see what you learn. Yeah, I would say that, you know, like the thing with chaos engineering that always, you know, on the, I understand chaos engineering, but my feeling, especially as a vendor, right, but also as a practitioner, there's a lot of chaos always happening. Like, I think most people have enough production issues as it is, like to, you know, they don't need to add more production issues. I think the play of like, how do I learn from one production issue to another? Like this is, this is my answer for that. I think, again, most people don't really know how to understand, like how to learn from one production issue to another. Usually, usually, like for every production issue, like two things went wrong. Like there's the actual problem. I don't know where I wrote the bed call or something like that. And something around the processes, processes or observability was broken as well. Because if not, maybe I would have detected sooner. And I think like doing those post-mortems, learning from one production issue to another, same if you're using chaos engineering, like, okay, now I know things are failing. What's going to happen next time? I think like this is something that not a lot of companies are doing and can really help them elevate their game as time progresses. Yeah, I mean, you could say it's depressing, but I actually think it's not, because there are so many things that are broken in your systems right now, and it's fine. Like, we're resilient to most things failing. Like, there's little, like, you know, bits getting flipped on disks all the time from solar rays. We're resilient to them, right? There are Linux kernel problems happening all the time. We're resilient to them, right? The thing about cloud native is that the, we're able to have a level of abstraction now that we can more or less stick around it and build our apps without having to deal. Everything breaks eventually, but we are getting better over time. Like, there is so much software now that you can just take for granted, and you don't even think about, which lets you, which frees up your time to actually focus on the code that matters, right? Your software. Which is why I feel like, your first question about, do you have to instrument and understand everything? No. I think about it kind of like a headlamp. You need to be able to instrument, you know, computers are having these blips and, you know, spikes all the time in errors. And until someone comes along and goes, that's meaningful, I need to understand that, you have the luxury of ignoring it. Which is the other reason that I think that, like, a lot of the LLM stuff is kind of getting it upside down because, you know, only people can attach meaning to things, and computers can tell you when there are spikes, but like, they can't tell you if it means anything. I will select the most, like, unpopular answer, and the real answer is, like, use less technology, right? Like, it's maybe not the best thing to say in a cloud-native, like, conference, right? But, you know, like, in the end of the day, like, I'm really trying to help, like, people, right? Developers should be faster and so on. But if I could, I would build all of my product on Excel or something like that, something that never breaks. Super simple. Now there's Python. I like Python. And, like, if it would be a possibility, like, I really try to keep things as simple as possible. Like, again, you know, like, it did work for, like, a lot of time.  If you can build your app in a lampstack, by God, do it. So now I'm going to ask the controversial question. Then why Kubernetes? I think the answer is because, like, life is complex, right? Like, again, I would really wouldn't want, like, anything no microservices. And this is how commas were started, like, one monolith, one micro, like, no microservice, right? Like, it was a monolith and one repo and one database and that's it. But, you know, it's life. It's, to be honest, like, it's life. Then you want disability and disability. And those two don't act well together. So I think it's always, like, on one hand, like, you know, I always love the new shiny thing. There's something inside of me that likes, like, testing your technologies. But finding this balance is, I think, the most critical thing. And I think a lot of the people also are right in the room, in the conference. We like those, like, being on the bleeding edge. There's something super fun around that. But then you are on the bleeding edge and, like, suffering. So I think, like, finding this balance is super hard. And a lot of the times, like, one of the questions that we ask all of our customers is, why did you migrate to Kubernetes?  It worked, like, your company is making a lot of money. Why Kubernetes? And some of them have really good black reasons. And some of them are, like, yeah, everyone is moving to Kubernetes. And we're also moving to Kubernetes. Someone in the management said that. And, like, now we're stuck on a seven-year migration plan. So I think, like, that's the reality, right? Like, again, maybe unpopular. But I think that's reality. Well, I think there's two interesting points, though, between what you said and what Charity said, right? Charity mentioned that there's layers of the abstractions and things that keep working. And there is simplicity. So it's not to say that Kubernetes itself is simple, right? We've all worked with it. It can get very, very complex. But with these abstractions, you're essentially, you want to simplify the concerns that you have. And so a lot of what we're doing, yes, Kubernetes is not simple at this point, but it makes life simple for your developers in some ways, right? Because they can just take their app, put in a container and deploy it, and it generally works, right? And so a lot of what we're doing with technology these days is just shifting where that simplicity is. And so, again, you know, keep things as simple as you can for what you need. And for what you need, you're all here. It's probably Kubernetes to make developer life simple. Unfortunately, it probably makes your life a little more complex. Yeah. You know what else wasn't simple?  Like you're sort of localized in complexity in one little place. I'm not sure if like Kubernetes is like the first thing. I really like Kubernetes, right? Like, obviously, I started the Kubernetes company, right? But I'm not sure if like developers is the first thing because for me, like personally, I had Chef, I had Puppet, I had like all of those, sorry, tool when I needed to work with them. And then I found out about Kubernetes like seven years ago and everything is YAML. I don't really like YAML, but it's much better again than the alternative. So for me, it's like more, oh, this is a much easier way of saying what do I need as an infra guy, not necessarily as a developer, like I worked with containers like a variable form containers and I had operation guys like doing the Chef for me and it was like good for most most time. We shouldn't be making these decisions for technical reasons for the most part.  And, you know, the only, the only like other reason is if you're trying to hire people and they all want to work on Kubernetes or they all have that skill set. That's also a super valid reason to do it, but we shouldn't really be like sitting around here talking about which is more complicated is probably the wrong lens to be making technical decisions. Okay, solid enough. I'm going to take a pause and see if anybody from the audience wants to ask anything. Any questions? Go for it. Stand up and ask your question at the mic. Thank you. Did we get way off topic? No.  It's fun. I'm glad to. Observability things are easier in that context. Great question. Did everybody hear it? Fantastic. I will say that like in general Kubernetes makes a lot of things much simpler. Like adding like fluent D or like agent and things like that. Kubernetes makes standardization much easier so you can add all of those things even without working like again like fluent D, elastic, Rafaana and so on. So on one thing it makes adding those observability layers much simpler. And by itself you would have expected things are actually easier. I think the main downside of Kubernetes and again I love Kubernetes is it makes things so simple that it's very easy to add more and more like stuff. It's very easy and cheap to add another service. So I think that like most problem with like Kubernetes observability is it allow an easy standardization for everything which is great but it allows everyone to start his own container or cluster or namespace. And in reality because you already have Kubernetes you are like yeah sure I'll add another service and another service. And before you know it you have like 500 services running out in the wild. So I think from an in-front perspective it's much easier. But from like you need someone strong saying we shouldn't like not everything needs to be a service, right? So that's my answer here. Yeah I completely agree. It's trade-offs, right? Some things have become easier and those because the scale have led to other problems. I think the number one thing that I think makes the most sense to focus on is making sure that we're increasingly collecting our telemetry through the lens of the application not from through the lens of the system. Because system level metrics you know they're powerful if your job is running infrastructure and if it is running those level systems but for most of us our job is not that it is making our end users happy and so you need to be able to capture all your telemetry in ways that you slice and dice and say what was this user's experience like? What was this group of users experience like? Because it's completely possible for the infrastructure to be healthy quote unquote and your users to be very unhappy or for the infrastructure to be unhealthy and your users to be happy and which one do you care about most? Well you always care about the users the most. Yeah that makes sense. Anybody else have any questions? Any other questions from the audience? It's a lot simpler but oftentimes the reality is like we operate pretty much everything on public cloud that have a lot of managed moving parts that do not expose somebody data. So I see it like in the long run we kind of get more less visibility compared to what we had when we were running on our own. So what are your thoughts? Because that I feel that I'm feeling the pain already but I feel it's going to be accelerating a lot in the coming days. I think like it's a great point because and I think everyone here talked about like the abstraction part right like on one hand you don't really want to care about the fact that there is like some Linux somewhere in EKS right like you don't want to care about this until you really want to know about this because you are using too much like I don't know a bandwidth or whatever I know through I'm not sure what. So as I see it like most companies don't really care about again from what we see from our customers they don't really care about the underlying infrastructure for big companies it's a huge pain it's a huge problem I think it's more like of a learning curve for the cloud providers like I do hope that like in five years no one will really need to know because EKS and GKE and EKS like all of them will be good enough I think we are currently living in a state that the abstraction is not complete and this in turn just like surface the pain on one hand no easy way to understanding on the other side which is like bad but it's mainly for from what we see like for very big enterprises for like 95% of the users they don't really need to care So I know we touched on this a little bit but I guess let's expand on it a little bit what problems are still unsolved and what is the industry doing to solve them? That's a very broad question Yes I think one of the biggest sources of pain continues to be just having so many different sources of truth when you've got like one thing over here for your metrics and dashboards you store that same data again for your logs you store it again for your traces and you store it again for your profiling you store it again for your security tool and like the only thing that nits these things together is you the person in the middle who's like just like trying to eyeball time stamps and go well this probably the same thing as that it's a huge problem you should have a single source of truth from which you can derive the metrics from which you can derive the but obviously I'm a vendor so I have a vested interest in this but I think it's a huge problem Yeah I think I will also like give my side which is basically and I think you said it like perfectly there are like two kinds of problems on a high level on Kubernetes there is the end users problem which are the developers or that are doing the things that are really moving the business forward right and there are the info people info guys people problems that is like I have problem with the nodes with the memory and and so on and so on and I think like currently there is like like a very very like two different camps like the developers and the ops people and something somehow need to bridge that it can be that the developers will be more like Opsi the Opsi will be like more developers or some tools will like be the bridge but I think where the industry is currently like heading towards is two very very different like type of users that are using Kubernetes and each one is cursing the other user because he's not happy with that and somehow they need to coexist and I think like this is what we see the most like we try to bridge that gap but in reality like you know like I've been I've talked with people here for like the last five hours or something like that developers are complaining on the platform team not like understanding them and the other way around and I think like this is something that really holds back our industry like those camps yeah I'd say that you know charity's point is great because I do find a lot of customers that are like it's it is that correlation thankfully like at Datadogs are you set me up not to be a bitch but we have all of those in one place and what I found is our problem is actually when I talk with customers it is become the thing of you know we talked about DevOps and moving developers and operations closer together and yet how many of you work at a place where you have like the observability team right and it's like your developers don't care your ops people have stopped caring because now it's that observability teams problem and so we keep having these silos in these divisions and really that's being the issue not a problem and I would note that they're in the same place but there are different data sources you can't actually correlate them but not to be a bitch but the observability team thing I think is actually a really great pattern in as far as they're like the they're doing vendor engineering in a way you know because I think that there are lots of powerful vendors out there who have these very powerful powerful tools but you need a team kind of be that that that layer did to be making for example some libraries that everyone can use that are that are very you know idiomatic that you know fit in with your tooling and all the languages that you need and all that stuff and and I feel like I feel like the observability teams are doing that are really doing God's work yeah I mean there's definitely like a mix right there I'm not saying that observability teams are bad by any means but there is a natural tendency especially in some large organizations where they become those people right and there's this disconnect and then people stop to care because that's not my OKRs or my KPIs for the monitoring and somebody else is watching that so I can just ignore it and obviously like that's when it becomes pretty bad you can't be an engineer if you don't understand how your code is working in production you can't call yourself an engineer I don't think I feel like we should coin a new discipline like DevOps or something like that like that's something that we should start discussing I actually think that DevOps is on its way out and here's why I think that things we didn't ask charity go for it in the beginning there were engineers who wrote their code and owned it in production and then systems started to get super complicated and we're like ah shit there's too much for one person to do I know let's slice it up so that half of us write the code and half of us understand the code that was a terrible idea DevOps is like trying to knit these things back together and it's a value it's a very you know valid you know thing to be doing but I really feel like that we're all moving back to a world where everyone writes code and everyone is responsible for owning their code in production some of this is more infrastructure code some of this is more you know application code whatever but like our systems are actually getting so complex that if you didn't write the code you're not going to be able to debug it and operate it and if you're writing the code but you're not operating it you like this is all about feedback loops right software engineering is all about having a feedback loop but if you're not subjecting yourself to the output of the code the shitty code that you put out there into the world then you're not you're not actually ever going to understand it in order in the way that you need to in order to write good code yeah all right I'm gonna we have two minutes left if there's any more questions from the audience yes can you talk to me about like costs and using observability tools to measure costs and like define value for not just observability teams but tools as well great we always love cost questions yeah we actually have some great blog posts in the honeycomb blog about we use our own observability tooling to do costs and cloud costs and everything I'm so glad you asked that I feel like I for one am glad for the death of the zero interest interest rate phenomenon I think that it's forcing us as engineers to pay attention to costs and real business problems I think that like there's been this pathology since the beginning for a very long time where CTOs, VPs of engineering are considered like the junior participants in the sea level suites because because we have historically not known how to represent our work in terms of that the you know the universal denominator of of dollars and cents so I don't know I've got to let Jason take some time to talk to this too but like you're on the you're on the right path we absolutely need to be asking those questions yeah I would say you know when I think of costs think of it as any other resource right and I say this because with things getting more expensive so many organizations are like spend less spend less right but if you were to deploy an app and someone from the sea level came to you and like less CPU use less CPU like like what do you do with that like you can't just continue to dial that down in the same with costs so my notion with cost is track it like any other resource and focus on like the effectiveness and and less on the driving down cost it's about the value that you created at the end of the day yeah I think like there are really hard thing about cost is you need to find some balance between like the cost and the reliability or like the how the code is actually running I do think like the industry is currently like lacking something in that area like there are a lot of cost tools there's a lot of observability tools but finding this balance is really really hard like I think it's it's like one of the biggest like problems I think currently in the industry we try to solve part of it but I think that like something do need to change in that regards like not sure what okay so I think we're just about out of time thank you all for this really excellent panel I think we got to touch on a lot of things I didn't actually get to ask the last question which I think was the probably the one that everyone wanted to hear was about like kind of open source versus commercial tooling a little bit on in the space but we're done we're out of time so we'll have to do it next time better luck next time but all three of them are here on the ground you can you know talk to them Honeycomb has a booth Commodore has a booth do you guys have a booth? They probably has a booth everybody has a booth everybody gets a booth so you can continue the conversations with them there thank you so much for attending thank you Sean thank you


Today's one of my favorite days. It's spring, we're nearing the end of the academic year, and that means one thing, undergrad research presentations. So I'm on my way down to the scholars week presentations that we have here at Western. These are presentations from every department. I've got two students in physics and astronomy. They're students in chemistry, biology, neuroscience. They're art students summarizing their research, summarizing the projects they've been working on for months, years. This is the only chance that I have during the year usually to hear about what's going on in other departments. What's cool is I've also seen on campus some other presentations, so like art and design program has been printing t-shirts of designs that students have been developing. One of the really fascinating things is how different scholarship looks like, how different scholarship is between different departments, and it's not just like do you publish in journals, do you write books, do you write articles, do you go to conferences, but also just the way we structure research projects, what we give students. Some people do a lot of programming, some people do a lot of design, some people are running experiments with humans and animals, just such an amazing immersive range of projects going on. And it gets really weird to me in humanities, but that's what's really fascinating is these are rigorous and fascinating subjects as well, so they design their undergrad experiences totally differently. So as an educator, as a mentor, it's a really valuable thing to go see how students are getting their projects structured, what makes for a good and successful research experience for a student. Also a behavioral neuroscience major. I'm Seth Novak, also a behavioral neuroscience major. So our poster is called The Road to Musicianship, effects language along the way, and we gave 15 non-musicians musical training for 11 days and found that their ability to discriminate small differences in voice onset time or temporal acoustic features actually improved with musical training. My name is Zach, I'm a racer student here at Western Washington University, and I've been studying the variability of EK draw. My name is Matthew Skargans, and I'm with WWU's Physics and Astronomy Department, and we've been studying stellar activity cycles. I'm Zach Domingo, Mayor Schoner, and this is our research regarding crime patterns. We generated four different models, and we found that crime, specifically domestic violence and alcohol crimes, increased from winter to summer, and decreased from summer to winter. My name is Serena, and I am creating an autism story to Dating App. My name is David Seaman, and I work with the Department of Physics and Astronomy at Western Washington University, where I work on developing a theoretical model of nanoparticle doped polymer film systems. Hi, my name is Hardy Tijuana, and I'm in the behavioral neuroscience program here at Western Washington University, and I study neurobiology of violence and disease. Hi, I'm Nick, I'm a statistics major at Western, and my project is on short-term volatility curve predictions using a singular spectrum analysis. Hi, my name is Liv, and I'm Michael, and we were testing to see if up-regulating the mox1 gene in mucophyllum melanogaster increased their lifespan. And if females, the answer is yes. As of every year, that did not disappoint. So many cool posters. I spent like two hours up there just walking around and hearing people talk about their research. What an awesome experience. My one regret is that I didn't talk to the person whose poster is about coffee. Maybe next time.


תודה, אני ביקשתי, פתחות את, תשומת לבכן, במסיבת האתרוניים. As to, לתפתחות של דעתי, Point out your intention to a development according to, in my opinion, as, risking our security, I am referring to the bad agreement with Iran. Why bad? Because, A, it gives massive funds to terrorism, והוא חälletם spine, ת пут presents, פלאה והם ״down blue color Open Santos Halina只 danced at the fair where it lies. ת jointly, שי consistsa yardland. באמיל entrepreneurship specialize in esprecht TO IT. נדמה.32existences existence communityvisor כאן cybersecurity. ע spoilersizing organization. נדמה. буלפת. נדמה. נדמה. נדמה. נדמה. 20 פעמים יותר than the older version of central folks. Currently they have 5,000 from the older generation and now within two years they can produce double the amount of 10 or 20 times more than what they have. It increases their ability to produce uranium and mass scope of nuclear power. The third thing is a nuclear immunity. Essentially Iran is essentially immune to any sanctions, the P5 plus 1 committing you can be a threshold nuclear state and you are immune from an attack. What the agreement does not include is person for most A, no demand to stop terrorism, not addressing the issue of ballistic missiles, no addressing that's whatsoever, not addressing the issue of stopping development of nuclear weapons. Only two phrases, it's a general statement, nothing, they're not getting into the guts of producing nuclear weapons. This is what actually triggers nuclear power, not to ballistic missiles, not to nuclear weapons and of course no sufficient international monitoring, not effective Iran. This is a bad deal and I would like to stress it makes Iran get anything, everything and not give anything, nuclear weapons making no distinction between right and left, Jews, Arabs, orthodox, secular, there's a real serious threat, we must address it, so what needs to be done? First we need to acknowledge the fact that the current agreement is far worse than the previous nuclear deal and when I'm telling that, I'm saying that not because the previous agreement was good, it was not good, we need to understand why it wasn't good, it wasn't good because even Obama admitted, deal, zero breakout time in 13 years, Obama in 2015 said that in 2028, Seigerman T himself signed will essentially facilitate Iran to reach near zero breakaway time to reach nuclear weapons. This deal was so bad that I was needed as a Prime Minister for my friends to make some steps that when you're facing an existential catastrophe, you need some extreme measures, for example going to the American Congress, I cannot not act in public sphere, the only arena that has some clout over policy, I went to Congress, I went to the UN, I acted in every possible way, numerous other steps, some of them you know, some like the revelation of the nuclear archives, what showed the world, the exposure to the world that Iran decades ago already planned to produce atomic bombs from the type that destroyed Hiroshima. All of this in a bit to counter the public support and of course to conduct, to conduct operational activities as the former Chief of Staff Eisenstadt said without Israeli actions, Iran would have been on a nuclear threshold a long while ago, but to long say that is the diplomatic action and the diplomatic action created the crucial decision of President Trump, the U.S. withdrawals from the nuclear agreement with Iran now, tell me, but it did not stop the nuclear plan, so first it halted the crippling sanctions and direct military threat, incredible one, the only thing that stops that, nothing else stops that, maybe the credible military threat will not, but the deals will not stop the nuclear ambitions, it did not stop Syria, it did not stop Iraq in the time of Saddam Hussein, and it did not stop Qaddafi, the only thing that stopped them is the threat of a military action or a military action where we haven't seen a military threat in North Korea, there you had a deal, North Korea signed the NPT deal, that was worth nothing, there we had no military threat and when there is no military threat, those autocratic states are developing nuclear weapons and North Korea, the failed state and this is an understatement, is threatening half of Asia, threatening Japan, the West Coast of the U.S., meaning there is no substitute they're telling you, staying in the agreement, getting out of the agreement with or without an agreement, it doesn't matter, what counts is the existence of a credible military threat alongside crippling sanctions and if the threat is not working, then you need to be ready to act militarily as we've seen being done in several cases.  Now that was our policy. Now what did Bennett, Lapid and Guns do? They did the exact opposite. First thing they did, or even beforehand, Lapid attacked me. Back in 2015, criticizing me that I led the counter pressure that led to this military threat and crippling sanctions. He attacked that, he said to them destroying the relations with the U.S. He knows the U.S. that I don't, he said what he said. Unfortunately, this thing came into expression later on. The secondest current government was formed in a decision that is so outrageous that I can't even describe it. But Bennett promised Biden I will not conduct a public campaign against going back to a nuclear deal with Iran. Now I want you to know, the Obama administration and President Biden himself asked me, repeatedly, let's talk about the disagreements between us behind closed doors we will send. Our officials, we will talk behind closed doors. Why did they say that? Because they know that the public campaign over public opinion in a congress, in a senate is the only way to influence the policy of the U.S. And they couldn't care less about the officials or the experts who said anything. They don't really give you anything. If they're determined to do this agreement you can speak to them behind closed doors as much as you want. So they saw that the public campaign that I led in the public sphere, they realized they need to stop the agreement. Obama asked me, let's do it quietly between us and Biden in our first or second conversation asked me that too and I told them both, my government, our government will act differently. And this is how we act it. And so I must say that what surprised me is that not just they gave this amazing promise not to operate public opinion sphere in America that this issue will stay between administration and between officials. The U.S. does not work like that. If you can't influence the public opinion in the U.S. you can't influence the U.S. You just can't. Especially if the president and the White House wants to go differently what will make them to work your way and only that way influencing public opinion. In every democracy here too they do not deny that but they went even further. God even said Israel could leave with a new nuclear deal meaning we can accept a new nuclear deal meaning their lack of action and their willingness to adopt it and bow their heads head of the in face of the American demands brought about this new agreement. Not just they gave up the diplomatic freedom of action they gave up the operational freedom of action because Israel committed to a no surprises policy with the U.S. meaning that it committed to report to the U.S. before every operation any operation against a nuclear deal and they asked me that too and I never agreed to that, never. Sometimes I reported sometimes I did not but I made clear I am not committed to reporting to you because I have a greater commitment to the security of my people of my country to my army and I will do whatever it needs to protect us from the nuclear from the Iranian threat and this is why this issue is extremely problematic why once you agree not commit, not to surprise the U.S. you are in fact giving up your freedom of action because if the U.S. they did when we did not report but when we did report all of a sudden you read about it post in New York Times they read about it there goes your operation or the alternative they can simply say no don't do it and this is before you had an agreement so what will happen after you have an agreement and you will come tell them well I want to do this or that operation they will say you will ruin the agreement and this is why this thing this deal essentially gives the Iran immunity and you are destroying this immunity so this is why Iran is a route to nuclear power and Israel is a bystander what is our policy A powerful worldwide global campaign against the nuclear deal New U.N. In a media, in congress every possible form that has a say to rapidly speak against the nuclear deal to provide the infrastructure the background to another administration to get out of the agreement and to legitimize to a possible action on our behalf if and when we act against a nuclear capability second second not be a country or shadowed by the U.S. but the sovereign state we don't need any agreement any approval for any operation on our end not a state sponsored by the U.S. today I interviewed them on Fox well I did it for years spoke in numerous platforms media outlets Americans and elsewhere in favor of our right to act against the nuclear deal and everywhere and today not just not just in mongo-western question audience but also Arab audience in al-arabia about an hour ago and they need to understand while we have created more I would say would not have materialized unless that issue the Arab countries saw that we have an un unwavering determination to act against a nuclear Iranian threat so peace and security are all based on adopting a complete opposite policy of the current government to commit clearly as prime minister with an agreement or without it with or without an American approval I will do everything but everything to prevent Iran from obtaining nuclear weapons and one last word about another issue you probably remember the Iranian intervention in recent elections it was not reported by me but officially by Facebook demolished network that got involved in the elections and attacked Netanyahu now we're seeing another intervention from another direction from the Palestinian side this time around there's a secret meeting between Oden TV and Ramallah they met with the head of intelligence of the Palestinian Authority urging them to do everything so I will not get 61 seats so now it's obvious to you that they're not they're not worried about the interest of the citizens of Israel not even the Arab citizens of Israel they did nothing for them we did for them plans that deserve that the Arab public deserved and many of the Arab-Israeli citizens know that but here they met with a hostile element that is trying to meddle in the elections and we think it's extremely severe and we have filed an official complaint today and a request to investigate this issue I was worried when Gunn said that all I have no problem with the representatives of the joint Arab list yes of course because without them you can't form a government they have only 50 seats in parliament Gunn's Lapid with merits and the joint Arab list and Lieberman in a good day they reach 50 seats without the joint list and the wrong party they have no government all those challenges are major but I would like to conclude with a positive note of hope for the citizens of Israel we can overcome all of these challenges Israel's opposition leader Benjamin Netanyahu holding a press conference in Tel Aviv right now speaking about what he calls a bad nuclear deal with Iran a lot to unpack he was addressing a television audience in Hebrew with me to unpack what we just heard is our senior international affairs correspondent Owen Ultiman breakdown with stood out we know he thinks the deal is a bad deal talk to us about the message he was giving in Hebrew right now listen I think it was a great performance I have to say I agree with the content I think as a communications performance I thought it was fantastic part fireside chat part greatest hits all the pictures are those greatest hits moments of course right of the Netanyahu theatrics over the Iran issue I don't want to say that in a condescending way obviously those theatrics had an important role to play as Netanyahu himself said of course what can you do we are just about two months away from elections part of it political messaging I thought it was done in a cheap way I thought it was done in a substantive way setting up a substantive contrast between his own approach to this issue and as he described the other side's approach to this issue of the Bennett and Lapid government and setting up that contrast in terms of substance in terms of the explanation the fireside chat part if you will about why the deal is a bad deal again I thought very effective from the communications perspective those slides with a very very big text to a few central points I thought very effective again going back and explaining his own history his long history with this issue this is an issue that I think sincerely drives him in a way that other issues may not and that he sincerely believes in what he says about this deal and in how he has handled it and how he has managed it again one can disagree with the tactics and the strategy in terms of going into the US Congress a little bit curious to put the Wiley his longest moments in this issue and then the political messaging as well again I thought not cheap but setting up a substantive contrast between his own position his own approach and the approach of Bennett and Lapid they certainly will have their own arguments but why they don't want to create conflict with the Biden administration very eloquently I think outlined his arguments about why that conflict is worth it and is essential for Israel and Israel's approach to this issue and again laying out exactly not sure this press conference was the place for that other issue at the end about the role of these Arab members of parliament it to me seems like a separate issue and Netanyahu was simply having a hitchhike on the rest of the presentation I thought less effective but the presentation on Iran I thought extremely cohesive Netanyahu again reminding viewers of his great strengths and Benita across the political spectrum here in Israel Netanyahu's ability to advocate for Israel to be a spokesperson for Israel and the Israeli point of view for the strongest attributes this was a way for him to remind voters of that again just about two months until election day and show images of other interviews he's done on this day with American networks with Arabic