Hello and welcome. My name is Anita Kress, and I am the Webinar Production Assistant for Dataversity. Thank you for joining the latest in the Monthly Webinar Series, Lessons in Data Modeling with Donna Burbank. Today, Donna will discuss data modeling for XML and JSON. A couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the Webinar, and we very much encourage you to chat with us and each other through the Webinar. To do so, click the chat icon on the top right corner of the screen to activate that feature. For questions, we will be collecting them through the Q&A section. If you would like to tweet, we encourage you to share highlights or questions via Twitter using hashtag lessons data modeling. As always, we will send a follow-up email within two business days containing links to the recording of this session and additional information requested throughout the Webinar. Now, let me introduce our speaker, Donna Burbank. She is a recognized industry leader and expert in information management with over 20 years of experience in data management, metadata management, and enterprise architecture. She currently is managing director of Global Data Strategy, an international data management consulting company. Her background is multifaceted across consulting, product development, product management, brand strategy, marketing, and business leadership. She has worked with dozens of Fortune 500 companies worldwide in the Americas, Europe, Asia, and Africa and speaks regularly at conferences. So at this moment, thank you, Donna, for speaking with us today, and I'll hand it over to you. Great. Thank you, Anita. It's always great to speak at diversity. And so, as Anita mentioned, today's topic is data modeling for XML and JSON, which is a little different from some of the other topics we have had. I'll go through some of the topics later. Anita already mentioned me, and I think many of you know me already. So probably the most relevant to this discussion today, some of you know me from past roles, either on the ER Studio team, whereas director of product management, or with the Irwin team, whereas I'm a VP of product marketing. It's got to have a lot of experience in the data modeling space. I'm both on the technical and the strategic. And I wanted to kind of address that today and kind of some of the, well, I don't say do when it comes to XML, but you know, non-traditional, non-relational areas when it comes to data management. This is, let's see, this is the last, this year in the series. So if you haven't joined some of the others, they are all available on demand. And you'll see that we tried to mix it up a little bit in terms of topics. We started out with data modeling as part of your data strategy. We have things on data modeling and metadata management, big data, UML. And this one gets a little more in the nitty-gritty of XML and JSON at the end. Keep you, keep you wondering until the end what we have a whole lineup for next year as well, but we tried to do the same thing, kind of some bottom-up and some top-down. And then we tried to judge by the attendance what are kind of the more popular ones. So definitely, you know, if there is something by the end that you don't see in the agenda, do let us know because we try to always go by feedback of what's been interesting to folks. So today on the agenda, we're going to talk specifically about XML and JSON. If you're not familiar with that technology, what it means with things like data modeling and metadata for XML and JSON. And then what it means to integrate some of these technologies with databases. When I mean databases, it isn't always relational databases, right? So we have, you know, SQL databases as well. And then a little bit, which is still modeling. It's familiar with things like RTF and the Semantic Web and how things like XML like fill in stuff into that before you open it up to Q&A. But before I go too much further, I told you a little bit about myself and you heard a little bit about me, but it's always awkward on these webinars because I can't see you. And I do recognize some of the names and familiar faces, I guess, on the webinar, but some of you I don't know. So we have a couple of questions, just so I can kind of level set in the presentation. Anita, would you be able to kind of bring those up? Or Shannon, I'm not sure who's running in the back. Kind of the poll questions? Anyone? I'm going to run one at a time so we can answer. That's why I've got it set up. Okay. I don't see it. Should I be seeing it or do I need to click on it? There we go. That's the first one. Okay. The first one is, how familiar are you with relational data modeling or edited relationship modeling? Like ER, kind of the boxes in line. I don't know if we have every button there if you want to just give a vote and let me know. Shannon, there's a time. We have 22 seconds left. I'm sure we can show. I think we can all see the results when they're ready. And Shannon, have they opened up or do we have to wait longer? I don't know if I should be seeing something. Yeah, this is where it's closing. It's taking time to... Oh, this is where I sing? I wouldn't sing. No, whatever. Come back. We need the Jeopardy music. My assumption here is that this will be high because generally, and this was sort of my assumption building the presentation, a lot of folks on these data modeling webinars are data models, funny that, or data architects. And it looks like we were proven right. So most of you folks are pretty familiar with the relational world, which isn't a surprise. That's kind of the data versus the core and especially when we come to a data modeling series, a lot of folks are familiar with that. When we look into XML, I don't know if Shannon's going to pull up that one. Same thing. Are you, how familiar are you with the XML technologies or whether you've coded it, whether you've heard of it and don't know what it is? Be honest here because no one else can see your answers but you. We do not catalog these in any way. So I have my thoughts on this one, but I will wait to be surprised. So I was good to know. And I saw some folks didn't vote last time. They were not allowed. They must not see what they come out to this time. The poll is assorting. It's adding to the suspense. I would say, but really that no one ever come back. This is things that Shannon's sick. Shannon's a singer. If anyone didn't know that, I want the cat out of the bag. My voice is not there today. All right. It's a little bit of a mix. Some folks are very familiar somewhat. And then some are not familiar. And some are still shy and refuse to answer. So you shall be punished for that. The next one you have to answer. But that was my assumption too. I think a lot of folks that are familiar with relational modeling maybe aren't as familiar with XML and vice versa. So to bring up the very last one, Shannon, we're going to ask a similar thing about JSON. Is JSON something you're familiar with? Again, have worked with, heard of or not? So again, don't be shy. We don't publish these to Twitter. This is how somebody we did post to Twitter. Joe doesn't know anything about JSON. No, just kidding. We did not do that. Just more, you know, as I go through, I kind of want to know what people's backgrounds are as I ramble along. It's always good to know the audience a little bit. And on that note, I neglected to say on the intro slide, I will go back to that. If anyone is a Twitter person, there is a hashtag today. Hashtag Lessons DM. So if there's something that you're, you know, questions or comments or something from the presentation you want to share, there is a hashtag out on Twitter as well. And I guess this is a long, long call. They need a better data model behind so they can calculate the queries faster. I think there's some people on the call that could help with that. All right, similar. That some folks are somewhat familiar with it. Some folks is new to them. A few folks are kind of very familiar with it. So it's a bit of a mix. Not sure if the same people who are familiar with XML are also familiar with JSON, but don't worry, we will not have you do another poll. But thank you for doing that. It helps me a little bit just to make sure that we're, you know, hitting the right target. So see if I got it right. On the next slide, my assumption was that most people on the call today are familiar with the relational databases in ER. So yes, there are cartoons about data modeling. And this is one from my first book, Data Modeling from the Business with Steve Hoverman and Chris Bradley. Some people read Data Mods to the kids to put them to sleep. I think Steve Hoverman is probably one of them. I think he has stories where his child was in kindergarten doing data models. But he's probably the exception of the rule. But again, yeah, often on these webinars, there's a little heavy, heavy focus on data models, data architects, et cetera. So the example I'm going to give in this presentation have that bias, i.e., a comparison with the relational database world. And I know that's an assumption, but I went with it. So this is not in fine. If you want to leave, we have to hear this. But this is not going to be a deep dive into the technical aspects of JSON and all the different options of modeling it, et cetera. There are webinars on that, and it's a fine topic. I thought, as we put through this agenda, it would be a good introduction to some folks that are familiar with relational databases, but maybe not so familiar with JSON and XML. Or you might be familiar with them, but kind of want some thoughts or conversation on how they might fit together. So here we go. What is XML? So I'm sure you've heard the term accessible markup language. It's really used to store and transport data. It has a very simple definition. And it was really designed on the concept of simplicity. So some of the basic design principles of XML, when it was created, was this idea of simplicity. So the fact that if I can make it easy to use, if it can interoperate with everything else and easy to understand, then it's going to be taken up in the market. Well, it's lasted 20 years now. It's been around for quite a long time. I think it succeeded in that. The other thing is this idea of modular design, to do one thing well. And the one thing it is supposed to do is data availability, transport, and sharing. So that is its purpose. Don't overload it with other things. If you go into the bottom bullet, what it doesn't do is format the information. That's what HTML is for. It doesn't transport the data. That's where something like a web service is for. It doesn't necessarily design to store it with large legacy amounts of information. That's where you may use a database. So it is what it is. It does one thing well and it does it very well. The other idea is to make it extensible. So you can easily modify the structure and content and also self-descriptive. So not only can it be machine readable, but a human can read it. And if you open up an XML file, I'm sure you can basically understand it because it has these things called an embedded descriptive tag. And we'll talk more about this. This was a pretty big deal, especially when it first came out. Think of things like object-oriented programming or some of the designs at the time where you're mixing the code still around, but mixing the code directly with the data or things like the mainframe where it's a big top-down design. This was pretty revolutionary. And I actually, in preparation for this, did some research that we'll talk about in a bit. But the big idea here is that it exists with things like data exchange. So if they give a B to C or a B to C, I have a company sending a purchase order to another company or you're buying off the web, right? Or I'm a research organization and want to share information. It's that way to be the ubiquitous exchanger of information. Which, again, we take for granted so much now, but it really was pretty revolutionary at the time. So if you were on my big data webinar, I talked about the same theory in a slightly different context. I'm sort of a fan of it, so bear with me. Because it changed in the way of my thinking. So if you're familiar with this idea of emergence and philosophy, not to get philosophical, but I tend to do that on some of these webinars. So there's this idea of the fact that complex systems and patterns can arise out of a multiplicity of relatively simple interactions. So let's think of a snowflake, right? There's all these different facets of a snowflake or crystals that form together to create this larger thing that we pretty much quickly look at and say, okay, that's a snowflake. But no two snowflakes are exactly the same. If you look at the snowflakes in the picture, they're all vastly different. But they're all made up of similar components and religion aside. There was no one person that decided that I'm going to design a snowflake. They just sort of appear, right? And this bothers me, actually, it did bother me. In my previous presentation that you may have heard, so I won't go too much into it, I think it's comfortable for us to say if we could just design it and we understand the world with something like the periodic table of elements and we could organize it everything as well. But the more we look into some things with nature, it does seem to be a chaotic mix of things that mesh together of trial and error and create beautiful things like a snowflake. Well, the idea behind the internet, really, or things like XML is very similar to that. Just do one thing well, I share data. When you think of something like the internet, the ability to share data across these disparate systems, across the globe in a very simple way is actually a big deal. So I think I mentioned when I went back to my presentation and was doing some research for some of my books, oh, from, like, 1990 something or early 2000s and they were talking about the revolution of data and this idea that you could actually order something online and that shipped to your house. Things that we're doing now and don't even think of, you know, and you could thank things like XML because it is very modular and it helps something very simple to do that data exchange across organizations. So it was also kind of cool because the way they read was very similar to some of the things we're saying now in the press because we like to make big deals about things. Internet of things or big data and, you know, they were right back then, you know, a lot of the stuff that they say was going to happen did. So who knows what's going to be here 10 years from now when we do a data diversity webinar on the new topic. But it's just kind of a different way of thinking of this modularity of data to share. So back to XML itself. So a little different than you might think with a relational database. It uses a hierarchical structure or a tree structure. So here's an example of one where this might be, you know, a purchase order where you can see. Again, you may never have seen XML before, but if you haven't, you still could understand it. So funnily enough, there's a ship to address with a name that's John Smith who has an address of city in a country. And it's sort of labeled with these different tags. So there's different nesting. So, you know, the ship to address, that's going to be your root element. And then you have these child elements. If you can almost think of that apparent child relationship similar in a relational data model. The other nice thing about it is this extensible. And for some of you folks will talk later about when you think of things like document databases, there's a similar theme here. So the idea is, you know, I have this ship to address, but maybe I didn't know everything at the time, right? So I have a name, address, city, but you'll notice there's no state in that. So the one on the right, we've added a state so that I know that this is Boise, Idaho, and not Boise, Utah. The nice thing is that any programs that have been looking at that, the original version will still work. You can sort of do this additive thing. So, you know, it's not like you have a relational database and you need to get the DBA to add the column. You know, it's frozen cost to each. But that is one of the nice things about it that is extensible because you're basically matching on these element names. The other nice thing is that it's self-describing. And I say sort of because I'm a metadata and data modeling geek and we'll kind of, you know, clarify that, but it is. I mean, the nice, the one nice thing is it's human readable. You can open up an XML file that's in text and basically understand that this is a ship to address with a name address city country. You know, fairly self descriptive. And they do describe the content of the element. So, you know, if I didn't know that John Smith is the name, it tells me, you know, the city is Boise. It's pretty obvious that's the city. But if we're true data modelers, which many of you on the call are, it doesn't provide the full metadata. So what's the data type of Boise? Does that need to be a string? Is it a character 10 field? You know, addresses that broken up. What's a numeric field, et cetera, et cetera? What's the business definition? What do we need by name? Is that customer name? Is that the ship to name that might be a gift? You know, what's the business definition of these different fields? You know, his name required all these kind of things that you might have in a traditional metadata tool or a database that kind of were these things or data model were these things you defined. It isn't. But, you know, we'll talk about this later. That's not really his purpose. It's just sharing some, you know, quick data across. But it does not fill that full metadata around it. It does have metadata. You don't need to, but people do. And it's a good best practice. You know, similar to DDL, you have what's called the XML schema or an XSD that helps define the structure and format of the data, which is basically metadata. So a little different. When you look at the data, which I'm saying is actually the XML file here in the middle as I kill my presentation. In a way, that could be a little bit confusing because you're saying, well, that's actually some metadata in there too because you do have kind of the name address, et cetera. But that really is your raw data. The metadata on the left with the XSD that helps you kind of define the structure. So you can say here that a name is a string type and a city is a string. But you're kind of noticing a lot of strings. It doesn't really have the full data typing that we may be familiar with in the data modeling world. And then the other piece of the data piece is that there's actually an order shipment here. That's actually what you're doing here. You have a shift to address for this person that you're sending some data to. So a little bit of the different information between data and metadata in these different fields. Okay, you can also show it in a model. And maybe I'm the same way. I think we can be sloppy. When I'm talking about a model, I kind of want to see a box in a line. I tend to be a visual person. And I like to see physical diagrams of things. And it can do that as well. So here's an example of an XML schema on the left. That's almost like your DDL that you might render into an ER diagram. Similar thing. So this is a web description of a literary work on the internet. So your resource could be anything. It could be a web page. It could be a document. It could be a blog. So you have the resource here. And then with this graph, I don't know, on the left it's fine. You might be a text-based person. You can read through that and see that you have... I can't even seem to grab it here somewhere. You're pointing it to it. There's a resource, right? And you can go through and see that resource has a title and a URL in the description. Okay, to me, that's a little... It has an information and that's fine. To me, that's a little hard to read. I like to see it on the right, where you can still say that there's a resource that has a title and a URL description with an author that it might have a first name, last name, et cetera. And it can parse it out. And there's different tools that can do this. I definitely don't like to talk too much about tools to keep this better and new cool. But just because people will ask. There's kind of two schools of thought there and we'll get into that. Maybe it would be a good idea to have the next slide. Nope, I'm going to knock the next slide. There's some tools that will focus specifically on things like XML and JSON and do that fairly well because you're modeling the XML, which is hierarchical, right? So when you're trying to... There's also ER modeling tools, your typical entity relationship modeling tools. They can either import or export XML, but there's always a compromise, right?  So, yeah, there's a UML class and there's a relational table and there's an XML schema and there's enough information. Maybe it's the 80-20 rule. There's some core information that you're still talking about customers with a name and address. Could we somehow reconcile and rationalize them into a single source? So that's a bit what some of the data modeling tools do. They're going to try to mush this into a relational table, which is just not the same thing, which is not good nor bad. That's just what they're doing because that's what their tool is. To be fair, tools do a little bit of both. But when you think about this, this is almost a one-to-many relationship with the resource being the one and then the works being in many. And there's different ways to model this back and forth, but it doesn't go so well the other way, right? Could an author be the author of two resources? You know what I'm trying to say. I mean, I sleep last night, but it tends to go one directional with the parent being the parent of the hierarchy being the parent of the relationship and the ER tools tend to do it the same way. There's different ways you can do that, but that tends to be the assumption. Okay, so the XML schema is good. It's good because it can tell you the schema but without having the data and it tells you some basic physical structure. But did that kind of hit on before? Is that name field required? What's the definition of a women by name? Are there code values when you might mention that there was, but I have Boise, Idaho. Do they put Idaho? Do they put ID? You know all that stuff we're used to in a nice data model or a relational database they don't have or can use a complex data type, right? Not just string. So I want to kind of talk about this and I have shown this in other slides too, this kind of idea of a pyramid. There's different levels of data modeling. So the XML schema can just define physical data, the physical structure of, you know, it really does not have much business or metadata. So you're not going to have the business rule behind, you know, how the, I don't know, prices counted or what the business, what a customer is and how it relates to account and kind of customer have more than an account. It doesn't have that. That's really where you add that logical or even conceptual layer on top of that. Doesn't mean it's bad, but that's not what it was designed necessarily to do. And that's where it is helpful often to integrate with something like a fuller data modeling tool or metadata repository to kind of add that on or source your XML from that even better. So this is some of the context that's often lost. And yes, there's yet another metadata data modeling cartoon, right? So from the same book. And if you've, you know, done any development in the past, you might get a giggle out of this. I know it's a data modeling cartoon, so I'm not expecting a full laugh, but maybe a giggle. Okay, we're almost done with our user acceptance testing and everything looks great with this new marketing application. Just one small question. What's a customer, right? And we can laugh at that, but I have worked for and worked with huge organizations that have not made that distinction and made very sensitive and embarrassing mistakes, like sending renewal notices to prospects who haven't brought the product, or I get this all the time from my bank, right? And I get a, I have a credit card, and then I keep saying, get this new credit card at a lower rate than I already have, right? So they can't figure out that I'm already a customer and they're sending me a mailing notice to a prospective customer, right? So all the stuff that we spend a lot of time with in metadata management and data modeling, that's just not what XML necessarily has. So you can see some examples. Can a customer have more than one account? What kind of customer is this? Do we assign the ship to address to the customer, or is that maybe a sign to the account, right? All these kind of things, valid values, that's not going to be in your XML file. Because you can almost see the XML file filling its eyes. Dude, that stuff isn't my job. I'm just sending the purchase order, right? Sometimes it's good to remember that. What XML was defined for, and the beauty of it, and the power of it, and why it has become so popular. It is really a data exchange. So I'm trying to send the purchase order from company A to company B. I'm trying to order something online. This is very good. So that idea of modularity and simplicity is the sort of power of it. So it comes with pros and the cons. It's just how you use something for its design purpose. So that is why often XML is used in conjunction with a relational database for that sort of permanent storage and integration. Or if you want to have that with your operating system, or your operational, or your reporting, or your reference data, you could, as I mentioned, often export from a racial database to XML and vice versa. Because they're fit for purpose things. Here's one way I'd kind of talk about this a little bit, but you can. You can take an XML schema and generate EDL from it. So you can see here in the example that if I have the resource table, I can create a table called resource. You'll notice that they're all text shields right now. Because again, you have that very limited data typing in XML. It will create an author, et cetera. I'll say you can. There is a translation with kind of caveats that I had mentioned earlier. And again, some comparisons between what we might say an XML diagram with its sort of native modeling XML as a hierarchy. And then some of the compromises that might be made when you put that into a relational model. Then you have to make some design decisions. In this case, it is sort of doing that. I have a resource and an author and you can have more than one. But there's some design decisions that can be made. It isn't necessarily a direct translation. Because again, they're designed to be different things. One's hierarchical and one's not. Okay. So a little bit about XML. Hopefully that was some thoughts that were helpful. When we talk about JSON, some folks think of it as a more modern version of XML. Some people have a slightly different use case. We'll leave the jury out on that. But similar, it's a JavaScript object notation. And it's a minimal readable format for structuring data. And again, primarily the transmit data between the server as an alternative to XML. It has some other use cases I've seen as well. But we'll talk a little bit about some of those. You'll see an example on the left is an example of the same thing. We have a list of our employees. So we have Shannon and Anita and Tony. You might recognize those names. So it's the same information one showed in JSON and one shown in XML. So you can make your choices. I tend to think JSON is a bit easier to read here on the left. It's just more concise. If you're programming world, obviously it comes from the JavaScript. You might be more familiar with that. And I was a programmer too. So I kind of like the look of that. One of the things that can be very annoying when more than our XML file, at least gets huge. Just think how long this took for these hash tags. I mean, the element tags. This is the way they're broken out. But similar to XML, it is sort of self-describing so that you see that this is the first name and the last name right there with the data. It does have that same hierarchy. And it is simple and interoperable. So it still has that same design. It is different. You can do things like arrays. You can, you know, the idea that it can be parsed with things like JavaScript. And as I already mentioned, kind of design-wise, it can be seen as simpler and shorter to write. That's always difficult to say because everyone has their preference, right? You can look at the one on the right and say, no, I like that because you can see it. It looks almost more like a table. It's more structured. Again, there's different ways of doing things. I will let you decide. I should take a poll for that. See who likes which one better. Okay, so similar to the XML, there's this idea of a schema with JSON metadata. And here's an example. You might have had a JSON-based product catalog. So you have sort of an ID, a brand, a price, and kind of an optional set of tags. I see it is a bit more advanced than some of the things you can do with XML. So here on the left, you can have your data, right? And that can be, look if you can do it, folks, because it's kind of a mix of metadata, too. You can obviously see this is an ID, a brand, a price, and you're going to do things like tags, like a metadata tag, right? So that I can say that this information has to do with camping and sports because it's a super cooler, maybe something like you put water or beer or something in. But still, you don't have the full context, right? You could say, well, can the ID contain letters? Well, you'll see here that the typing is a little bit more advanced here. You can say, oh, yes, it's an integer. Whereas the brand is a string. You can put things like description here as well. And you can do things like it's just required. So yes, you can. The brand's ID is required. So a little bit more robust in some of the things people traditionally do in XML, which can be very nice. So one of the use cases that's become popular with JSON is this idea of a document database. They often use the JSON documents to store their records or the collections in their databases. So one of the rise in the popularity of things like a document database is that idea of the ability to store unstructured data in a flexible way. So if you think back to the idea of XML, it's extensible, right? So in one version, you could have name and address, and in one you could have name address, and it's not going to break things. So similar idea with document databases. And one use case might be, I'm a museum, right? And I have all this stuff about China. So one says I have this nice beautiful vase or vase, have you seen, that comes from China. And I could do a search on that. I also have some great books in the lobby that have some information about China. And you can see that that field matches. But everything else about it is different. One's an artifact and one's a book. One has things about what medium it's done in. Is it ceramic? Is it cloth? Is it paint? The other one has a title, which may not make sense for other works of art. So that's one of the beauties of a document database and why they have become more popular is that idea of the kind of flexible schema. And Jason can do very nice things with that. So that's kind of why those things fit very nicely together. A little bit of a jump, but not too much. So jumping back a little bit to a different topic is this idea of the semantic web. So there's this idea of the resource description framework, RDF, that really helps drive a lot of the things in the web. So back to the earlier point where I had the internet interweb of a thing to thing and how XML sort of shared that the data exchange between them. Well, that's sort of thinking of a server to a server, kind of sharing data between them. The goal of the semantic web, and we should like this as data people, is that you're trying to share data so that it's data to data. And they're trying to make, and this is kind of that second bullet here, goals to move from a web of documents where you have URL, and you just go URL to URL to a web of data so that you can actually see what's in that URL and add some semantics and some contextual meeting. So I guess you can think of it as a data model of sorts, a very simple one. So you basically have subject object predicate here on the left. And it could be something like ACME Publisher is the publisher of RDF is easy. And it's really limited to that because you're trying to see basically just the basic connections between them. So these are sort of, if you think of graph databases, related to things, and trying to see connections between them. And we'll get into some things like vocabularies. And there is a vocabularies, friend of a friend. So you can see who's related to whom. That's kind of the power of some of this. Well, there's certain serialization formats for this. And you'll see that there's an RDF XML, the common one. There's also a JSON, right? So when you're thinking of the exchange format, this is a very good use case of where things like XML and JSON are used. So there are things like vocabularies or schemas to describe what's in that to you make free words, things like the Dublin Core or schema.org. And again, this is adding some of that semantics and context around the very web itself. And again, this is an idea of creating a web of data rather than a web of documents. And this is an example I've stolen from Data Diversity so they can sue me. No, I was kidding. It's actually out there. You can do it yourself. You can go to the website and right-click and save you source. And you can see some of these examples yourself. So here's an example of, OK, so last year we had the Enterprise Data World Conference. Hopefully some of you went there. And it was at a place, right? It was at the Sheridan San Diego Hotel in Marina. So what the code on the left is saying, this was taken from schema.org, one of the vocabularies. There's a type of thing called a place. What are some common things about a place? It's going to have a name. It's going to have an address, a postal code, et cetera. And those are some common things that define placeness, if that makes sense to a modeler. Well, at that conference, and I was lucky to be there, I took a picture in the upper right. Won't win any awards for that picture. But I took a picture and say I posted this out on my website or Twitter or something, right? There's certain tags around that. But if I had this on my website, I could put some of this code in there to define. This one is not real. I made this one up. The one on the left is real. You can go look at that. I could say that there's a type of a thing in this website called a place. And you'll see there's some of the same field, the name, address, et cetera. And so why is this important? Maybe I'm trying to find out everything that's happening at this certain place here in San Diego, Marina. The place stays the same here, right? What I'm interested in is the place. And I know that there's a conference at this place. And I have an idea what it looks like because there's a picture about the place. That's making it a web of data, right? So you could have had that this is an event. And you might have wanted to know everything about data events. And that might have been a different type of data, if that makes sense. So it's sort of giving meaning to what's on that webpage. And then you can do sort of smart queries around that because you're saying, tell me everything about a place. Tell me everything about speakers. Tell me everything about a book that might be referenced on that. And the authors of that website can kind of put that in there to help a help their search results and help provide more information. So you may be familiar with Dublin Core. And that's a common set of metadata standards for things like media and books and that kind of thing. So it creates this kind of standard for having a title. What are the things about a work of, I don't say art, but a literary work that has a title and a subject and a description, et cetera. And again, you'll see that some of the resources can be described in things like XML. This example on the right is not an XML, but it gives the idea of what it is. So what do we do on the internet? We post pictures of cats. So I wanted to make sure my picture of a cat got out there. I could create a resource description and kind of say, well, here's the title of my favorite cat video, the subject is cats, a quick description of it, that it's a video and not a book, what language it's in. I guess it's my cat speaks English. It would be in English and publish it with cats online. So it just gives you a little more information about that resource.  Schema.org is another popular one. Webmasters can use to mock up their web pages. So if you haven't heard of it before, you should at least be impressed by some of the folks that have. So it was actually created by a group of some of the search providers, things like Google and Microsoft and Yahoo and Yandex. And these vocabularies are created by kind of an open community process. So there's a GitHub project out there. There's also a public mailing list through the W3C. So again, they create these types with certain properties in there. Again, you can describe them using JSON, RDFXML, et cetera. So again, what type of thing am I describing? Is it a place? Like we gave the example of the Sheraton in San Diego. Is it a type of person? Is it a creative work? Et cetera, et cetera. There's also certain extensions for certain industries. Health, life sciences, auto, et cetera. Because certain industries, again, if you think of the main purpose of things like XML is sharing between B2B and B2C, you probably want to have some really shared attributes and some commonality in your schema if you want to be sharing information. There are other common schemas and vocabularies out there. Again, Dublin Core and schema.org are probably the two popular ones that come up if people are familiar with this and mentioned some. There's actually a site, the link to open vocabularies, and they kind of do a heat map of the different helpful listings of them. So I mentioned earlier, and there's a website here if you want to look. The friend of a friend, that's again linking people to people. So when you think of social networking and that kind of thing, you can help with that as well. So in summary, XML are used for transport and interoperability of data, and that is what they're good at. And that's important to remember because they don't do a lot of the things we're looking for in a relational database and vice versa. So they are simple by design, and that's what's good about them that also can be frustrating about them if we're familiar with products and have a lot more richness around things like metadata. They do other things like this idea of simplicity, modularity, and extensibility as well as being self-descriptive again. We can, as a data modeler, open it up and say, oh, they just addressed. They don't say whether that's mailing address or PO box or, but we knew it was an address. That's a lot further than we would have been if we didn't have something like XML that was going to share it. You can integrate with some of these databases, either translational with some of the compromises we mentioned to a relational database, or at some of the core storage for some of these no SQL databases like document database. Whether it's relational or XML or JSON, you can still do graphical models. And that might be something folks aren't familiar with. I think often, especially with some of these, you're used to seeing text-based format, but there are tools in the market that do that. And for me, personally, as I said, I am a visual person, so I like to see it that way. It kind of makes a lot of sense to me to see it. It's kind of a use case of some of these. You have a semantic web that turns the web from a web of documents related to a web of data and has semantic meaning around it. A little bit about me. In my company, we kind of specialize in data design and how that kind of helps businesses do what businesses do best. Here's me. If you want to contact, I would do it with Shannon, and I need to post all this information. So if you do want to contact me for any reason, after that, please feel free. A little bit of a plug. We do have a metadata management course that goes deeper into some of these topics, which is available through the university. And another shameless plug is for our lineup next year. So we have already planned a full list of lineup. Our very next one is on January 26th, where, again, we try to do a mix next time. It's going to be talking about enterprise architecture and how that data modeling fits in with that. You can see throughout the list there's a whole wide range of different topics. Hopefully you'll join us for one of them. So without further ado, I will open up to Anita or Shannon for questions. Great. Thank you, Donna. Well, to answer the most common question we get, we will be sending a follow-up email within two business days with links to the slides, links to the recording of the session, and anything else you requested throughout the webinar. If you have any questions, put them in the Q&A. We'll be working through those, and I'll give you a minute to do that. And in the meantime, I think we have a few. Donna, related to slide 14, do you know if Irwin supports the XML modeling that you showed there? We'll be talking about it. So I'll make sure I'll go back to the right 14. Well, in general, I don't generally like to talk, but I think it was one of the ones for that. So in general, I would just say I don't like stock products, but they generally do more of the translation model than showed. They don't show a native diagram of XML. They're going to translate that into an energy relationship model. So Irwin is one of them, because they're the group in the ER world, they don't show native XML models. They're going to do that translation so that when you import XML, it's going to show up as sort of an ER rendering of an XML file, if that makes sense. Great. So I'm not seeing any more questions at the moment. I see one, and Mary's more of a comment rather than a question. So in my view, XML and JSON, quote, modeling doesn't make any sense when we model something. It depicts a logical concept of the solution under consideration. More likely you're referring to diagramming, which is not the same thing as modeling. And wow, that's kind of a lot of different things. Well, I think that we could be a modeler and say, how do we define a model? Right? So there's several different parts of modeling. So one is the very physical nature of modeling. I would say modeling is the structure of your database and the format. If we think back to that pyramid that I have, this is a logical conceptual and physical modeling. I think JSON and XML modeling do an excellent job of that physical layer. And if you're thinking about creating a data structure, which maybe 75% of a lot of people doing data modeling, you're using it just for that. So I think in that sense, they're great. I think there's some also some design decisions of how you do that modeling and what makes sense and how you break up the fields. So in that sense, you're doing some design. If you think of modeling as terms of that, I would agree with you. It's not necessarily a thing of modeling and the kind of conceptual logical doing all the relationships and integration. But again, that's probably not what it's designed for. So I have some agreement. It's probably some clarification in kind of my view. But I would also agree. Some people think of modeling as diagramming. And I did want to kind of make that point that if we think of modeling as diagramming, which some people do, I mean, when I think of it as diagramming often because it's easy to kind of do that design in pictures, you can do some of that with XML and JSON. I want to let people know because I think a lot of folks kind of think they're stuck in a text editor when they're doing that. But there are solutions to that. Great. Thanks. Are there any other emerging standards that might replace XML or JSON? I am going to defer that. I think those are two popular ones out there. I'm not following some of the other ones that might be. I am reading another one where it said, you refer to XML. In my company, we create an XSD, which is called a message format that is shared. How does JSON relate to this? So XML is going to be the format. XSD is kind of the schema. And JSON has a schema as well. And then I showed a picture as well. So they have their version of that as well. Sorry, I keep jumping ahead of you. You can't trust me. I get excited about a question. There's a question. Where can collections and field definitions be captured? Are there better options than Excel? I would say that's a good use case to use. One of these modeling tools. So self-model natively can track some of this interface. And I showed the JSON schema. There was a field where you can use definitions. And sometimes, depending on your use case, if you are integrating with things like a relational database, you can store them in kind of a relationship model. So if you just... Also, there's things like metadata registrants. When you're thinking of some of these vocabularies, that is often where some of those, you know, on the web, where some of these are posted as well. We also have a question about healthcare. I'm not sure. Do you have any thoughts on FHIR? I am not familiar with that. So I am going to defer on that one as well. Okay. And then what is the best practice to translate from relational model to JSON format? Ooh. That's probably a bigger question than just this one topic. I would go back to your use case. I mean, I think if it's already in a relational model, it's probably there for a reason. I wouldn't necessarily start there if I'm building JSON. I guess it was my rambling way of trying to answer that question. But often that is where it is. So one of the things is what's kind of different with those is the idea of the relationships. So you're going from an ER to a hierarchy. So you want to think through those and see what makes sense. Actually, there's another day diversity webinar that went a little deeper. Sometimes it makes sense to break them up into separate documents. Sometimes you want to break it up into a hierarchy. Maybe a general thought of giving some thought to that is that relationships are just handled a little differently. You can have separate documents. You can have embedded hierarchy sometimes that has to do with performance, readability, et cetera. So a couple of thoughts there, but probably a deeper webinar to do on that one. Okay. Are you seeing many people transform your models to JSON-LD or most just starting with JSON text? I am seeing a lot more uptake right now from folks to going from relational data models to XML. I think JSON is a little not as... I almost think there's almost two camps. I think a lot of folks in the programming world are a little more comfortable with JSON. So I have not experienced too much going to the JSON-LD more of the other. But that would be my answer to that. So I think that was fun. There's another one. The programming tool. I'm sorry. I'm jumping ahead of you again. That's okay. I don't want to answer that because that's kind of a vendor plug, which I don't. But there are. But yeah, I would just... I guess my general comment on that is that do take a look. There's some that have been out in the model. There are some categories. There's some data modeling tools that can export out. There are some native ones. And with the native ones, there are some that kind of grew up in the XML world. And they're adding JSON. Some are native JSON. So there's also a difference. And most of them are kind of start-up mode. Some are actually getting this idea of a semantically as well. Often they do kind of focus almost by defining a necessity on this physical layer. Take a look. Some of them actually have this idea of adding the semantic layer as well. Okay. Can XML JSON data modeling replace ER modeling completely? And if yes, in what cases could that happen? Oh, I don't think any technology replaces anything else completely. I think as I mentioned in the call, they're kind of different use cases. So I would see XML more as data transport. I think ER modeling is good for ER. You're looking at kind of acid transactions, right? That's what ER modeling is good for. So this is probably more succinctly in other presentations when I was thinking more clearly than I'm not doing today. ER modeling is good for what ER modeling was designed for. I want transactions that are linked. And I have the accountability for that. If you want to use something like JSON, the one example I gave was things like document databases, which is a very different use case than your traditional ER model. So often they're used to kind of complementary. I wouldn't say any technology is going to replace. I would probably still put my accounting system on the ER system and the relational database, a lot of data warehouses. There's a lot of good reasons why folks are doing that because of the rollback, because I mean a lot of the great things relational databases have. If I'm trying to do something behind a website or have a lot of multimedia, something like a JSON based document database is an excellent thing as well. So right now, no, I don't think there would ever replace ER modeling completely maybe as things mature. But I kind of doubt that because they're really, right now, different designs. But that said, what we're seeing in the market with even big data databases becoming more, having relational features and more relational databases, adding some big data type stuff, things are merging. But I guess my general thing is the big thing of the use case for it, and I wouldn't say any. The vendor tells you they're going to replace everything else. Just give some thought behind that of, you know, often these technologies. A lot of the big companies I work with have heard of the different use cases that you're using it for. Yeah, ER modeling does its place in many, right? Great, thanks. Okay, sorry. Go ahead. I was just going to wrap up. I was going to say I could do the email if they are any others, or did I miss them? No, I'm not sure that you missed any. Oh, hi. Well, thank you, Donna, for another great presentation on QA. I'm afraid, you know, it's all the time we have for today. So just to remind everyone, we will be posting the recorded webinar and slides to DataVersity.net within two business days. And we will send out a follow-up email to let you know the links and other requested information. Thank you again for attending today's webinar. Thank you, Donna, for another great presentation, and I hope you all have a great day. Thank you.


Hello everyone. This talk is titled 10 more ways to blow up your Kubernetes by Airbnb and this talk is titled 10 more ways because we have found more ways to blow up since last year's talk that was presented by our colleagues Melanie and Bruce. So, who are we. My name is Jen. I'm a software engineer at Airbnb on the computer infrastructure team. This is a team that manages our Kubernetes clusters and manages the abstraction we have over it to make developing services easier. Hi, my name is Joseph, and I'm on the compute info team with Jen. I've been on this team and at Airbnb for about eight months now. So the outline of this talk will be one, we're going to give a brief intro of Kubernetes at Airbnb to we're going to dive into 10 cases of how we actually mess things up. And three, we're going to go over and do a recap. All right, Kubernetes and containers at Airbnb. So Airbnb started our migration to Kubernetes from the beginning of 2018 and since then has really taken off Kubernetes is actually running in production and we rely on it is a critical part of our of running Airbnb. Our environment is mostly Amazon Linux to for the actual host. We use into images. We use canal for CNI plugin. We do use no port services and and smart stack for service discovery. And we support many languages such as Ruby Java Python and go services. Now let's dive into the real fun stuff. The first case we're going to talk about is called replicating replica sets. So for context, we have a job that regularly scrapes our clusters, and it runs a command something like cute detail get pods on namespaces. This job starts getting out of memory errors and one specific cluster. So let's check it out. Here we see, we see that in most of our normal clusters, we have maybe on an order of 1000 to 2000 replica sets, any guesses to how many this problematic cluster had. It was 56,000. So let's see what else is up with this cluster. When we dive into all these replica sets we notice that all of these are actually under one deployment one name space. As we dive deeper into it, we also notice that the deployment object, when we check its collision count, it was up to 100,000. And so right now our theory is likely that the deployment object is creating replica sets that aren't getting adopted by it. So it keeps on creating more. As we dive more into this incident, we realize that something different about the deployment of this job on this specific cluster that was different from its deployment of other clusters was this one we were actually testing this new feature called topology spread constraints. So we noticed that were that these specs were actually not getting picked up by the replica sets. And the theory here is because the replica sets being created by the deployment didn't have these specs. The deployment didn't know that it had already created these replicas sets. It didn't know that these replicas that belong to it. And so it would just keep on creating more and more replica sets. And good thing we have so many, we had so many test clusters, so we can actually like diff the specific deployments to see what was going on. And in this specific incident the actual fix was simply just deleting the whole name space, which is fine because this was just a test cluster. And good thing we were just testing this feature out in our test cluster. And so the takeaway here is do you test new features and test clusters because for us that's what they're for we were testing this new feature. And what we noticed was it kind of blew up our cluster and just kept creating infinite replica sets, but not a problem. We know to investigate more before we roll this out to production. Our second incident is called mutating time bomb. And what we see here is, we have the service expressing higher rate and the service cannot deploy. And we get this cryptic Kubernetes error saying unable to access invalid index five. So we dive deeper into it and check the status of it and what do we see, we see that to two pods are running at what HPA thinks there should be nine replicas running and the service is panicking because, well, we only have two pods left. And if we dig more into it, we notice there's only one deployment object, but three replica sets, which is really weird. And the replica sets that actually have pods are 16 days old. As we dug more into it we notice, yeah, there's something really wrong going on with the HPA here and the replica sets. Later, whatever we tried, well, we tried deleting the HPA and manually scaling it up because the theory was perhaps there was something wrong with the HPA. That did not work. We tried deleting those old bad replica sets, thinking perhaps it was those old replica sets that were blocking new deploy from happening. But that was not the problem. We also made sure not to delete the replica set that had our last two pods running. We also tried creating a new deployment object just to update it just to kick it and see what happens. That did not solve our problems. And we thought about perhaps just recreating the namespace, but that would definitely delete the last two pods running, which is no good because this service is production. So what do we do now. We page a teammate, we call a friend one hour later. And so when we started digging more into it, we come back to the first page where someone had called out this cryptic error message. And we should have done this earlier where I should have dug more into it, because we realized that this was actually the exact problem. When we googled for the problem we noticed it was actually a problem with JSON patching and related to our mutating a mission controller, which was recently rolled out in the last month. Specifically, the bug was this with our mutating a mission controller. Imagine that on the top left is our input and this is our spec and our desired output is the spec at the bottom left. Here we see we are deleting two items from an array. What the bug was the generated output is this. It generates two operations. It removes index two and removes index three. When our first class glance of this patch looks reasonable, but the problem is after the first operation, the array doesn't have an index three anymore. So the correct patch is you actually remove index three then index two. And luckily this issue was already well known, but we googled for it and we can actually apply it. And so once we realized it was a mutating a mission controller to stop the bleed, we actually deleted the mutating a mission controller from the specific cluster, which immediately resulted in successful pods coming up.  So we're done, right. No, as we dug into it more we realize our change had actually happened seven days ago. And we realized that for some deployments over the last seven days, new code actually wasn't even being deployed. Services were being kept alive by their old replica sets, but over time as pods of the over because set slowly died off services were slowly getting degraded. Yeah, so this problem was way more insidious than we had realized. And the takeaway here is be aware of the existence of mutating a mission controller. This was something we had recently wrote out and so for many of us, when we see an incident, it doesn't immediately come to mind to think hey, it could be the mutating a mission controller be willing to ask for help because paging our additional team member to help us definitely got us moving. So we do load the error messages and do create alerts. When we dug more into it later during the post-war and we realized we actually do have metrics that could have detected this problem earlier. All right, our next incident is called one to auto scaling. So we're going to begin with this Kubernetes issue. This is still an open issue. The gist of it is, let's say you have the situation where you have a deployment with three replica. Later you set up an HPA or horizontal pod auto scaler with a much larger number. You then update your deployment and apply it. But this is where the problem happens. The moment you apply that and or update that deployment, the number of replica resets down to three. So the suggested solution outlined by comment was remove replicas from the last deployment and remove replicas from the current and future deployment to let the HPA do its thing. And that sounded good to us. And so we ship it. Great. So our fix right now is if there's a deployment and HPA, we followed the advice of the comment by editing the last deployment and deleting the replica account from it, removing the replicas from our current deployment and applying the deployment. But that was not the end of our auto scaling woes. So this next case is called zero to auto scaling. So if you actually look at the HPA algorithm, this is the algorithm where desired replicas is a function of your current replicas. Some of you might already see the problem. If current replicas is zero desired replicas will always be zero. So this is a sounds like a quick and easy fix and we ship it right. And so now our fix looks like if deployment and HPA, and if replica is not zero, then we apply that fix. Otherwise, if replica was zero, we can actually just leave it alone and the HPA will do the correct thing. Zero to auto scaling again, but that that was not our final problem with the HPA. In this incident, someone alerted us when they when they did a deploy their number of replicas actually dropped to zero. So let's see what happened here. Someone followed up and realize, yes, this is actually a known issue which we had already fixed, but we were actually migrating our deployment services to Spinnaker. And Spinnaker did not have this standard of work around implemented yet. And so the lesson here is migrations are hard. And so now the fix is, oh, and remember to copy this logic to Spinnaker. And once more, we have more auto scaling issues. So what do you think happens in this scenario, you have a service that just migrated to Kubernetes and still has old EC2 boxes running. So we deploy this service to Kubernetes with replicas equals zero. We then scale the service in Kubernetes using kube kuddle scale deployment. And nice. We realize that Kubernetes is now taking traffic successfully. And so we delete all of our old EC2 boxes. And so now the service has fully migrated to Kubernetes. And now we deploy again with HPA. That's right. And incident happens. The actual bug here is we had manually scale this deployment with kube kuddle scale deployment. But this means the last, the most recent proper deployment was actually still zero replica. We didn't go through a proper deployment or what our infrastructure teams consider a proper deployment because they had used the kube kuddle tool directly to scale up the replicas. And now our fix is add a warning to kube kuddle scale deployment tool to warn people that this is a temporary tool, but they should still go through a normal deploy process and to actually check what the current replica count is. And our takeaway here is turning on horizontal pod auto scaler for the first time is not trivial. So we tested edge cases such as zero one, etc. And remember the non paved paths, because even though we had fixed this issue in our normal deploy tool Spinnaker was a new, a new path that was still being paved. In our next case is called did we really delete all our master nodes. Yes, yes, we do. So what happens when you delete all the master nodes on a cluster is it means there's no API server, kube kuddle commands do not work. There's no new deploys in a cluster is just broken. But luckily for us, this was just a test cluster. And so as we try to figure out how can we bring this cluster back up. Our options were one completely delete the cluster and and bring it up all over again, or to actually try to fix it in place. So these are pretty bad situations to be in but luckily this was still a test. This was just a test cluster. So as you guys realize, actually, the situation was already documented in our run book, where the suggestion was one, delete our mutation controller as an immediate action via the kube kuddle command, and to terminate the kube DNS pod to get a reschedule. The problem with trying to bring up your masters, once they have all gone down as the order in terms of how you bring things back up really matters. In this specific case, following the run book allowed the necessary services to actually start up again and get unblocked for the masters to actually come up successfully. And the takeaway here is, don't drain your master nodes. If you do do check your run books. Our next scenario, our next case is called masters out of memory. In this specific incident, we noticed our master's CPU usage going up. We noticed that the cube API server is going wild and using up lots of CPU and memory. Our CD resource counts are shooting up. And we're getting out of memory errors on our API server. More alerts are firing. More alerts are firing and more alerts are firing. So this isn't this is a real incident. We also realize at least as we dug more into it is that we actually had no access either. When we try to SSH into debug, we noticed we noticed that the nodes were having real issues, and we actually couldn't get in. What is happening here? So our API server is uming and crashing a control plane nodes are severely degraded. We don't have SSH access to them. We wonder is this a CD problem is a CD having issue, because at CD is having issues of things would be really bad. We noticed that it was having elevated objects being created, but it's actually still healthy and fine and existing workloads are are also still fine. So our our sponsor here was to spin up bigger instances for the control plane for the masters. And, and this was done just in time because our old instances actually just completely died. And we noticed that memory was still getting eaten up super fast for getting these new instances are much bigger. And so for postmortem we realize this was actually one of our largest clusters that was happening that was having this issue so we were wondering is this a new scaling limitation that we've just hit. We also realized the incident lined up with a deploy that had dramatically increased their max surge value, and that deploy was having crash loops. And the theory here is that that deployed with a high max surge value. So it was actually overwhelming our API server. And the takeaway here is a restricting max surge would help protect the cluster. And it's a good idea to be ready to scale vertically. We were lucky that we were able to spin up bigger instances for our control plane to make sure that they scale. And we actually still have some ongoing unknowns, such as what exactly is a breaking point in terms of pods or max surge. We've also had larger clusters in the past before. And, but we didn't run into this issue so the question is what is different now. And three, why did memory usage actually jump suddenly? If you remember from our graphs, the API servers were actually using up to like 50, 90 gigs of memory, which is odd because XED can only use 8 gigs of memory. So what was actually using it all up? Authentication and the world of containerization. At Airbnb, we rely on AWS IAM roles as a primary way to authenticate our services. This is simple enough in the traditional non containerized world where each EC2 instance can have their own IAM role. And they can assume that role and they can request the EC2 metadata API to receive temporary credentials. But what happens when you have multiple pods and services running within a single instance? We could let each pod assume any of the other pods roles or make one big IAM role to encapsulate all of the pods that will be running on it. But we don't want to give each pod more authentication or permissions than it needs. To address this issue, we use an open source project called Kube2IM. Kube2IM provides IAM credentials to containers running inside a Kubernetes cluster based on annotations. But what does this really mean? Well, Kube2IM works by running as a daemon set within the cluster when the pod makes a request to the EC2 API. Kube2IM intercepts the request, then using a special IAM role of its own, it assumes the pods IAM role. Then it makes a request the EC2 API itself, receives the temporary credentials for the pod and then forwards it back. Now, Kube2IM wasn't perfect. It came with some race conditions. So the expected course of events are as follows with Kube2IM. First, the note should start up, then the Kube2IM pod start up. Then we add an IP table rule to forward request to the EC2 API's IP address to the Kube2IM pod. Then Kube2IM starts watching for new pods, after which an application's pod can start up. Then Kube2IM notices a new pod. Caches the pod's IP address to its AWS IAM role mapping. Then, finally, the container in the pod can make requests to the EC2 API. However, what we were seeing was that containers would make requests to the EC2 API before Kube2IM had even noticed the pod come up. And because so, when Kube2IM receives requests, it would not be able to find the pod's IP address within its own cache, and it would just reject the requests. This was especially problematic with init containers, which would run before all of the other containers. So our solution, another init container. That's right. We made a new init container called Kube2IM wait, that would basically just keep pinging the EC2 API until it received a successful response back. This works because we had already set the IP table rule before to forward request to Kube2IM. So if you've got a successful response back, that means Kube2IM responded successfully, meaning it had already noticed the pod come up. Now, this is only a temporary solution. We're currently working on a new authentication scheme by which we have the API server use its own private key to sign a token and inject it into a pod on startup. Then, when a pod needs to get authenticated, it uses a different API to get credentials for the pod's IAM role using the sign token. Then the STS verifies the token signature as valid with the public keys. After the validation is complete, STS returns the temporary credentials back to the pod. Our main takeaways here is that init containers are very versatile, despite their simplicity, and they're good solutions even if they're only temporary. CPU limit equals number of CPU cores in the node still has throttling. So one day we got a message from an engineer asking us, can you explain why we'd still see throttling when limit is set to 36? With only 36 cores, how can we exceed 36? So to kind of give context on this, let's first talk about how CPU limits work on Kubernetes. Kubernetes limits pod's CPU uses using Linux's C groups and the CFS, the completely fair scheduler. So let's take an example of a 36 core machine with a default scheduling period of 100 milliseconds. And let's say we have, in our Kubernetes manifest, limits set to CPU of 1024 millicores. That's equivalent to one CPU core. This gets translated into Linux's CPU.CFS underscore quota of 100 because we have the 100 millisecond default scheduling period times one core, which gives us 100. Now let's take a few different scenarios. Here we have a bunch of different scheduling periods. And in our first scenario, we have one container running one thread, and it can run the full 100 milliseconds without any throttling. Now let's say the container actually had two threads, then thread one and two can run simultaneously for 50 milliseconds, after which it will get throttled because the combined usage within that container would have reached 100 milliseconds. Now, in our final scenario, we have thread one, two and three running simultaneously, and then thread one and two either terminates or gets descheduled after 25 milliseconds. Then thread three can run for an additional 25 milliseconds until it would have reached the full 100 milliseconds of its quota, after which the threads from that container will get throttled. So, in our previous scenario, we had in our limits, a CPU limit of 36 times 1024 millicores, which gets translated into a CFS quota of 3600. So, in that case, we should be able to run our threads on all 36 cores for a full 100 milliseconds within that period or within any CFS period and not get throttled at all. So, to figure out what was going on, we ran a bunch of tests. We made one app with a limit of 36 CPU and we made another app with no limits at all. And we ran them on different nodes, swap nodes, and we found that on average, they all ran pretty much with the same performance. But what was surprising was that despite the similar performance, we actually did see throttling on the app with CPU limits at the 36. So, our main takeaways here were that some throttling is inherent and is okay. And that we should be more careful of our metrics. Docker built in CPU usage information may not actually be granular enough to capture fully what's going on. We also learned that percentage of throttled periods is a better metric than absolute value of periods, since the latter is very heavily affected by the number of threads running within the container. CPU limits cause out of memory kills? In Kubernetes, we have memory limits and CPU limits. And intuitively memory limits cause out of memory kills if they are exceeded, and CPU limits would cause throttling. And they do. But we were also seeing that CPU limits were causing out of memory kills. So here's an example. We have on the top left our pod usage spike a little bit after 12. After which it was immediately followed by a huge spike in memory usage and eventually an out of memory kill. So what's going on here? Well, the first reason was that these were Java processes and the JVM garbage collector is intensive and spawns a lot of threads. And the more threads there are running concurrently, the faster each container can get throttled. There's two main flags to be concerned with here parallel GC threads and concurrent GC threads within Java. And these two parameters are used for different stages of the garbage collection. But the number of threads used for garbage collection for either of these flags can easily exceed the number of threads in the main application. And when both the main application and the garbage collector is being throttled, we can have a situation where data isn't being processed and any data that is processed is not able to be freed. Our second and third problem was that only certain containers get throttled and the back pressure we had in place to kind of mitigate this wasn't working properly. Since Kubernetes relies on Linux C groups to implement throttling. This meant that certain containers in the pod can get throttled while some other containers aren't. In our case, we had the main container processing the data get throttled while another container, Envoy, accepting requests was not. This meant that we kept this meant that we were continuously accepting requests. While we were unable to process them, and we were unable to, again, because the garbage collector for the main container was also in throttle, we couldn't free the memory properly either. We had back pressure in place to mitigate this to tell the clients to stop sending requests in an event something like this happened, but it wasn't working properly at the time. So our main takeaways here. Be wary of what other processes may be running on the pod and adjust limits accordingly. Tune JVM garbage collector threads if necessary. And to make sure it's without proper back pressure to avoid overloading our services. Rogue ECR cleaner. At Airbnb, we make a lot of Docker images. This data is from 2019. And at the time we were making over 2 million Docker images per day. And obviously we can't keep them all. But we also can't do something naive like deleting all the old images because we don't know which images might be in use. We build a service called the CR cleaner that runs as a cron job every hour. And basically all it does is finds all the images in use, and it loops through each of our ECR repos and deletes all the old images except for the ones in use. Fairly simple. And it was working great. Until it wasn't ECR cleaner started deleting images in production. And to figure out what's going on. Let's go back and look at what it was doing. And specifically let's get the find all images and use function. Basically, it would find all the clusters, then keep a list of active images, then loop through each cluster, figure out what images are being used by all the pods. Add to that list and return the list of active images. Well, in the recent change. There was an error case. And on error, it would just return an empty list. And you can guess what happens afterwards. Since our list of clusters is empty, it finds no images and returns an empty list of images. Then coming back up. It would loop through each repository and delete all the old images except for nothing. So it would just start arbitrarily deleting all the old images. And our second mistake was a lack of proper alerting ECR cleaner was failing. But we weren't getting alerts for that. And we also weren't getting alerts for image pull back off errors in production across multiple pods. But once we figured out that this was happening, our fix was to first shut down ECR cleaner to prevent it from deleting more images. Then look for crash looking containers, look through ECR cleaner logs and kind of cross reference them to figure out what images were missing and manually rebuild them all. On the right, you can see a teammate rebuilding all these images and keeping track. And this is only a short snippet of all the images that we had to rebuild. And it was very tedious. So thank you kind teammate. And our takeaways here were that the more critical the service is the more crucial and thorough testing and review should be. And we should have proper error handling and alerts for infrastructural issues. And we should also try to make sure that our fixes aren't causing problems elsewhere. In this case, because we were rebuilding so many images, we were causing other people's builds to slow down a lot. And we should have seen forcing that be careful what you break. As part of our uncle readiness schools, we started doing fire drills. So this is what they looked like. Someone would set something up and send a message in our slack channel. So for example, fire drill CI is broken and we cannot do a build for ECR cleaner. We need to ship a new version to production that displays a new logline on startup without relying on CI. So this is all disclaimer. CI is not really broken. Do not panic. This is a drill. This is only a drill. And we did a lot of these and we found a lot of bugs, a lot of places to improve our code and our documentation. And it was great. But there was one particular fire drill that was pretty interesting, but also pretty insightful. And it had to do with the admission controller. This was actually one of the first fire drills that I did as a new member in the team. It was pretty simple. Basically, a change was made in the admission controller to block any creations of a replica set. And it looks something like this. It's a new rule that said if the resource being applied is a replica set, return an error. And the fix was supposed to be simple, just find them all work and commit and deploy that back onto the test cluster that we were using. And that's exactly what I did. And that was that. And it was all going to find until a few days later, the same person who had made the fire drill send this in our team chat. I deployed onto the test cluster, but it's as if the admission controller didn't get deployed, even though the deploy went through. Has anyone seen that before? Well, I didn't know the time, but as it turned out, I had seen it before. When I went back to deploy an old commit. I thought the deploy had gone through, but admission controller didn't actually get deployed. What happened was that there was a bug in our deploy process reporting success for failed deploys specifically to admission controller in this case. But why was admission controller failing in the first place? Well, it was because the mission controller was not whitelisted by the admission controller. This meant that since the admission controller was called replica sets, it was also blocking itself from being deployed. So we were locked out. So our solution just delete the validating web of configuration for the admission controller. And that way we should be able to bypass the admission controller step. But this didn't work either. And that's because when the request is sent to the API server, it goes through a bunch of steps. It eventually lands on the validating admission step, at which point it takes a look at the validating web of configuration resource, which is what we have deleted to figure out what webhooks it should get confirmation from. But as it turns out, on our admission controller deploys, the validating web of configuration is applied before the webhook. This meant that by the time it got for us to update our webhook, which is where the bad role was, the validating web of configuration has had already been replaced. So the actual solution just nuke the whole thing, delete the mutating admission controller webhook, delete the deployment and then redeploy the entire thing from scratch. So our takeaways here, double check that your deploys actually went through. Make sure that the thing that controls your deploys can actually fix itself with another deploy. Be aware of Kubernetes apply ordering, and that even the simple fire drill can reveal a lot of insights and bugs and TLDR never never write a rule that blocks all replica sets. Alright, to recap, some of our top 10 takeaways are one, do test new features in test clusters to be aware of the existence of mutating admission controller. Three, remember the non-paved paths or whatever new paths you might be paving. Four, don't drain your masternodes. Five, be ready to scale vertically. And eight containers are versatile despite the simplicity and are good solutions, even if they're only temporary. Some throttling is inherent and is okay. Be wary of what other processes may be running on the pod and adjust limits accordingly. Have proper error handling and alerts for infrastructural issues. And be aware of Kubernetes apply ordering. Right. Thank you everyone. This is the end of our presentation. If you would like to learn more, check out our engineering blog. Airbnb is also still hiring and a lot of the work we share today were not done by just me or Joseph. It was all done by our wonderful team. So if you would like to work with us, reach out, apply. And if you have any more questions, feel free to contact us to follow up. Thank you.


Greetings, Lujban friends! In the last video, I translated the second sentence of the Hobbit, and I went ahead and opened myself up to some creative criticism, and I'm glad to say that I learned a lot, and I got some really useful advice. First of all, in the comments, and second of all, in the Lujban chat, so I just wanted to go over that, because I think there are a lot of good points and good alternative translations. So the first point was made by commenter microkernel, who says that there is a nice word called banzu, which means that x1 is sufficient for the purpose of x2, and it sort of captures a little better the meaning of the English. English, it basically says nothing in it to sit down on. My translation was zero things with the purpose of for sitting upon, but by using banzu instead of secosmu, you would get zero objects sufficient for sitting on. So in other words, there literally is nothing to sit on, or there literally is not nothing to sit on, like a chair would have purpose sitting on, but a rock would not, but a rock is certainly sufficient for sitting on. So banzu is actually quite a nice usage instead of secosmu. Let's see, aced gravity commented with a few points. One of them was using the word klahu. Klahu, if we look that up, it means x1 is without x2, and I was using not containing zero things. So of course that double negative is always going to be confusing. Not klahu would mean not without things. So not, well of course there is kind of a double negative in there, but it makes a little more sense than saying not containing zero. Not without, and then low banzu, bello nu zutze. Not without things sufficient for sitting on. So that was also a nice word to find. Ace gravity also translated the last sentence. It was a hobbit hole, and that means comfort in a really nice, concise fashion, which I like. And it goes ka ridder hobi kevna tefihe lo nu kufna. So ka it ridder hobi kevna was a hobbit hole. Tefihe is a modal tag. It comes from tefinti, finti means that x1 creates object x2 for purpose fx3. Te takes that x3 position and puts it into the x1 position. So tefinti would mean x1 is an object made for purpose x2. So tefihe is a modal tag, meaning made for purpose whatever. And in this case, it's this lo nu kufna, which is basically an infinitive meaning to comfort or for comforting. So in short, it was a hobbit hole made for comfort, which I really like. It's very nice and a lot less wordy than my awful translation. He also points out that there really is no need for shu whenever you use ka, and the reason is that ka is actually a prosumpti, like me, like doh. So everywhere that I use ka, it's just as if I used me. I don't have to use shu because there is no reason to, there is no way that ka could bleed into the salbury of the sentence. So ka na rigni, ka na sudga, and so on. So that is definitely something that I would just do right away, because that's very useful.  And I'd like to go over some of the differences and interesting points of that translation. So the first thing that was translated was, in a hole in the ground, there lived a hobbit. Now I went through contortions in the very first video about how to say this. And the reason that I went through contortions is that I did not use the place markers, fa, fe, fi, fo, and fu. These are markers or tags that you can tag sumpti with that basically mean this goes in the x1 position or this goes in the x2 position. So when you tag a sumpti with fa, what that basically means is it doesn't matter where the sumpti is, it's actually going to be in the x1 position. Fe would be, it doesn't matter where the sumpti is, it's going to be in the x2 position. And because I avoided that, I was not able to move the sumpti around. So in this case, while Nihou is basically a paragraph separator, but fe lo kevna be lo tum sfa, that means in the x2 position is going to be a hole, a hole in the ground. So they use the bet construction rather than the tanru that I use, tum sfa kevna, which is fine. Shuhabzhu and then fa lo ridochobi. So basically what this means is you can't, by just reading this, you can't really say this means in a hole in the ground. You can only say a hole in the ground was indwelt by a hobbit colloquially in a hole in the ground, there lived a hobbit. So it's not that, it's not that I don't like the translation. I mean, I understand the translation and it's probably, it's probably idiomatic lojban to do this. I just, obviously in English, I'm used to using prepositions before sumpti, essentially. And this to me doesn't have the preposition in it, so I'm not quite sure where the sentence is going. Now to be fair, Latin is kind of the same way, right? Because you can reorder Latin based on whether it's an accusative or a nominative and so on. So sometimes you really do have to wait for the verb in order to fully translate the sentence and that's perfectly valid. I'm just not used to it. You know, my first language is in English and I've never really learned any second language or if or, you know, even if I tried, I always failed miserably. So anyway, the next part is the part where I really learned something interesting. So here's that fa again, right? So basically, we're talking about la kevna, the whole. We're not talking about it yet. But what we're saying is namabla, which means not, did I use mabla? No, I didn't. So let's look up mabla. Mabla is a cursed or deplorable. Which is which is fine. It stinks.  I probably wouldn't use it, but it's a reasonable word to use. It's deplorable. So not a deplorable, I guess that could be nasty in in the in the more human sense.  I probably wouldn't apply it to a hole in the ground, but that's okay. Let's go with it. So not a deplorable. Now here we're using jet. Now I thought that jet was just for Tanru. And it turns out, and this is mentioned nowhere in the complete Lajban language and nowhere that I can find. And yet the parser accepts this jet is an afterthought for brihla, not just for Tanru, but for brihla themselves. So it's perfectly reasonable to say namabla, jet oljinsa, jet shilmo. And now that's the entire celebrate. Fa la kevna.  So this is really nice. I used gihe and then of course I had to repeat na. They used oljinsa instead of tohe jinsa. Tol is a prefix that you can add to a word that basically is the same thing as tohe, meaning the polar opposite of. So toljinsa, unclean. So that was nice. So here's one other interesting thing that I've noticed. And I kind of agree with this. I mentioned that we have jhe here. Now jhe means and, so it means all of these things altogether, simultaneously. So we're saying that it's not a nasty, dirty, wet whole. Nasty and dirty and wet. Whereas the na comes outside that. So again, you know, I made the point that if you have not a and b and c, well that simply means not a or not b or not c. When apparently what I really wanted was not a or b. In other words, it's not any of these things even alone. So it's not just, you know, it's not even a nasty whole.  It's not even a wet whole. And I think actually now that I think of it, I think this is probably the better way of putting things because what Hulkin is doing here is he's he's drawing a contrast between two extremes. So the one extreme is a whole that is nasty, dirty, wet, filled with the end of worms and an oozy smell. All of those together. And the other extreme is that it's a dry bare sandy hole with nothing in it to sit down on or to eat. That's the other extreme. So he's basically saying it's neither this extreme nor that extreme. In other words, it's not anything in between. And that's why probably jet should go inside and not should go outside. Whereas my translation had not for each one of these things. And then I ended them together. So I think that in the end, this is probably the better translation and probably the more true to Tolkien's words. Okay, so let's move on to the second part. Ijenai, this means and not. So remember that Ije basically is a Breedy connector with and and in order to negate the second Breedy, you just suffix with nai. So Ijenai and not ka it Vasru was filled Los Pisa. Now let's see what Spisa is. Spisa is a piece. So Spisa bello, sure new pieces of worms, which again is a reasonable translation. I kind of like ends a little better because you know, really what Tolkien is talking about is the wiggly end of the worm sort of sticking out of the walls pieces, not quite as Andy, but that's okay, we can go with it. Gihai and Der Specku, Sepanchi, which is basically the same thing that I used. So now Ijenai and not Kuhi. So Ijenai and not Kuhi, however, so in contrast, right, and there is that yet. So yet was nicely translated into Kuhi. Ijenai, Kuhi basically means nor yet, Sudga dry, jet. And again, there's that Brievla connector. Jadashau. So what's Jadashau? Now that would be the word for bear. And let's look up Jadashau plain. Sure. That's that's probably better than I had a bear. Let's see, I translated it as where is it? Kunti. And what was Kunti again? Kunti, empty bear. I think bear probably, probably, sorry, Jadashai plain, plain probably gets more of the nuance of bear, as opposed to, you know, empty, completely empty, which doesn't really make any sense. So I like that. Shanri, so Shanri, Shanri. Okay, I think this is supposed to say Shanre. There's a rule about Gismu that says that to Gismu cannot be identical with the exception of the last vowel. So this was supposed to be Shanre. So not Sandy Gihai and Klahu. So there's Klahu again. And without Tuha. So Tuha is an interesting schmafo. It's something that I that I never really got into. And I don't really like to use that much. But it has a great deal of uses. Tuha is basically an abstraction. And it basically says it basically means something to do with or something about or, you know, some kind of action to go along with. So for example, Loser Zutze. Now, if we look up Zutze, which is the sitting thing, right? So set Zutze would basically mean a sitting surface. Okay. So what this means is without something to do with sitting surfaces. Luhu is the terminator for that. And then of course is the connector. And here we've got another Tuha, Loser Schietka, which would be, you know, nothing to do for with eating. So without, again, it's it's difficult for me to kind of understand the way this is being used. We know what the translation is supposed to be. Nothing, nothing to sit down upon. And basically what this means is nothing to do with sitting surfaces, and nothing to do with eating, which again, kind of kind of goes along with the translation in that, you know, you have no sitting surfaces and you have and you are devoid of things for eating. It doesn't really specify anything about what those are. Again, I'm not really that used to using Tuha. So I can't really comment a great deal about that until I, you know, sort of understand more uses of it. And then we have cut reader Choby Kevna, it was a hobbit hole, Ije, and Loduhu. So what is Duhu? Duhu is an abstractor. And it basically means and that means that or and that means Loduhu gohi basically means go he means the previous breedy, which is the fact that it was a hobbit hole. Shunibli, nibli logically necessitates, in other words, it implies loka, the property of kufra, comfort. So what we're saying here is that it was a hobbit hole. And what that means, what that logically entails is comfort. So that's that's a nice, a nice another nice compact way of putting it. But honestly, I think that this really is the winner right here. I really like this this way of putting it. So anyway, that's all I wanted to really go into all the feedback that I got. It was really excellent. I learned a lot of new words, a lot of new idioms. And I definitely have questions about tuha. I really need to understand the usage of this more because I've seen it a lot of times in chat. So anyway, I guess that's about it. It seems like people do want me to continue with the third sentence. I have gotten a warning that the Tolkien estate is is pretty rabid about protecting their copyright, which means that for all I know this video is going to be taken down because it actually contains a few sentences out of the Hobbit. I am not going to say anything about that because otherwise I will just get just completely apoplectic and incensed. So anyway, that's it. And I will see you on the next video.