So we're going to do this without presenter notes. So I don't know how well this is going to go. But anyway, thank you for your patience. My name is Rick. I'm here with my esteemed colleague Ray. We work on the engineering team at Segment. We're going to talk about how we saved a ton of money migrating from Aurora, which is nice but expensive, to FoundationDB, which is quite efficient and also nice. We're going to talk about a few things. First, we're going to talk about what Segment does, since that seems to be a question many people have. We're going to talk about the use case of identity resolution. We're going to talk about the data model we use on top of FoundationDB. Then Ray is going to talk about how FoundationDB saves us money, how we got to production, and some potential future use cases for FoundationDB at Segment. So what is Segment? Segment is a platform that you use to instrument your mobile apps, your websites, to get data from those about how your customers are using those products, and forward them through Segment to standardize them, to make sure they're clean of PII, and to forward that information onto downstream analytics tools, AB testing tools, mail campaign management tools, lots of stuff. So the idea is Segment delivers these amazing customer experiences. This is common. We see people using tools. They connect things all over the place. There's a lot of inconsistency. Data is going to the wrong place. It's full of stuff like PII, where it doesn't need to be. You may recognize some logos here. I hope that this isn't offending anybody for some reasons. We call this CDI, which is customer data infrastructure. Now, this is kind of a... We have a few slides that are kind of from our marketing team, but they help kind of illustrate what we do, because it's kind of complicated. So the idea is we provide this single point of customer understanding and action. We help you synthesize your data. We help you standardize it, so clean it up, make sure it's all in the same format, and then collect all that from various endpoints, from the Roku box sitting on top of your TV, and then power all the same tools from that same data. This is kind of a slide. You can kind of see this has nothing to do with actual reality. This is just how the thing works conceptually. Data flows from the sources. We clean and synthesize it, and push it out to destinations. So this is kind of a timeline. This is only really inserted here to kind of give you an idea of where we are relative to FoundationDB's open source release in 2018, which is where we started to really be interested in using FoundationDB. So around the time that we launched this product called Personas, you see in 2017, we built this thing on top of Rora, and then we started to look at FoundationDB afterwards. So what is identity resolution? This is the use case for FoundationDB at Segment. It's kind of in the center here. What we do is we try to build profiles of users that are using our customers' products, and that often means that they have various identities across the product that we want to unify into a single picture. And so what we do is we build an aggregated graph of data associated with a unique user, and this helps simplify analytics and provides a unified view in your downstream tools. So a common problem you might see is a user goes to a website or opens an email, and then they're redirected to a website, and all we know about you is some random ID we've generated and given in your cookie. So you can see this anonymous ID. You might have opened an email, and through the data that we collect over time, we're able to basically build a unified ID. You may log in and you may complete a transaction, and all of those events have different identifiers associated with them, and the goal is to basically provide a unique ID that connects all those other identifiers together. So we do this using two different types of data, one we call mappings and one we call merges. And a mapping is what we do to connect an external ID, what we call an external ID, like an email address or a database ID, various other things, to what we call a segment ID, that universal ID. So right here in the example, these external IDs would be this user ID, 8Y, et cetera. The email is another external ID sloth.segment.com, and we're trying to map those all to this universal ID, use 1, 2, 3. Now this comes with a problem, which is that we generate these new IDs, these new quote-unquote universal IDs all the time, and we want to be able to, at a future time, merge those together when we realize they're actually connected via some other element of data. So that's what we call a merge. Now let's talk about what this actually looks like on top of FoundationDB. So we don't use any real mapping layer. We decided to go directly to the KV store. Some of that is because we use, all of our services are written in go, and there isn't a ton of layer implementations for go. So we directly use the KV interface. We use subspaces to structure our keys, and we have two types of KV pairs. So we call KV pairs objects, which are basically surrogate IDs, keys to JSON serialized objects, and indexes, which index those objects based on the fields that are in those objects. So again, we do mostly point gets on these object KV types. Value is a JSON serialized object, and when I say surrogate key, I mean just a generated key we use that doesn't really mean anything outside of our system. We also have these index KV kinds. We use the subspaces to segment the compound identifiers. We generally look up by key prefix, so we use get ranges for these, and then the suffix of the key is the actual target, the thing that we're mapping to. So the prefix will have multiple targets that it's identifying in the system, and the value is usually empty. So this is kind of what it looks like. Everything is prefixed by tenant because we're a multi-tenant system. This is one of them might be an identifier type. To an identifier, this would be a range prefix, and it's connected to this mapping ID, J2, LK, et cetera, garbage. So let's talk about this first type of data, which is mapping data. So this connects an external ID like rick at segment.com to a unified segment ID. All right, so this is kind of what it looks like. It's very simple. We have a tenant prefix and a mapping ID connects to the JSON object with the information. Very simple. Sometimes we want to list all of these. We run various reports and things on these. So say you go to your interface and you wanna see all of these mappings. Literally people wanna do that for some reason. And so you can actually, we have a big index of all these mappings that is ranged by tenant. We also have an ID index for the segment ID field. So you can see we can go search specifically for segment ID and point directly at a mapping ID. There may be multiple of one or the other. Okay, so merge data. So this is what, when we realize, we finally realize, hey, there's actually multiple mappings that connect to the same ID. We need to bring these together. We create what's called a merge entry and those associate two quote unquote unified segment IDs. So this is what a merge object looks like. Similar to the mapping, it's a tenant prefix with a merge ID that's generated, surrogate key. And generally there's some other fields in here. There's some timestamps, et cetera, but the only things that really matter is it's like a graph relationship. It's a from and a two. And these are the segment IDs. So we have an index, the from index. Very simple. It's the from field to the merge ID and a two index, a two field to the merge ID. Very straightforward. All right, now I'm gonna go to my friend Ray and he's gonna explain the rest of the gory details of the system. Thank you, Rick. Can you all hear me? How about now? It's better. Okay, cool. Yeah, so I'm gonna talk a little bit more about sort of the path to production and a little bit more about operating the system in production and then some future use cases. So these are essentially some of the needs of this workload, as you can imagine. It's a high throughput sort of workload. It's, we project the system needs hundreds of terabytes of space on disk replicated. It's not that large right now. It's more in the order of tens of terabytes but it's rapidly growing. And of course the key piece is we need serializable isolation levels. It's also a very high volume sort of a read workload. So the read volume is much higher than the writes as you'll see in a moment. So a little bit about how data flows through the system. So the identity resolution system sort of is teed off of a stream processing system that handles the core data ingestion and then routing that data to all the destinations that we integrate with. So data comes in starting from the left-hand side of this graph. Data comes into our API and into our core ingestion pipeline. There's a lot of systems not featured there but we're just focusing on the identity service here part. So there when it first comes into the ingestion pipeline we start in Kafka and then we have some systems basically performing validation, schema enforcement, things like that. Any customers who are using Persona that data is then teed off into the identity ingress topic and then we have a service called the identity resolver. That service scales in and out horizontally as needed, depending on the volume of traffic and that queries the identity service right path essentially on the top left there. So that is where for every message that's coming in essentially we're going to FoundationDB looking to see if we have seen this identity before this piece of data and creating the mappings and mergings as necessary. On the bottom half we also have a read only path essentially so that's just read only request to FoundationDB for other systems downstream that are looking for identities. Okay, so as we mentioned we originally built a system with Aurora and it actually worked pretty well. There are some nuisances for the developers and FoundationDB is actually a bit better.  It was sort of a bit of juggling but Aurora worked but the primary problem was the cost. So this is just like some baseline cost for Aurora just like an idea of the back of the napkin math we did when we started to consider FoundationDB. So basically a terabyte and a billion IOPS a month it's not too expensive it's about a K with a single instance but if you can imagine even like a medium-sized workload that basically grows to tens of thousands of dollars really quickly and this is only like 20 terabytes and a hundred billion IOPS a month. You're looking at tens of thousands of dollars. So you can essentially basically provision we determined that we could run essentially the same workload on FoundationDB for about a fifth or six of the cost. So roughly with the resources here. So 20 I3 extra large instances, some stateless and some transaction nodes. Okay, so a little bit about our path from prototype to production. So prototype was pretty quick. We were able to stand it up pretty fast but we hit a couple like hurdles on the way to production. The first one was everyone's favorite game. And so we weren't exactly sure. We weren't getting this sort of performance that we thought we would get based off of some of the benchmarks we saw online. And we weren't sure was it something we did wrong. You know, we're systems under provisioned. It turned out it was mostly us doing things wrong. We weren't like we're basically not using the API correctly. We weren't pipelining requests, things like that. So we quickly improved performance but there was some things that was confusing. So for example, like the default recruitment sort of confused us. We found the tuning cookbook and that sort of led us down a path of like evolving our cluster configuration. So like when we started, we saw this as like CPU utilization with the default basically settings with everything on set. And you see CPUs sort of all over the place. It was a little bit difficult to reason about what was sort of going on. So we essentially came up with a configuration. We went through a little bit of an evolution but we came up with a configuration and ended up with heterogeneous three tier sort of cluster set up. So we ran, we run a stateless cluster, a transaction cluster and a storage cluster. Actually, sorry ASG, auto scaling group. We're running all this in AWS but that allows us to run the C5 instances for the higher compute workloads and then we run I3 instances with NVMe storage for the storage and the transaction tiers. And that allows us to scale these clusters independently essentially. A little bit about how we get this all sort of provisioned in production. So we use Terraform. We sort of set out to then, so after we solved some of the performance issues and we were getting the results we expected, we sort of started templatizing the infrastructure. We run, as I said, three auto scaling groups and then we run a container orchestration system on top of that. We'll talk about that in a moment and we run a bunch of supporting tools that we wrote in-house and we hope to actually open source and share with the community on top of the instances as well. We run the FTB server process directly on the boxes and I think we run FTB monitor as well. So just a little bit, look at the Terra code. Like if you look at the top of our installation we basically have everything broken down by environment and all the foundation DB config is under there. We have an input and a main file. Just running through this quickly we can specify the instance type, the number of processes to run, any sort of tuning we wanna do, knobs and things like that and then we actually provision each individual tier. So here we're looking at the storage tier and the stateless tier and then at the cluster level this is where we actually have our implementation. So we're running for container orchestration. We're not on Kubernetes right now we're on ECS but this is, we hit AWS implementation and then we have some of our services down there. So you can basically stand up a new service and create a .tf file in your environment and basically it will stand up autoscaling groups, it will stand up the container orchestration systems and then run these services that we wrote, the supporting services on top of those systems. A little bit about the sort of how we provision the boxes when they come up. So we use the user data configuration. So we build all of our Amys with Packer and then we use user data configuration to basically specify what class this is and any other properties associated with like tuning so knobs or any other parameters we wanna pass and then when the instance is booted up we have a bootstrap that runs and a set of system deprocesses that go ahead and look at it. They determine what type of, what class this is and says oh okay it's a storage node, I need to mount the NVMe, I need to format the device and then I need to finally build the foundation db.conf. So we have this script called configure fdb that gets executed and it basically sets all our parameters, sets our locality zone ID for replication, sets up the backup and then we're sort of off. So that's like the core of the Terraform configuration and then as I mentioned we have a few services that support the system that run on the ECS or any container orchestration system. So the first one is fdb discovery and I guess you all are probably familiar with this problem. So basically what fdb discovery is is this is a small service that runs on all the clusters, it's fronted by a load balancer and you can curl that service and basically get back the fdb.cluster config, pop it in fdb.cluster and add your node to your cluster. We have a process called fdb trace and that is basically parses all the trace log files and converts them to metrics and we feed those into Datadog today. So just a look at some of those, for example this view right here it's showing like a chart with master recovery commits, it shows you queues for storage servers and transaction nodes. You can see data in flight and moving data that's actually queued. You can see mean and max bytes per commit and then we also track transaction latency P90 and max. And then the last process, we actually have a couple more supporting tools but the last one we're gonna talk about today is fdb backup and that just basically kicks off the backup process. So we moved, when we started, we were on a six auto version, I can't recall what it was, but we moved to 18 for the blob store backup support. It was pretty smooth, we ran into a couple issues with expiring data. I think we were one of the first people to use it because I was posting on the forum asking questions and I think we were sort of early on with it. We're considering moving to maybe a hot standby with the DR agent and kind of because of the limitations we've seen around how quickly we can recover from backup and so I think Evan talked about this earlier but this is a screenshot of a game day test we ran where we basically rebuilt the, restored the cluster from S3. Looks like we were mostly maxed out on CPU but after we pulled down batches of data, it takes a while for it to apply. So this was like, it was a couple of terabytes on disk and it took several hours. Okay, and then so I'm gonna talk a little bit about game day and chaos testing and then we'll wrap up with some future use cases. So before we put it into production, we wanted to run through and try to break it as good as we, or as well as we could. We tried to induce tons of different failures the system behaves just as advertised, it was great. So we were able to kill pretty much anything in the cluster and it behaved well. We did notice that partitioning storage nodes resulted in high CPU utilization for example as the cluster healed. We also created a scenario where the performance dropped by basically exhausting a disk but we did that sort of not through the foundation DB path we just basically allocated a bunch of space on a drive and so we spoke with engineering and they're like, yeah, don't do that. So this is an example of like losing a storage node during game day and we're like, okay, well the cluster's healing, healed pretty quick. We saw increase in latency but our workload is the data's in Kafka so we can handle a little bit of latency and then we just burn it down quickly after and this was the case where we essentially exhausted storage on a node and then once the node basically handed off data the system recovered. So other than that, it's been great, it's fantastic. It's been really sort of hands off as far as operational stuff, it's been really good. Some ongoing issues we have though and I think these were mentioned today, one of the data distribution unpredictability. I think it's probably predictable but I'm not exactly sure how it's behaving so maybe just some more observability or some metrics associated with that would be great. We need to internally consider three data hall. We're running across three AZs and we're doing triple replication now and I think that's really ideal if you have five localities zones so I think three data hall would be better for us when we're running with three AZs and then I'd like to see maybe some ability to throttle things like healing or things like that if I want to trade off latency for recovery time for healing time or things like that. Finally, let's talk about some potential future use cases so we're trying to get to the point where this is sort of a primitive inside of a segment and new product teams can come, start up new foundation DB instances, get off and get running and just build their stuff on top of this. One of the use cases that we're considering is for a system we have internally called DDoop. Basically a lot of the messages we get come from mobile devices and mobile devices are on cellular networks and those are not necessarily reliable. People are going under bridges, going indoors, losing signal all the time so we get a lot of duplicate messages so we have a system called DDoop that sits sort of towards the end of the pipeline and essentially the way it works is we hash messages by message ID to a Kafka topic and then we have a worker per topic and topic in partition that consumes that and says it looks in a local database, it's RocksDB today and says have I seen this message before? If it has seen it, it would suppress it from going downstream. Otherwise, if it's the first time it sees it, it writes to Kafka and it commits to RocksDB.  But anyway, when we restart, we read our writes. But this system is, it's pretty good because it's fast and basically you don't have to deal with any issues with network partitioning and it's an embedded database but the problem is it's difficult to scale because it's tied to Kafka. So if you wanna increase your Kafka partition count if you need to increase your count to add more processing power to the system, we have to jump through a bunch of hoops to basically shuffle all this data. So, it would be nice to move this workload to FoundationDB. So we could have an elastic tier depending on how much, how many messages are coming in, what the volume is. Those could scale up and down. Query FoundationDB to determine if we've seen these messages before and suppress if needed. Otherwise, forward down the rest of the pipe. And that is it.


More than half the world is still offline. What's the main reason and how can this digital divide be bridged? Well, there are a number of reasons for this. Historically, attention has tended to focus on supply side issues of availability of network, infrastructure and coverage. Nowadays, attention is shifting somewhat more to the demand side, including affordability issues and literacy, people's ability to access content in their own language, which more increasingly in minority languages. In fact, in many developing countries, we're seeing a lot of either or choices coming up about internet access. So, people may have to give up something else in order to go online or they may be time poor as well as suffering from seasonable or vulnerable incomes.


When the idea behind adding tiling to you know, I'm show was first introduced a couple of years ago I didn't actually think much of it I didn't think it was that great of an idea because if you want tiling just use a tiling window manager it makes sense Because tiling window managers are so much more customizable than a genome extension ever is going to be so for the most part Over the last two years. I've just kind of pooped it like it's not something that I've ever really decided I even wanted to try because it just doesn't seem all that great of an idea to me now I have tried some of the k-win scripts for Kd plasma that do there's something similar and I also think those are kind of janky and not very good So that led to me to a resistance to even trying the tiling mechanism in pop o s simply because I just assumed that it was bad and As we all know you shouldn't assume things so what I thought I would do today is Try it out now. I've been messing around with it now for often on for about a day in a virtual machine And I have some thoughts so let's go ahead and take a look at pop o s So this is basically just pop o s straight out of the box The only thing I've done is changed the dock so that it's not full, you know Full screen. It's just the part in the middle. That's all I've changed It's just because I prefer that look if you want to go through and try tiling in pop o s Here's how you do it. You just go up to this little button Which is installed by default and click this little slider so that it's in the on position And then as you open up new windows or new applications You'll see that they are tiled just like you'd expect them to be now if you keep opening up clients And if I remove my face here You'll see that the layout looks similar to what you'd expect in BS PWM and similar to BS PWM Where the client spawns really is determined by what window you're focused on so if we're focused on this Window, we'll see things spawn like that if we're on this window Like that and so on so forth and you can also see that because we're dealing with GUI applications This is it. This is gonna. I'm terminal and it has a lot of GUI aspects to it the bar at the top title bar The window decorations all that stuff the more clients you add to your desktop here The harder it gets for to tile so you end up with things where things are kind of overlapping It's messy now It's not anything that's unusual when you when it comes to a tiling window manager GUI applications oftentimes can only get so small before they won't get any smaller and That's basically the case on a lot of tiling when managers as well So it's not a huge deal, but it's definitely something that you'll have to keep in mind now Nobody's ever going to go through and try to create a layout like that and expect it to be usable So it's like that's not a huge deal It's just again something that you'll have to keep in mind because if you wanted to go through And resize things like I just did there you'll see how things are actually overlapping even there because Gunam terminal can only get so small in one direction So interestingly interacting with and moving the clients around Is fairly easy. You can go through and do that with a keyboard shortcut You can also go through and use your mouse if you wanted to use your mouse And it's kind of cool that you can use your mouse and it allows you to kind of create your own workspace. It's I'd say moving the the clients around with your mouse is kind of like bspwm's like pre-selection and that as you can see it kind of allows you to Determine where it highlights where it's going to go and what position and how big it's going to be all that stuff It's not as customizable as pre-selection and bspwm is But it's kind of similar also You'll see as on the screen Is that if you have things overlapping in a certain place if you try to drag them into Like on top of each other You'll then get tabs which is very much similar to what you'd see in i3, which is really cool, right? It's not something that I expected to exist The fact that you can go through and have tabs now if you you've noticed when I tried to add a third window It doesn't seem to allow you to have a third window As far as I can tell two windows in a tab stack is as much as you can have I may be doing that wrong But as far as I can tell you can only do two now the layout is Overall fine. The problem is other than you know dragging things and customizing basically making your own layout so that there's You know different widths and different heights and all that stuff There are no other default layouts that you can select so you can't select like master stack You can't select a centered master or any of that kind of stuff that you'd be able to choose in a like a tiling window manager What you get is the Fibonacci layout and that's pretty much it out of the box and you can't change it So and that is my first criticism is that there's no settings for this at all As far as I can tell I've searched now several times and have not been able to find settings for the tiling functionality of pop West They just don't seem to exist. They're not in the standard settings app They're not in the extensions application and they're not anywhere located here Right now there are some things that you can change. So for example, you can create basically what amounts to rules So if you've ever used a tiling window manager before you can know most of them allow you to set rules So that certain applications function in a certain way So for example, if you want to have say like gimp or something float You could go through and do that and the tiling when the tiling functionality of Pop OS has that functionality built in so you could go through and create exceptions to the floating rules so basically Allowing things to float and that's really cool. There's also several built-in ones that are automatically Set to float by default So that is basically the only settings that I can find that actually exists now The only other things that I did find so let's go ahead and close a couple of these things with super q and open Up the settings app If we go down to keyboard and then customize shortcuts and then tiling you can go through and customize Some I should say very few of the keyboard shortcuts So if we actually go up here and just open up the keyboard shortcut cheat sheet We'll close this so you can actually see this It tells you all the keyboard shortcuts that you'll need to know and most of them are pretty intuitive But the problem is of these the only ones that you can change are these And there's like what five different keyboard shortcuts here And that's all you can change and there's way more than five keyboard shortcuts now That's not a huge deal because you can adapt to these keyboard shortcuts but one of the things About a tiling window manager that most people like is it allows them to make Customizations to their keyboard shortcuts Not everybody prefers the same keyboard shortcuts and this is very restricting in that manner There's only so many you can change all the rest of them as far as I can tell Are hard coded and you can't change them The other thing that bothers me Seriously bothers me Is that the super key pulls double duty So you can see all these keyboard shortcuts shortcuts are using super which is the windows key Now I don't have a problem with the windows key being the meta key. It's the key that I use It's the key that I think is probably the one that most people use although I've heard from many people who use alt But it really doesn't matter super is the one that they've chosen to use And the problem is that it also pulls double duty in that it brings up the launcher So sometimes if if you let up on the super key when you're trying to Do a key binding that's going to come up when you're really wanting to do a key binding like super key or something It actually just did it there because It's finicky because that button is doing two different things. It's not all the time It's just sometimes and it kind of bothers me. The thing is They could have changed it so that the alt key was the meta key in this case Or whatever switched them around. It doesn't matter. I mean, I understand why super key works this way Because people who use gnom are used to using the windows key to access Either a launcher or the app drawer or whatever Usually the super key does something like that so I can see why they have that but it's also Does all those things that we just showed you or I just showed you for the tiling functionality And a key doing double duty does not always work that well And it's always going to lead to some problems now as we open up new clients You'll see if you if you pay close attention How some of the clients think they want to just open like up like normal like in a random position on the desktop And then they snap into whatever floating position they are So you'll notice up here where the mouse is if I open up a you know the terminal down here That's up there where the mouse just was There was there's a flash of red And then it goes down to where the floating where the next floating window should appear I don't know whether or not that's something that pop shield always does or if it's just a virtual machine thing, but it Makes it feel not so smooth So it's not it's not like if we go into just dwm here and go into another workspace and then open up a ton of terminals It just is very smooth. It just opens them up opens them up until you know You fill up the whole screen and there's no Weirdness right and that's because this is meant to do tiling and you know, it's not a huge deal of course But it does make it feel a little janky and that's where My final conclusion comes in a little bit is that It's better than I thought it would be It's better than I think it was when it first was announced way back when the problem is that it's still even after all this time Feels a little bit Paint it on if you know what I mean. It feels like it's kind of if well, it feels like an extension You know A lot of genomic extensions feel this way like they're kind of weren't meant to be like they were tacked on That their functionality that doesn't really kind of fit with the way genomes supposed to work And that's the way this feels it feels tacked on and it feels hacky in some ways Now it's not as bad as that cronkite cronkite and plasma for my very very brief experience Well, freely admit that it was I only used it for Maybe a few hours, but in my experience during those three hours That felt way more janky than this does like this is usable Cronkite for me just wasn't it had too many weird weird things going on with certain windows and kitty Spawns windows weirdly anyways unless you really dig deep into the settings and it was a mess The poppa west one is way better than that, but it still felt janky to me So but the thing that bothered me the most wasn't the jankiness of it Like I could deal with that if I were going to use this the problem that I have Coming from a window manager coming from a tiling window manager is the lack of customizability Like the thing about window managers that is great is that you can get into the config files And you can change things right you can alter key bindings. You can alter rules. You can alter layouts You can do all this stuff And as far as I can tell that's not something that you can do with the extension for popo s Now I will be completely honest and say that I did go through and search for a config file Like I went through and looked in dot config. I looked in the home directory for hide hidden files I didn't really go through the root directory all that much because I'm assuming That if there was something hidden there that you probably weren't supposed to mess with it It's usually the rule if you if it's in the root file If this if it's in the root directory don't mess with it unless you know what you're doing So I didn't go messing around in there. So As far as I can tell there are no configuration files for this extension and there's no settings really either There's just those few key bindings that you can change and that's it If I were a gnom user And I'd never use a tiling window manager before this would be cool But if you're a if you've used tiling window manager before even just floating window managers anything like that But you've had you have experience with things that allow you to customize the crap out of them This is not going to be for you simply because there's just no customizability And that's that that's the bottom line is that it's better than I thought it would be But it's not for me because you can't customize it and I would like to cut The very least I'd like to say hey you want to know it Let me change to do a different key binding for this or this or this And the fact that you can't do that Makes it pretty much unusable for me So in the comment section below if you would tell me if you've used this tiling functionality and whether or not You enjoyed it or not I'd really be happy to hear from you if you would like to follow me on twitter You can do so at the linuxcast you can support me on patreon on patreon.com slash linuxcast before I go I'd like to take a moment to thank my current patrons Today devon chris east coast web gen 2's fun 2 petrico primus marcus maiglin jack snipe tool steve sabergate linux garak michael artster carbon dated shaw and jeremy odin merit camp drash li j dog the beast He's rock peter a crucible and dart bandit six. Thanks everybody for watching. I'll see you next time


Welcome folks here to the Dow finish up to 16 as like up 319 S&Ps up 66 we had talked enough about the market where is it going this is insane folks so listen to this here put Boeing up here for a second and we know that you know Alaska Airlines huge problem you see what just happened here 10 minutes ago 15 minutes ago you know you had Boeing up at 233 it was still down 17 bank as the 228 while what's happened there is this they start inspecting these planes United just come over to press release listen to this man this is sick that they just found on five aircraft okay I don't know how many are going through loose bolts this is the same problem where the nine plug doors are for we call it nine plug door Boeing 727 max nine plug door bottom line is that yeah they have they've already got five aircraft with loose bolts on it it's like are you kidding me man I mean this is like insane I didn't pull up yet how many United planes they actually have but that's where that little dip was down again the Boeing's the mind screw on Boeing is that you got an icon company and they lost it man they lost it I mean out of the amount of 737 max is they're going to get inspected we'll see how many actually could be in trouble and if you heard Tommy show this morning this was amazing that prior to this you know on the last one in my flight the exact problem three alarms had gone off in the past month and what they did is that they turned them off I don't know they might have looked at or something but then that particular plane they says okay we can't put that plane over the water until we find out what's going on it's like really so there's been a lot of lax inside of Boeing inside the FAA in a monster way I mean really and you know as your booking trips we're all going to be looking at okay what type of plane we're on because this is getting kind of strange man folks have a great night have a safe night come back and visit Tommy tomorrow morning kicks us off 9 a.m. great show folks