Okay, good morning, everyone. I'm Matthew, this is Mike. Mike needs an introduction. We're both at Wavefront. We've been at Wavefront for about two years. And we're both on the team that run the service called Wavefront and Found HDB. And this is our story. Before I talk about FDB, I'll tell you a little bit about Wavefront. We're Silicon Valley's best kept secret. We're a SaaS time series metrics, tracing, monitoring platform. And one of the few places running FDB at scale. And so because of that, we've learned a lot of things about running it over the last couple of years. And people like us, we have startups and large SaaS companies as well. So we talked about scale earlier. We run about 470 instances across 80 clusters. Some of these clusters actually span multiple AWS regions. And that for us is about 330, or sorry, 3300 FDB processes. And our largest clusters often exceed about a million writes a second on the memory engine. Since we're all showing off architecture, I'll show you ours. So this is a real simplified view of a Wavefront stack. There's three tiers and applications, sorry, web tier and application tier and then a database tier. And I'll go into there. The web servers here run a small three node key value store running FDB. We share content that's common to both, Wavefront runs two mirrors, so primary, secondary mirror. And this key value store shares, stores content that's common to both sides of the mirror. So things like alert or dashboards or user names. On the app tier, this is where all the magic happens. We take in telemetry. We do stuff with telemetry. We do queries, we do alerting and we store it into the database, which is FDB. On the database side, we run, as I mentioned, we run a memory engine and the SSD engine and a co-processor. And I'll let Mike talk about the rest there. Yeah, so this is a very high level view of how we architect our cluster. Every Wavefront cluster, as he said, is actually two databases. We run the memory and the SSD engine. Actually, it'd be easier if I come out here. On the SSD side, we put a stripe across a set of EBS volumes, just basic GP2 volumes. And we shim in in front of them using Enhanced IO cache volumes. Those cache volumes we use to cache our reads. This is important for AWS and we'll talk about that later. And the writes are, it's just a write through cache. So the writes actually fall straight through into the EBS volumes. The memory engine is not using a stripe. The memory engine has, we can kind of take advantage of being a time series database. We know we're always writing to the tip of the database. So we use write throughput optimized, ST1 disks are actually magnetic volumes, but they're highly tuned for doing sequential reads and writes. Never at the same time, but the FDB memory engine is only ever writing or reading and it's doing reads during recovery and it's doing writes during normal operations because your reads come from memory. And in between the two databases we have on each node, what is an FDB co-processor. Let's actually see if I can, if I have to click over here to advance it or... Where'd it try to go? There we go. The FDB co-processor lives between, the FDB co-processor lives between our SSD and our memory engine. And because this isn't like a Cassandra database where it's consistently hashed, it's an ordered key value store. It's performing compression, sorting, optimizations, aligning the data and the boundaries in the database so that we get the maximum reads and writes out. And it's aware of the operating space available to the memory tier. The memory tier is very sensitive to how much space is available to be written into it. So we have our FDB co-processor that basically can shut down writes to the database, shovel data out and then resume workloads. And we queue all the workloads up, keep everything durable so we don't lose anything. We run our databases with a one-to-one processor count. So earlier we talked about like there were different roles like proxies, resolvers and logs and storage. There's these stateful and stateless transaction processes. Those actually aren't like specific things like you have to run. They're just a denote on a process that this is a transaction process and it can be any one of these roles. So we make sure that we have one-to-one for storage engines and transactions are a two-to-one. So we run two storage processes for every one transaction process on the memory tier. Because we're doing such a high volume of writes, we need lots of proxies and resolvers to push into the transaction logs, which are carrying those permutations. I'll pass it back to you. All right, we run Linux and so we've become experts at running on Linux and experts at running in a somewhat hostile virtual environment. So we've learned a lot of things on how to tune Linux or the machine we're running on around disk IO, around CPU and memory and kernel tuning. So NUMA, it's a thing that no one remembers until Snowflake told us to think about it. So on multi-CPU instances like the i3 16x or 8x, we're pretty prescriptive. We bind a process to a CPU and a memory so we don't have the CPU thrashing across from memory access. You know, Mike touched on our disk layout a little earlier. We're heavy users of a disk caching called Enhanced IO. We focus mostly on read caching and so we have a one-one mapping between the EBS volume and the instance store itself. Mostly because Amazon doesn't discriminate between IOPS and so we sort of cheat. We get free read caching IO.  And what this means in practice is you can see the blue line is our read cache rate across one of the larger clusters we have and the green is the writes which we don't really cache at all. Yeah, so we get about 100% read caching because we shim and NVMe in front of each one. So the only overhead really on reads is just what it costs in FoundationDB because it's lightning fast. The only cost is also the time it takes to build the cache which is not very long, about a day or so. We're also experts at tuning the kernel or we've become experts. These values work for us on our workload. You should experiment. But this is something that we have done. Yeah, it was really important for us to tune the networking layer on the kernel especially for the cluster controller.  It's doing the health checks. It's deciding who's gonna take what role and what roles are gonna be assigned out. If your kernel isn't tuned well, you can actually flood and DDoS your cluster controller and can take the cluster down. And we're not saying it's ever happened to us. No, not on production. But so it was important to make some of these changes so that we could go from being able to run three, 400 process clusters up to much higher now. We're almost 500, 600 on a single cluster just out of tuning the kernel so that we get the best performance. That's all you. All right. So one of the things that's also important if you're gonna operate FDB at scale is the instance lifecycle. It's hugely important because we have some clusters that are operating at up to 20 nodes in a cluster. So being able to ensure that these instances come online and they're ready to go, that they are 100% operational ready is important. So we use Terraform. We have a system called landing party that configures these instances on first boot. And we also make heavy use advanceable so that we can do entire fleet replacements at once. Which is we wanna be able to quickly and easily change out all of the infrastructure. So our database configuration in Terraform is dead simple. It's how many nodes, you know, of what type, how much storage do you want on it? What version of Ubuntu are you running? We were locked to 16.0 for a bit. We've just been unlocked because we had to rewrite some of the Enhanced I.O. Cache's support later kernels to get us on to AWS kernels as well. That enables us with just some very simple helper scripts that will go out there and deploy these databases. And of course, because we're observability, it's important for us to be able to immediately see when these nodes come online and how they actually perform and what it means. You can see new memory storage nodes coming online, how much CPU, you can see a spike in the data that was being processed because there was likely a slight backup as the node joined. So honestly, we wanna let computers do the hard work for us. So that's what landing party does. We have landing party and post boot systems. They actually, when a system comes online, they're gonna go in there. If it's a brand new cluster, it's gonna configure the memory and the SSD tier for us automatically. We don't have to touch it. It just configures new double redundant memory or SSD engines. It's going to prepare the foundation DB configuration files like our NUMA settings. It detects, are we on an I3? Is this a 16X or an 8X? So I need to push out the NUMA configurations. It's gonna get everything ready for us so that all we have to do is turn FTB on. That's the one thing we don't do. We don't let it turn FTB on automatically. The instances join, they're prepared, they're ready to roll. We turn them on with intent. It allows us to stage work. It allows us to prepare a customer to grow their infrastructure and then turn it on under controlled. In the background, there's a process that's taking the cluster to config and pushing it into S3, so on launch, we can pull it back out so the machines kind of come up ready to go. Yeah, so we can, yeah, exactly. So the landing party also pulls down that cluster file so that's synchronized across all of the machines, all of the nodes, so that we don't ever have to manage that either because that can be rather error prone. So fleet replacements, really. Let computers do the hard work for you. The thing with the fleet replacements is it's we need to be able to change the tires on the car, wait, it's to the left, right, yep. While the car is moving, we don't want to have to deal with it. So we have written tooling that actually will go in there and completely replace an entire cluster without any human interaction besides starting it. It goes through, identifies which nodes are to be removed, excludes them, re-coordinates the cluster, checks to make sure the excludes have completed, checks to make sure the coordination states have changed, and let's see if I can crane my head enough to speed this up for you guys. So that's actually what it's going through and doing right now. This is just an ansible job. We wrote, we had been challenged, when I had first started at Wavefront, I was challenged by one of the founders who said it wasn't possible to do this. He said, you're not gonna be able to automate, excluding, and replacing an entire cluster all at once. And he said, all right, so we did. So this allows us to do in-flight OS upgrades without basically hit list upgrades. We can just drop a whole new set of infrastructure, the tooling understands what was new, what was old, and goes through the exclude process without the human interaction piece of it. And at the very end, it disables term protection and turns off the instances for us. Yeah, so yeah, the last thing it does is, before it does that is it does a sanity check, coordinators move for both tiers, our excludes finished. Did we actually exclude the right number of nodes? And if everything passes, disables termination protection, destroys the instances, and then re-includes the excluded IP so that you don't have wasted exclude sitting out there. I love the motivation for the tooling was we learn the hard way what happens if you terminate a cluster coordinator without having a replacement for it. Yeah, which is... It doesn't work after that. Yeah, it doesn't work after that. So this, I believe, yeah. So it's going through the cleanup phase now. So all of this is really like prime candidates. We have these in Jenkins, they're schedulable jobs. We don't want operators to have to remember syntax and have to go in there and do this. This is just a good show for how we, what it looks like, how we do it. Like, yeah, it's only three minutes, it feels like. It's the longest three minutes of your life, man.  What happens if you terminate your node before the excludes finish? Well, if it's only one, it'll heal, it'll re-replicate. If you don't have enough replication factor or it's too many, you have data loss. So these tools were specifically built to remove that human error, get rid of that element of it's possible that I could, you know, fat finger something, forget to exclude something, forget a re-coordination step. The goal was to enable future engineers as they joined the team to not have to worry about those things and just join without there being any concerns. They could focus on operating and not learning the scary parts of FDB. What is it? Oh, there we go. I don't know if that went too far or not. So I don't know how many of you are familiar with the FDB trace logs. This is what the trace logs look like. If you enjoy reading XML, I don't. They're incredibly spartan and I actually found reading the source code for FoundationDB lighter reading than trying to grok these. So we have a tool internally. It's called Wayfront FDB Taylor. It specifically takes this eyesore and it turns it into this so that we can see this is on one of our larger clusters. It takes and it shows you where the roles are assigned, what machines, what process and port they're on, how much data is flowing through overall on each node and then how much per process so that we can find when there's data distribution issues. Sometimes data is not properly distributed. Sometimes the CPU is hot somewhere and your T logs, which are CPU bound are gonna have some issues. It shows us how much storage memory is being used, how much, if there's any transaction processes, how much is being used. This is everything that comes out of the trace logs, but readable. So, which leads to the next point of monitoring, which is basically the bread and butter of what we do. So we dog food, we use our own platform to monitor and internally we have some tools, the Wayfront FDB Taylor and we also have some Python scripts that pull data out and push it up into our monitoring platform. But instead of really just showing you more pictures of dashboards, I think what I'm gonna try and do is just show you live what it looks like on our platform and how it works. So while we figure out the technical aspects of if this is just gonna work, give us a second. So Mike Mitch in the Wayfront FDB Taylor, at the end we'll have a link, that was open sourced today. Yeah, that's a big shout out to Devon sitting over there and Jay Baal right behind him who put in a lot of work to make that possible. Because we weren't sure if that was gonna happen by today. Yeah. So, the big thing in our platform is we had time series, monitoring, alerts, events, tracing. But I think that really what sells me on it and why I love it so much is just how easy it is to see what's going on. Now, this just looking at it doesn't look like an incredible amount but this is actually running a derivative of the roles that are assigned. And we can see immediately that there are changes in the rate. So if the role is a log, the value is always some amount and if that changes, you're gonna see a flick on the value. And we can see a big red box. That's an alert that we have cash errors. There's something just broke. So, this is how we monitor at FDB. We have to know every piece from top to bottom what's working, what's not working. Yeah, there we go. And hence I owe cash errors. We can see which host is dead, which cash is dead. So this was the start of an event where we actually lost an instant store. And we caught it right here. So one of the fun things that AWS will never admit to is that things do break all the time in their environment. It is a very ephemeral cloud. Things come and go all the time. Instant stores will die, EBS volumes will go corrupt. You have to architect to deal with that. So like we put an NVMe in front of each of the EBS volumes. When one of them goes, we have to start, stop the system to get an instant store back. We have to rebuild our caches. But we have the alerts that show us this and we can actually see the FDB data that tells us that, hey, something isn't performing, right? And then here, near the end, you can see a lot more role changes as the cluster is starting to restore order and come back to life. Even in this field, I think it's gonna be fine. So this is more of what comes out of your trace. You can see around 1045 that the latency in FDB starts going awry. We're missing data. It's spiking up and down. The cluster is actually struggling a bit. The storage and log cues, which we had talked a bit about earlier, your log cues, writing the permutations of what's going to eventually make it into the storage cues. Typically should never reach above a threshold. And for our configuration, it's about one gig of data.  This is all stuff that we get out of those, the trace logs from the tailing system. We pull it in. We can see where the SSD is here, which is actually this area right down here. The key value store drops out a couple times. And that is from the initial we lost a drive to we had to restart. One of the other nice things we get is you can see which processes are dying. You have immediate observability into processes are dying, they're restarting. Those are just memory storage notes that went down with the event. And there was, they begin the process of self recovery. This is a very, very durable database. It's very operationally, it's... It's a challenge. It's a challenge, but it's not a challenge because you're trying to keep it from corrupting your data like MySQL or Mongo. It's not a challenge because it has insane defaults. It's a challenge because it's so durable that it favors keeping your things together, which means it's gonna make some hard decisions for you, which is it'll stop the world and recover itself if it thinks it needs to. But the alternative is losing customer data and you don't wanna really afford that. This is actually right. This is one of MRZ's favorite charts. This monitors the file sizes on disk for the transaction processes. It monitors the actual transaction log itself and it monitors the rate of change in the transaction logs or transaction processes. As we are recovering and it's pulling data in and it's reading those files back into memory, the processes are growing in size. They're changing. They're recovering and we need to be able to see that and that black line right there, which is his favorite line, tells you whether or not the transaction logs are still reporting out at FDB. If those stop reporting, FDB is hard down. You have to then go in and do a little bit deeper surgery on FDB, but this is how we can quickly and easily identify and show people, hey, FDB's recovering fine. It'll come back on its own. Don't worry. It's re-reading in T-LOG files from an earlier outage. And some of the other items around this event, you can see we track all the metrics because they matter to us. Let's see if I can... So you can see in the window when it first went out, our IOQ depth, if you're very familiar with IO, it was not suppressing, which means IO is not being read or written. Nothing was happening. You can see where we lost it and it came back, that particular node. So this is all what goes into observability for FoundationDB for us. And without it, I don't know how we would survive, to be honest. And we build alerts on this? Yeah, we do. We actually build a lot of alerts on this. So one of the charts I have scrolled past quite a few times is this PEG CPU process. So what's fun about this chart is so those trace logs are constantly emitting values. And a very important one is the CPU seconds. How much time, CPU time is this process consuming? If a particular process dies and it becomes unreachable by the cluster, it stops emitting that value. Well, the FTB log Taylor actually continues to emit the last value it saw. So we can abuse this knowledge and write alerts that look specifically for processes that are no longer emitting this data. And we can find processes that have fallen out of the cluster and they have fallen out of the cluster for any number of reasons, but we can go identify those processes because the rate of change in that process is now zero. And it allows us to nullify all other processes that are showing rates of change and highlight only the ones that are no longer reporting their CPU time. And we use this, find the process, kill the process and restart it. FTB monitor goes in, restarts the process. Cluster goes back to normal operations. So these are just some of the operational challenges for us, but having those trace logs turned into telemetry enabled us to really dig in and find these kinds of issues where we would typically before be banging our heads against the wall like why isn't this recovering? Why isn't this coming back? We built dashboards, we built alerts and now computers do all the heavy lifting and work for us. And that's all I have on. Yeah, I think that's it. I don't know if you guys have any questions about how we do what we do. You wanna put your phone for the last slide or? Good, sure. I have to check it. So he tries to find the last slide here. It'll have the link to the GitHub repo. Or they're posted too. The dashboard that Mike showed you, we've explored the JSON, it'll be up on the same repo too. So if you were to try it out, you could experiment with the same sort of dashboard that we use in production. I think that's all we got.


What's the word y'all I didn't even man Bitter sweet right of course. I'm rooted for my guy Chris Paul to win a championship and they are two games away from doing that But as the basket literally I always try to tell people there's a hierarchy here, right? above all setup ESPN above all I am a NBA fan. I am an NBA fan before I'm a Chris Paul fan before I'm a of Dare Rose fan even before I am a Bulls fan. I am an NBA Basketball fan, right? So even though my favorite player of all time is just two games away from getting this first ring after 79 years of playing I'm not enjoying myself because the overall product of the finals so far. Have it been good one team is Bringing what they supposed to giving the fans something to watch. They're living up to their side of the bargain It's the other It's the other man. Listen, I made a tweet early This is crazy that we have to it literally explain every part of a tweet and I think most people understood Devon Booker had a move where Jeff T did one at ease like he was chasing his own tail and I tweeted Devon Booker Showing his whole bag today or something like that go through my menses, bro. Kenny. He's three for 12 right now What did he end though what did he end though and I'm not saying that he was hella efficient What I'm saying that the bunkers that he was getting were like showing his bag showing his whole offensive repertoire I guess what he did he ended with 12 out of 25 7 May 3 anyway One team is living up to the bar to their side of the bargain very early on even though the milky Bucks had I think at one point like an eight nine point lead in the first half first quarter I mean my guys it's six of us in in the party I was like the Sun's you're gonna win this game and you know why because the McKell bridges hit two threes in the first quarter Jay Crowder hit a three in the first quarter and Tori Craig hit a three in the first quarter if you did not know There's the statistic I think I heard on on Zach Lowe's podcast so try to be given credit But I might I might be wrong him before the finals even started He said that when Jay Crowder scores over 10 points. They are undefeated in the playoffs How many points he had tonight 11? Streak lives on So even though the Bucks had started the game off as the hotter team I just knew if McKell is gonna hit his shots if Jay is gonna hit his shots of Tori Craig is gonna hit his shots and Cam Johnson gave two threes is over with right? So let's talk about first of all sons. Congratulations.  He is making himself more money every time he plays the Andre Aiden didn't wasn't the efficient monster that we normally see But he played a crazy amount of minutes because Darryl Sarge is not there You don't really trust Frank Kaminsky, but he played he played well Chris Paul had his moments where the son the Bucks are going on the run and Chris Paul made quiet in a little bit He was a little loose with that ball a lot looser with that ball today Then he has been in this entire playoff run That's you can't do that but luckily they got it there And of course like I said Devin Booker ahead ahead of many many moments in this one But I'm disappointed The Bucks ain't doing it man I Swear to God if I see if I open my Twitter and I don't really follow people that be doing stuff like this Have I see any Yanis slander after today? It's a rat, bro. I'm sorry I don't know how you can watch tonight's game and Slander Yanis this man gave his all on the offensive side of the ball ended with 42 points He shot 11 for 18 for three I know it's only 62% but for Yanis to shoot 62% on almost 20 free throws is amazing for your team He had 42 points. He had 12 rebounds for assists and he showed Defensive player of the year things today the help defense was ridiculous He ended with three blocks on the steal the defense was amazing today He gave it his all all he was asking for all he needed was for one of his other two Offensive minded players to step up and none of them did we got we got Chris Middleton five for 16 One for six from three and didn't get to the line at all You would think hey my mid-range can test the three pointers all of that ain't work And let me try to get to the basket a little bit. Let's see an easy one Gonna let's go draw some contact so I could get to the free throw line right not one time Chris and if this Series is gonna be anything we need Chris Middleton from game three last series I think it was game three. We need Chris Middleton from game six. We need a quarter Chris Yanis needs a quarter. They had a little run They had a little run in the fourth, but I think they got it all the way down to five Yanis comes out because his ankle or something is messing up He's taking off the shoes the train is looking at him immediately three pointer sons and one sons another basket It's back up to over 10 He couldn't sit for a single minute and I mentioned this yesterday on the last video Yanis typically is not a guy that's gonna give you 40 minutes So the fact that he gave you a 40 minutes of elite elite's best player on the court basketball And y'all still lost by 10 is ridiculous. You also got your holiday Who y'all traded a lot for you gave a bag to because he was supposed to be that third player Eric Bledtoe had been disappointing for two playoff runs Now you bring drew in and anytime drew plays and Yanis are playing in the same game Drew just can't do anything I tweeted a video This is this is fourth quarter at this point. I think it's a seven-point game Drew Holly is on a left wing. He passes it to the corner. He gets it right back pump fake two times on the defender He was thinking way too much in this game. It looked like he was getting his own head. I love you I do believe that you how there's a very good NBA player I'm not slandering you holiday, but he needs to play better because today Again as a basketball fan above all his performance and Chris's performance was Infuriating when you think about what Yanis gave y'all in the fact that y'all didn't really even have a chance They had their runs like I said, it was down to five at one point, but it never felt like they were actually in the game Packon it's a good performance.  I even made a tweet I had to go look this up because because I think it was my guy snail season shout out to snail season I feel like I'm shouting him out on like every podcast every episode But hey, well, I had tweeted something about Chris Middleton at one point had seven points and PJ Tucker also at seven points I'm like, bro. That's unacceptable and I think snail season was like it's way too much isolation So I was like, yeah, it is way too much isolation. Let me go look up some statistics They are passing the ball less than any other team that has made it to the finals in the last 10 years and probably longer But I can only see statistic from the last 10 years because how who the heck is tracking the amount of passes being thrown Somebody is and it probably just started 10 years ago The least amount of passes is heavy isolation Where Chris Middleton is settling for a contested to it is Yana's burling his way to the back Is that the word going straight to the basket and injuring people literally injuring Tory Craig get well So sorry that like me and my guys in there are three players in a league at the minimum three These are the top three guys you don't want to be in front of when they're running LeBron Yannis Zion I'm not trying to be in front of anybody.  I think he even drew the charge too. So sacrificing his body for extra possession. I Don't even know where I was going. I Just can't believe through two games. The team has disappointed that much I'm not saying it's over like again. I won't say it's over until it's actually over bro I'll let's okay. It's over if next game are they lose if they lose game three in home in Milwaukee Call it a wrap Yannis get well soon get get that knee well, but it ain't looking good Let's address some of the stuff that happened in this game early on Because because the weird thing about Maybe a fandom I guess is that when you announce that you are a die-hard fan of a person or a team Anything that person a team do they own your back about it and earlier in this game Chris Paul had a terrible play terrible play on Yannis, right and people were coming at me like Kenny This is your boy. This is what he's doing. He is yes He is still in my boy, but if you watch any video I talk about Chris Paul when he does this extra trying to get under your skin Flopping trying to get the extra calls That is the worst part about Chris Paul's game and I hate it But all the good things kind of outweigh that in my personal opinion. That's why he's still my favorite But like no, there's there's no I said something about Patrick Beverly when Patrick Beverly did the Chris Paul push There's no world for that to be acceptable whether you're my favorite player or my least favorite player Things like that should not be acceptable Chris And luckily Yannis was okay. He was okay But those things cannot happen on the basketball court because I know you want that ring big fella I know I know you do Don't you also want to beat the team at their absolute best imagine hypothetically if Yannis goes down. He can't play no more They already seeing you getting a fake ring.  Nobody cares. I ring is a ring, especially when you're playing so well Game three. We got a couple days in between game two and game three. Actually, I think it's like to Thursday Oh, no next Wednesday. I think the actual game. Yep. Next Wednesday. Here's the thing ladies and gentlemen I said I want to be there for game number four was on Sunday or Saturday or whatever And they happen I Know sad Sad something came up and I also don't want to spend $3,000 on two tickets. That's That's also it. I would it would be kind of bad though If it was a Sun sweep and I could have been in the arena for game four to see it happen Memories are priceless is what I definitely heard somebody say But you're not the one spending the $3,000. It would be me So they they have a price on them There's a limit now if this was my team if it was the Bulls hell. Yeah, I'm paying the three. I'm paying more I want to be corset when it's happening That's not my team This is a team I mess with I'm not a Suns fan if Chris Paul left for free agency. I would still enjoy watching the Suns But I'm not rooting for the Suns anymore. It's Chris Paul. So you know I'm saying I got players on the team I mess Michael Bridges is the homie Devon Booker is not the homie, but it's cool But if Chris Paul left they would just be another one of those teams. I enjoy watching but not really rooting for or gets There's a tweet from Kevin Arant that I I threw in my own DMs Because I figured it would come relevant eventually It's something along the lines of like NBA fans don't even know what NBA fans want and I I understand that Understood a thing, but I understand them more Right obviously we've mentioned it a thousand times these two teams will not be here if there wasn't for injuries But on the same tip us NBA fans have always said bro, we wanted a little bit of parody It's unfortunate that it happened due to injury. I'm saying that we want a little bit of parody We're tired of seeing the same teams But now we see something a little bit different and may hey you you have the opportunity or you have the freedom to change Your opinion maybe you're like man Actually missing LeBron in the finals, which is cool It's just cool, but even if there's a weird bro We got two teams who's never been there before both trying to win it all and everybody's just talking about how Bad it is and it looks the product ain't great, but I think it's still okay for the league to see two teams who ain't been there before Fight it out the statistics of gate number one came out and some of them were good Some of them were bad But what I always want to tell people is like looking at the overall viewing numbers of an NBA game Don't really showcase the the engagement of it because I know of plenty of casual NBA fans that are keeping up with the finals But they're watching a 10-minute recap, you know I'm saying so just sitting down and watching it on TV ain't for everybody Especially in 2021 where we're vine hit was that half a decade ago and people Attention-spanning six seconds tiktok got it at a minute. I watched a three-minute tiktok today in the whole comments section was like Why is this so long? It was a very interesting piece about some birds. How did I end up on bird tiktok? I don't know it was an interesting video, but all the commenters is like this video's too long. I don't understand it I've been seeing the exact opposite. You're watching this video with 13 minutes in I don't even talk about basketball anymore I'm talking about a bird video. I saw a tiktok So you're the you're the viewer base is actually watching the game, but everybody's not there But they still keep up with the league Like I don't I honestly don't know if My dad is watching every minute of every NBA game, but he's definitely keeping up with it Whether they be watching my podcast watching these updates He's 100% keeping up with it, but I don't know if he said it now watching the fool 48 Either way, um, I hope game three is a lot better. It better be a lot better. I'm gonna be upset Milwaukee This was the a Amen walkie. I'm sorry. I'm going back to this honestly real quick. This was the game Where you look up to the gods and you say Thank you for letting or making Yana sign that extension this offseason because him putting a 42 in losing in the finals Is the is the moment that LeBron had in and Cleveland where he's like, you know what I Can't do this no more dog You got lucky you lucked out because he probably rethinking that extension right now I'm giving the city my all and I ain't got one mother one dude other than White man can't jump back on it's not that the two people to three people It's three now because I'm counting you Brooke Don't think you can off easily the three people who've been an all-star in the past It's insane Okay, I have another channel. I have a baseball channel. I'll put that link in the description if you're interested in baseball I highly doubt it. There's overlap here, but I ain't upload the video yet. It will be tomorrow What is the Wayne way doing it? Why is the way way to every game Devon Booker depths of Dwayne wait? He's just got that that private jet money. That's what it really is when you have a problem You have a private jet. You can just go anywhere.  I wasn't blessed with six three sides and crazy athleticism All right, that's too much talking


with what Megan did and what she said on camera about her family. I grieve a lot. I mean, I've lost my father. I grew up as an only child, and I wished I had siblings. I would be probably more upset. I don't think I would have it as together as these people if this was my own family doing this. So I understand the anger. I don't think they did anything first. I think the behavior here is honest. It's angry. It's spiteful. It's reacting to hurt, but I think it's honest. She was mourning the loss of her father. Sounds as though you date. Yeah, exactly. She's helping and then mourning the pain. Megan literally said she doesn't have siblings. She lost her father, which any rational human being, we use the rational person standard, would think that that means that somebody died. As you can see, Thomas's face here in what I think is actual emotional pain. I don't see a lot of anger here. It's more sadness than anything, and I believe right now he would still take her back without hesitation, just based on his behavior here in this clip.


Sailor's fleet activity, Sausubo, took part in an active shooter simulation in the base gym during exercise Citadel Pacific. Sea fast security, fire department and medical personnel responded to the active shooter and were able to train for a potentially deadly situation. The best part about it is just being able to really kind of help us know what we need to do in an actual threat like this. So being able to do it out here, this one, it's adventurous. Everything went well. It's always nice when we can have all these agencies work together to achieve a common goal. Active shooter is a big thing going on everywhere and around the world. So for us to be able to play our part, get our training in and be able to respond and react is really good. Petty Officer Jeremy Graham, Fleet Activities, Sausubo, Japan.


So, good afternoon everybody, I'm glad to see you all here, the one observation that I was able to make, not a lot of young people here, probably that is a little bit too low level stuff for youngsters, but anyways, that's good, somebody join me on this presentation. Well, I'll be talking about Uboots on quite memory restricted platform, and I call this Uboots bootloader for IoT platform with the question mark, so we'll see if it is feasible. So, let me start them. First, I'll start from introduction, my name is Alexey Brodkin, I work for Synopsys as an engineering manager, lead in a group of, quite small group of engineers doing development of open source projects for architecture. And even if we are not talking about Synopsys, personally I use open source software a lot in my spare time and doing my day job, so whenever I face a problem I try to debug that, fix it and I then contribute my fixes upstream, so that's how I got involved in a couple of open source projects and some of them are listed here. Mostly notable, most notable of them, Linux kernel build routes, recently I started to contribute and open embeddits, but regarding that presentation what is important, I made a port of Uboot bootloader for architecture a couple of years ago and since then I've been maintaining that, enhancing and all that. So that's our agenda for today, I'll start from introduction of my target platform that I'm going to talk, then I'll explain what I had to do to strip down memory footprint so that my Uboot build sees usable on that platform and then I'll talk a little bit about how to execute Uboot from run because by default this is not what usually happens. And in the end I'll go through two issues I faced during first execution of my Uboot port on that board, so let's start then. Uboot, it is basically a bootloader and it's main reason to exist to load some real payload and start it on the board. And typically it would be some kind of OS, it might be R-TOS or what's at least here we may think of it will be Linux kernel plus RouterFest and all that. And initially I added support for AXS-1 release board which was quite interesting development board, basically that's it but with a different CPU. What was important it was 600 MHz CPU with a lot of memory you see these mention 2 GB of DDR, then I added support for that board with a little bit different CPU which runs a little bit slower but still I saw that it is more than enough for running Uboot bootloader. And so then last year we've gotten Uboots which is even more powerful it is quad core RKHS38 CPU running at 1 MHz and it has even more memory. And that is very typical situation when we use Uboots it is so-called single board computer we use Uboots there every day. But there is such a thing which is called low of an instrument or also known as low of a hammer which sounds like if all you have is a hammer everything looks like a nail. And that happens to me in that sense that whatever new boards I have on my desk I try to port Uboot on that and try to execute it and see if there is some use of that. So that's what happens I was given a new board with idea to use bootloader and package that into a real product that we are going to ship to our users.  So it is called IoT development kit and it is meant to be used as a platform for software development and debugging in areas like sensor fusion, voice recognition, face detection and something else. And idea to use a bootloader was to allow users to run their own applications without additional debugging tools like you don't need to use JTEC whatsoever. You just build your application put on an SD card or on USB storage plug that into that board you may see that there is USB OTG connector and so there is SD card connector but SD card slot but probably you don't see it on the other side. And you may just run it without additional tools which at least for me looks quite nice because you don't need any extra. That basically meant we needed a bootloader which may support this peripherals, may support fat partition which is usually used by developers on Windows machines and so here I think Uboot is quite a good match. Now speaking about things which are important for Uboots like we see here that CPU runs at 150 MHz and from my experience with my previous board I know that 150 MHz is more than enough to start Uboot as fast that you don't even recognize that it will be quite fast. But what differs a lot from previous boards is amount of memory which is available on this board. So what we have here first is E-Flash. This is quite an interesting flash memory which we may use for direct code execution right from it. So that's good until it is relatively big compared to all the others but the problem is you cannot write to the random address in this flash. So right there first you need to raise a page which means you cannot use it as random access memory. So we may use it for code storage but we cannot use it for runtime variables and something like that. Then we have ICCM which stands for Instruction Closing Coupled Memory. This is interesting on chip memory which is mostly supposed to be used for code storage and we may execute code from there but nobody actually stops us from putting data there as well and so that's why for us it is pretty much normal random access memory which is again relatively large here. Then there comes SRAM. SRAM is SRAM. It is random access memory on chip but it is quite a bit smaller and so then here comes DCCM which is pretty much the same as ICCM but this is data closely coupled memory. And there is one but very important difference between them. DCCM is not connected to Instruction FHQ which means we cannot execute code from that memory. And which means this is random access memory but it is not very flexible and so what I'm going to do on that board I will use flash memory which is not usable for writing data as a storage for our code constants and initially even data before it gets copied and I will use DCCM for data which is used by UBoot. That gives us possibility to allow user to use ICCM for loading their applications and starting their application from there and if they want they may use SRAM as well. But anyway so once you load some application with UBoot you don't need UBoot any longer and you may reuse DCCM as well if you want. Also we have a spare flash it is good to store something but again this is not RAM so it is not usable. As for peripherals on this board we have a plan to them but in terms of UBoot what is important is SD card controller and this is designed where mobile storage controller USB OTG this is designed where all USB OTG controller and standard UART. So that's nice and so while using UBoot here again we have support of pretty much everything here. You may see a lot of architectures including ARC. We have support of different peripherals. We have support of the design where UART. We have support of design where USB OTG controller and we have support of SD card as well so nothing to do here. We have support of different file system and so FAT file system is here as well so again nothing to be done. What else it supports networking protocol but we don't have networking on these boards but anyway so that's an interesting benefit. And keeping all that in mind it allowed me to add support for these boards created UBoot for that board literally in a couple of minutes. And you may see that's a change lock of what I had to do and I tried to highlight a couple of things. So what is important? You add boards here, you add device tree description here and you add configuration here. The minimal stuff you need to do anyways for any boards and everything else here. It is only required if you need to implement some per boards which I had to do but anyways this was quite simple. Now I said so that it was a working port. Well it could have been working with the only important differences. We are not actually feeling in memories that we have. As you may see result in binary would be of size 400k and we cannot squeeze it into 256 kilobytes of ROM that we have. So that basically means we need to do something about it and we are going to work towards shrinking the memory footprint of that UBoot build. So what we may do, the simplest thing we may do we may just go and tweak our configuration. Thanks to the fact that for quite some time already UBoot uses Kconfig the same configuration utility which is used in Linux kernel. It is a matter of firing up that like make menu config and then go through the options and disable stuff that we don't need. So for example we don't have internet controller so we remove networking. We are not going to start any operational system so that's why we get rid of that as well. We are not going to load stuff through serial console because we have peripherals which are much more convenient. And we are not going to load L files because to load L file you need first to load L file in one memory and then extract sections from this L file to another memory. We don't have enough of memory to play with all that. We expect that user will load binary and just execute it. So we've got it removed and already you may see we have a little bit different number here. So resulting binary will be smaller but still we are not yet there. So what we may do else, okay, we may try to squeeze a little bit of size with help of a tool chain. I understood that something was missing in our port of view boot and what it was. It was that interesting feature when we asked compiler to put each and every variable in function in a separate section and then we asked linker to throw away those sections which are not referred in any other sections. It is very simple thing but for some reason, well I know why because previously I had way too much memory so I didn't think about that, so I didn't do that. And so now I added corresponding options in CPP flags and LD flags and if you want you may click on that committee if you download slides and you'll see how it was done. And so then I realized that actually most of the other architectures they already do that except ARC and for some reason another architecture where it is not supported is MicroBlase. I'm not sure probably tool chain doesn't support that because support is required or probably it was never required because they always have a lot of memory. So with that nice thing I was able to shrink another 5% of size and we are getting even closer but still we are not yet there. And the interesting thing for me was immediately like you see BSS which is more than 100 kilobytes and BSS it is basically not initialized statically allocated buffers like what's of that size. Okay so let's do some analysis and see what that could be. And with very simple NM utility I was able to figure out okay there are two buffers which is 64 kilobytes and so well that was not very good because I was very limited in memory so when I started to look at them I realized okay they are of size of this variable which is defined fortunately for me it was another configuration option so what I did I went back to configuration utility I just set another size and boom I was able to save 100 kilobytes and so already that allowed us to put that resulting image in our room and we don't need already a lot of memory for days actually because it is 7k here and 80k here. A little bit of background actually that initial value of 64k this is on a safety side because in theory FAT file system may have such a big cluster but in reality I don't think there are many cases when it is larger than 4k so 4k is actually quite safe for setting at least for me I was quite happy with that but still by default it is 64k and if you have gigabyte of memory you don't really care. So it looks like we were able to significantly reduce our memory footprint you see from 422 kilobytes to less than 200 kilobytes which is quite impressive and what helped us the most significant result we were able to get after analysis of our resulting binary and I used just size and an M here that's what I showed but also during my experiments I used quite a lot of blood on meter that's a very nice utility which allows you to compare two different health files and it will show you how different symbols became bigger or smaller and how much of a memory you were able to save like it will show you now you are plus 15% compared to what you were before another thing which helped I was practical I just removed all the crap that I didn't want to use and so with some tweaks of a toolchain I was able to save some space as well what could be also tried it might be LTO link time optimization I tried to play with it a little bit but for that I needed to make a bit more changes to you would build system because link time optimization requires to use GCC driver all the time even when we invoke linker or archaver or anybody else and so that requires more changes and I didn't expect to get a lot of achievements here and I was already achieving what I wanted to have so I just decided okay probably that will be the next exercise so like we are pretty much done but there is another thing we want to use we want to execute U-boot solely from ROM and this is not what usually happens usually U-boot gets self-relocated probably not all of you know about that so I will try to show you what happens there so that what happens first we load U-boots into some initial location it might be ROM it might be even start of the RAM or some other location and there we load entire image and then what happens U-boot gets copied in the very end of available memory and start to locate needed memory for heaps, tech, environment and something else this is required because on some platforms for example on x86 and on many others you don't have DDR prepared for usage and you have to set its clocks you have to train it and to do something else which means you need to execute from some safe location which is not DDR for example in x86 what they do they lock a couple of cache lines and execute from there it is not so something people may expect to happen but these hacks are really happening and then even if you start from the very beginning of the DDR already there was a pre-boot loader which we have for example on our board that I mentioned before you want to still move U-boot here because in the beginning of the memory you'd like to put real payloads which will occupy entire memory when U-boot execution is done so there are a couple of reasons for U-boots to relocate well and actually there is another reason because our data is still somewhere here and we cannot write to this area when we are in ROM which means we still need to relocate at least data and that will be still a requirement for us and anyways that kind of relocation works quite fine when you have a lot of memory but in our case it looks like that you see the difference so that's a U-boot bin and that's our available random access memory which means most probably we won't be able to copy ourselves together to that area and do something else so we need to keep everything that we can in ROM and only put required stuff in RAM area so what we need there? We need to have heap there essentially stack data environment and we will load payload but we are not going to squeeze payload here because we have a separate location for that but that's just for generic case and for that to happen what we need to do we need to skip relocation even though it sounds quite simple it requires quite a few architecture specific fixes and generic codes but it doesn't work right away because different code paths should be used and even though there is a generic flag which we may use to say that we want to skip but then we still need to have a couple of defines here and there again if you download these slides you may have that link where you may find all the changes I had to do actually not that many but anyways that had to be done so once we have that modification we signally in our platform code that we don't want to do that for allocation and we only will do some data copy later on and so then we may execute from real ROM and let's see what we need to do initially we have our entire U-boot bin where is the pointer? initially this is our ROM and so we have everything here we have interrupt vector table we have text section here we have raw data and even data here because this is our initial U-boot bin because data it is initialized data we cannot just get rid of that we need to still have it even before we start execution and then we need to copy data somewhere in our ROM it sounds simple thing but there are two questions we need to answer first question is ok so we need to copy data but where are we going to copy that because we have relatively large ROM at which location in the beginning in the end or where the second question ok so how can we have such a strange situation when linker creates U-boot bin with data here but then we move somewhere and from this code we still may refer to variables for example in that area in completely different memory region so let's try to answer that to answer the first question we need to understand what U-boot does during initialization in terms of memory allocation first thing it does it essentially allocates its space for itself if we are going to relocate so somewhere here it would be place for entire U-boot binary and in fact it was happening even if we signal that we wanted to relocate if we wanted to not relocate so I had to change that and not relocate that again for bigger platforms with more memory we don't care where we allocate here 200 kilobytes here 200 kilobytes and don't care but here I have to be very careful then once that allocation done or not done we allocate space for environment and we know the size of that allocation we specify it in our configuration header you see config underscore it is either kconfig variable or something set in our configuration header then we allocate space for malloc we sometimes call it heap and again we know its length already and also what is important we need to allocate somewhere space for a stack and this is done by simply pointing to initial stack pointer allocation and in my implementation I decided to put it here in the very beginning of the RAM and so there is a good reason for that and on the next slide somewhere I will explain what was the beauty of putting stack here so from that picture I think that's clear so if we use that value which we all anyways as I mentioned set in configuration header we may use it as a point where we load data so that problem is solved and answering the second question how we deal with data which is linked here but then copy it here it turns out that GCC, GNU-LD may already do that and they use concept of different virtual address and load address if you are interested again you may follow that link and get some more information but what do we need to do for that unfortunately that's not the very good position for me so first we define two different memory regions so this is a ROM and so we set its original size and this is RAM again we set original size and then for each and every section we tell linker where to put it so for example IVT, text and raw data we put in ROM and that's the interesting part we ask linker to put data in RAM but then put it in ROM and what happens we use addresses which match those addresses so compiler use some addresses to access stuff in that section but linker will put it in a different area in expectation that it will be copied to virtual addresses that we use during compilation later on during execution start up and so yeah then there is BSS section which we don't put somewhere in ROM because we don't need it there and it won't even appear in our UBIN because BSS it is nothing we will just allocate some memory region in runtime and we'll zero it so we don't need to mess with that what is also important here we put a couple of link time defined symbols for example this is RAM start, RAM end, BSS start, BSS end this will be required and this one is also important by the way ROM end this will be used for runtime tricks we are going to do early on start and here you may see how I calculated a couple of different values so it requires some time to understand what's going on there so I wanted to do that thing RAM data size, this is not entire size of the RAM but this is size well which will be used by exactly data, BSS, heap and environment so this is that area but not entire RAM and some other variables and that's what we have in memory layout if we use variables or constants we define by linker you see this is ROM base so this is quite clear it was a text base before this is ROM end and this is important because that will address where data exists before copy so that's the place where we use to start copying data and here you see RAM start is a location where we are going to copy so basically we copy in a cycle everything from here to here and so we know the size as well so we may calculate it even during compilation and so that is very simple and we know as well location of BSS so we may easily zero it later on and that's what we need to do basically first we set that flag, gd, flag, skip, log and so I'm doing that here and note to the name of the function which is executed very early on startup so we signal that we are not going to allocate and then we copy data with use of those symbols that we defined previously and so finally we zero BSS actually it is not entirely correct it will happen in a different function so I put it here as a matter of demonstration but this step happens anyway even if we do relocation so it looks like we are ready for the first execution of Uboot on our board and now we are going to execute it and see what's going on so in fact console works but not everything working as expected so we want to start using USB and first thing you need to do you say USB start and what happens? well it says error minus 12 like what? ok and error minus 12 essentially this is you know mem which means we don't have enough memory during allocation in our heap area and so what helps here if we take a look at our backtrace we may see a couple of interesting things so we start with USB initialization then we do device probe and then we do allocate P which at least for me hinted that each device has its own private structure which is used for a storage of device specific information and I thought ok so if I look at corresponding driver at its private structure probably there is something interesting and indeed I see there is some buffer of size DWC data buff size ok and if we look at its definition we see that indeed this is another 64k and so then after quite a short discussion with Marek he hinted me that actually this is not a strong requirement this is a buffer which is used for data exchange via USB and we actually make it we may make it much smaller and even if we want transfer a larger amount of data it will be just split into smaller buffers and that's fine and so what I did then then I added another K config option which allows you to fine tune that value to whatever you want so 64k was way too much for me so I set 16k and since then it was working perfectly fine what helped here that I was able to look at the backtrace and well actually some knowledge on how drivers work in modern U-boots and so this driver is DM so what is it? driver model implementation because if you use legacy implementation it will work a little bit differently so now when we get that thing fixed I faced another issue when I tried to access finally FAT file system so what I saw it was quite interesting because nothing happened fortunately I have access through JTAG to that board and I said ok I'm in memory exception so how come that memory exception happened and so then I took a look at registers and noticed that my stack pointer register which is a separate register at least on our CPUs pointing way outside of our RAM area and if you return to that slide you see so it was pointing somewhere here and that is the beauty of stack being here because there is no memory here and my memory subsystem just returned an exception if I have put that stack area somewhere here for example after data you may imagine what would happen I would silently start to corrupt my data and somewhere later down the line like during execution of something else I would face significantly more significantly worse situation when I need to debug something which might not even be reproducible easily so that's why you either put it here and so expect an exception here to get when you access in this area or if you have like memory protection units you may use it as well it will definitely help yes so also what may be used as a hint so yeah we understood what was the reason so then well not the reason what happened and what was the reason again if we look at a back trace we see that last function which was execution is part test does and in that function if we look at that we see in the very beginning allocation on stack indeed of something this is a couple of macros which are wrapped around each other and so in the end so that thing if simplified looks like that so what it does it allocates memory of size of that structure multiplied by amount provided here and here sector size is 512 512 bytes and so it is like what it is 78 and if we multiply that we get 40 kilobytes but if we look at that function in that function a little bit down we see that's actually what we want to do we just want to see file system magic number which occupies like a couple of bytes we don't need actually 40 kilobytes here and when I took a look at the Git history I understood that there was a previous fix that changed allocation function from something else and semantics of that function was different and actually what that guy wanted to do he wanted to put here one because we want to allocate only size of that structure and so with that fix which is mentioned here as well I've got this problem resolved as well and so from that exercise I concluded that you have to be very careful when you see something like that because if your memory can restrict it a lot you may allocate that here a little bit more than you expected so in the end I've got a working system and so those goals that were mentioned in the beginning they were met so you may now load image start it and play with that with anything and basically the main conclusion of all that exercise is you would is perfectly usable on a platform with not that much of a memory like for this particular case I was able to squeeze it into 200 kilobytes of ROM and 100 kilobytes of RAM and still with that I've got drivers for serial ports USB, MMC and support of a FAT file system not even for reading but for writing as well because we expect user to keep environment on SD card or a USB storage and so if you have tools that really helps a lot when you are trying to analyze where you may strip down memory a little bit more still sometimes you need to enhance existing generic U-Boots codes or sometimes even fix stuff because things are done wrong in some places but anyways that helps and what also is good to remember that when you are fighting with a limited memory situation allocation happens not only statically during compilation but in runtime as well and it happens not only in heap but in stack as well and it is always very complicated problem when you have stack over flow another thing that I want to mention in that particular situation you need to bypass U-Boot relocation and unfortunately that requires some work if you are not using architecture which most probably you don't do because there are not that many boards without architecture but hopefully at some point people start using it more you need to patch code and hopefully send patches in upstream as well and so yeah the last conclusion is most of runtime problems are due to problems with memory so I think that's pretty much it that I wanted to tell you about my excursion with U-Boots on that platform thank you all for being here and also leaving me here alone in the middle of the session any questions so far? at the very beginning you started removing some features unnecessary and you removed an ability to boot any operating system so what this bootloader will do after it will boot so what's the purpose? the world doesn't tend to on the operating system right? the point is you built your bare metal application which does I don't know sensor fusion whatever and it is 1200k in size I mean entire binary you may produce with a compiler not only L file or U image you may build binary image of your application then you load it into memory so we return even further so you load it it's what is my pointer so you load it say here your application and execute it and do whatever you want with U-Boots on U-Boot parlance what you do you say fat loads mmc0 michaelab.bin and provide address and then what you do you say go to the same address and what happens you would just jump to that location and start to execute your application from this point there is no U-Boots any further it might still exist somewhere but you pass control to a different application and so that's the reason for bootloader to exist so this option doesn't prevent from running a bare metal no roll binaries so this option doesn't prevent U-Boot to from running bare executables right? again what is the question? this option doesn't prevent U-Boot to run no that is very specific option which does quite a lot of things in preparation for starting Linux kernel yeah and then loading stuff changing like device triblop and then doing a lot of checks prepares registers caches and then passes control Go command is very simple it is implemented as jump okay thank you I have a really simple question how small can you actually make U-Boot be the binary? can you strip it down say to 64 kilos is that possible? is there some mechanics for that? well yeah you see what I had to do here I had those limitations and fortunately actually in the beginning I was not 100% sure but I said okay I will do that and fortunately I was able to squeeze what I have in what I wanted in what I have but can we go even further so if you take a look at yeah so if you like remove the USB from the file system and so on how small can you get? well I need to try again I had to implement something which supports both SD and USB and essentially it takes especially USB and file system it takes quite a lot but then the question is if we remove all the peripherals and leave only U-Boots core why do we need that? it's just a thought experiment really how small can you get? well yeah so we may well actually there are systems that have SRAM limitations and you need to load U-Boot into that and execute it but for that case we have SPL we have all those tricks which will strip down all the print case print apps and all that so yeah that will be the thing I would use in that situation can you speak a little bit about SPL? well it was not implemented here because it is not what we wanted because SPL is not meant to be used as a full-blown bootloader that we wanted it does one simple thing but I think you're not better than right here you're kidding I actually want you to say that to the audience about SPL the spoiler is we may have very even more stripped down U-Boots it does only one thing that we wanted to do it loads from only one media typically and it doesn't throw debug messages but it is very tiny so it could be fitting like one page of something and for example I think on iMix platform what they do they have SPL which fits into one page of memory like 4k or something no it's 32 32? well anyway quite quite small I do believe you can make a boot as small as one jump instruction no no and what Marek was saying about it is a practical thing because what happens you have a multi multi-stage boot first you have boot ROM which actually does pretty much nothing it might be a couple of lines of code which copy stuff for example from SPI word by word then you start execution from the first page for example as on x86 you have cache lines in which you log, you load and you have 32 bits k no more then you start execution from there and then with SPL you initialized your DDR and then you may load real full scale U-Boot from SD card from that DDR because you already have driver for SD card and file system which you are not able to have in a pre-boot loader because you only had a couple of lines of code there so you see on each stage we are becoming more and more powerful and in the end we may do whatever you want so that's beauty so what's your long term plan here for maintenance is it all going to be upstream it's already upstream and are you going to maintain these options which options to be able to strip off different features and strip off well you can strip features yourself that's configuration for your boards well that's my board that's my configuration if somebody wants me to add another feature ok I may add it if somebody says that we don't need USB there any longer I will just strip it down essentially I will support that because I'm a maintainer of architecture that's my board so it's my responsibility that's my child there are framework related stuff as well where you create each symbol goes to a different section and then you strip off at the linking time where you realize the symbols which are not being referenced anywhere and then you strip it off I would guess that will be at somewhat more framework level rather than your platform board yeah as I told it should be generic feature but probably back in the day it was not supported a couple of years ago and so then it became supported by I only added that feature now because again for previous solutions it was not needed I had enough memory and for for example microblades we may ask me Charles Simic I think I saw him if he just didn't think about that probably that's the reason and if it is possible to be done on microblades then it is no brainer we should get it into generic code and actually that's a good idea but first we need to figure out what's wrong with microblades so yeah that's the next step indeed ok thank you very much