Everybody welcome. This is cinder 102 pouring a solid foundation for block store services. My name is Jeff apple white I'm a technical marketing engineer with net app and so glad y'all can be here today This is not the fire trail session contrary to what's on the board outside. So Good to see everybody So I was kind of astounded when I saw the hands that went up when they first You know they asked how many people are new at the summit and just you know hundreds of hands went up near me so When I saw that I thought well It might be good to kind of capture the new people that are here and so even though this is cinder 102 I'm gonna do a one-on-one recap for just for those of you who might be completely new to sender I'm not gonna spend a ton of time on that So I don't want to scare anybody away by covering 101 But just just a brief kind of overview of that and then we're gonna move into the other things that are on the Agenda still have plenty of time for all that So Here we go try that again Getting a lag here Sorry, my slides are lagging here still not happening for me. I could turn my laptop around, but I don't think anybody could see it Can I get some technical help that's a popular slide that's not the one I'm looking for but Okay Let's see does the HDMI having that worth trying there was Now there we go great HDMI all right, so the agenda today is as I said, I'm gonna do a brief sender orientation talk about You know the basics of block storage what it is and I'm also gonna talk a little bit about what it isn't because Honestly when I came into open stack It was a little bit confusing of what exactly is sender what's included in sender and was not sender So I'm talking a little bit about that as far as you know, a federal storage and other terms that you'll hear Around sender talk about back up or in store of volumes and also some new functionality and that respect that's available now Talk also about consistency groups replicated volumes some new administrative operations that are available within sender and Some other new features as well, and then finally I'm gonna do a and if you pay close attention to some of my examples We'll reference that reference back to that when I get to the troubleshooting part so The there'll be an error that you can that we'll talk about later But actually it's it's fixed in the early slides because I figured I would just use whatever I hit when I'm doing my demo to Demonstrate here how to troubleshoot sender so keep an eye out for that So basically sender one-on-one blockstores basics, you know, so it's it's the service within open stack It's the equivalent of of Amazon's, you know elastic block service Basically, it is the ability to create extend delete block devices They get provided to virtual machines or instances in an open-stack cloud very basic It's appropriate for any, you know any scenario where you need blockstores whether it's a database Where that's a root disk for a VM and I'll talk a little bit about when do you use sender for a root disk? When when would you use what's called ephemeral storage for that instead? Also snapshot functionality is provided through sender, you know point-in-time Capture of the state of the of the disk at any one point in time through a snapshot And they can be used to do the restores as well and create new volumes And also, you know, there's interaction between sender and other services and to me That's where things really get interesting with sender sender You know creating a block device in and of itself is not so snazzy But when you start interacting with glance and Nova and all the other services That's where you start to get the real value of sender that's enabled So just an overview So if you look at the way, you know, there's different processes that I'll be running in a sender deployment For every and I'll get into a sort of a brief description of how that the blocks of the configuration relate to a process But basically for every back end you configure you'll have a sender process. It's listening on the backside But at the front end you have, you know users that are making restful API requests to the sender service And so they're not actually generating rest through, you know, like W get or something like that They're either doing it through the sender command line client or they'd be going through horizon or there'll be some automated tool That's actually accessing those API's and making restful calls through to the sender API service And the API the API services job basically is to stuff those requests that come in Into the message bus a MQP and then there, you know There's interaction with sequel at certain points to determine what services are available and different metadata They get stored in the sequel database but in general the requests are going in the message bus and if it's a backup job sender backup takes off But in most cases the sender scheduler is going to come by pull those requests off And it's going to do the filter scheduler is going to decide where does this this block device need to get created So we get to talking about trouble shooting It's good to keep in mind that even though your sender volume back ends are actually, you know providing the block storage service It's the scheduler where you're likely to see things go right or things that happen that are unexpected You might have expected one thing to happen, but something else happened So looking at the sender schedulers were typically where you're gonna go. We'll talk about that So this is where I talk about what sender is not sender is the block service Where is Manila, which is another project that's an incubation with an open stack is the file service So whereas blocks are provisioned through a hypervisor up through to the VM provided as a block device A block device just shows up in the virtual machine, you know, it might be dev VDB just appears in the virtual machine I can format that device. I can you know, put a file system on it mount it up Provision services through it and then snapshot it do all those kinds of things Manila the the point of that is to actually enable File sharing, you know, whether it's NFS or sys or you know a Gluster FS or GPFS various different File protocols within an open stack tenant namespace or network namespace So Manila is emerging, you know, it's not actually a formal project yet It's very close within liberty time frame that'll be available as well But just wanted to kind of draw that distinction. So whether you need block services or file services There's two different ways you can go with an open stack And then so another thing term you'll hear that is not really sender, but you'll hear the term a femoral disc What does a femoral disc mean? So if you think of a hypervisor, and it's really just a Linux machine bare metal machine running KVM Quimew or you know, it could be yes acts it'd be other hypervisors on the market Essentially, it's a disc that is designed to as a bubble would indicate is going to be around further You know for a while, but it when that VM is performed its function is shut down and terminated that disc goes away And it may just be sitting on local storage on the server It could be mounted in cases where you want to do live migration, you know, you have to have You know typically with KVM, you know like var lib nova instances will be mounted and shared And then you can migrate a VM from one hypervisor to the other So in that case, even though it's technically a femoral it's it's a shared disc You would use it in cases where For instance in our OpenStack team at NetApp. We use a femoral disc We've had, you know tens of thousands of the femoral VMs come up and go down in our continuous integration environment where we're testing our OpenStack and our OpenStack code and things like that. Well, you know VM will come up It'll run a job those jobs get logged and you know put into log stash or something and then that VM is performed At service it goes away. The disc is deleted. No need to keep it around. So that's where a femoral would come into play and I got a Basic sort of this is sort of the center 101 I wanted to play this because this is a kind of thing that I do anytime we bring on new people on the team It just sort of light bulbs go off when you see something happening You can actually see a volume name getting entered in The type of you know if I had multiple types of storage on the back end I could select that and then create a volume Very simple stuff. It's just to me. It's just endlessly fascinating I don't know why even even the basic stuff when you really break it down. It's actually pretty complex So this is a case where we're booting up a nova instance. I select the flavor Which is a basically the memory in the disc. I select an image. I'll launch it and so that's a femoral It's booting up that at instance is booting from a glance image It's coming up memories spawning the IP address is available and then here this this video is kind of rolling along pretty fast But what I'm doing now is attaching the DB volume that I created to the to the instance So that disc is now going to magically appear within that VM as probably dev VDB or dev VDC or something of that sort Then you can provision it you can create a file system on it use it within your VM another Mode that you can do is you can create it from a base you can create a sender Volume from a base image in glance So basically what's happening is creates a blank volume pulls the glance image that gets copied into the sender volume And then you can boot and you can even make that bootable so you can boot up a sender volume That is not a femoral it'd be persisting in whatever your block storage is on the back end and And use that without Without fear that it would disappear So now that was 101 a 102 we're starting to get into the command line what I used to call We used to call it the dark place all this is a white terminal You know you're actually doing things on the command line here And so whether you're whether you're doing sender commands here or you're hop operating through horizon You're creating those restful calls to the sender API service. That's putting them in the queue the schedulers taking them off Handling that request however was appropriate whether it's a snapshot create or delete Etc. So In this case, I'm just I'm you know showing that you can Extend the volume the from images the volume name. I grew the size from 25 to 30 It shows that it's extending and then it's available. So that that block device grew You know from 25 to 30 then you'd obviously have to extend your file systems or whatever operation you need to do to utilize it and That's about it So now that you've passed 101 your kung fu is strong You're ready for the real world now, all right, let's get into this a little bit more a little bit heavier stuff That's kind of a basic recap of what you can do with sender and you can see even the base functionality is very very powerful if you think about open stack as a way to Wrap an API around multiple hardware vendors whether it's nap or emc or solid fire You're running stuff or whatever you're running, you know LVMI scuzzy in the back end There's lots of choices. They all plug into this common API You can write code and that code will work anywhere no matter what the back end as a very powerful, you know basic concept So so the backup and restore Basically, you know, it's what it you what you would think it's a command line interface to create backups of a volume And so It's a little bit to me was not very intuitive at first, but the backups in the fault case go to Swift So they'll go to your object store that that block device gets copied into made into chunks stuffed into Swift Swift is whatever the replication is in Swift takes care of moving those blocks around to create the data protection that you need and You know so basically on the command line client You would do sender backup create and give it the volume ID, you know the GUID or the ID of the the volume or the name The reverse process would be sender backup restore from that and they need pull from Swift Restore that sender volume to the state that it was in prior to any data changes one thing that's new in in Just just recently there's an option to back up to NFS and so Swift So if you want to you made just a two-line change to your center conf You enable that NFS backup driver. You can back up to an NFS device. So That's a good option for people that might want to use that for replicating data around or you know There's a lot of different use cases around that so Swift is not the only target in here I'm just kind of going to illustrate a simple listing of a volume Creating a backup from it If I could type and then I want I kind of wanted to show you how things actually appear when you once you have a live backup and Then here at the Swift level where things actually get stored So I'm doing there a list I see that there's a volume backups container Then I do a list on the volume backups and look there's all the files that got broken up from that single one gig sender volume If I then Remove that backup delete it and do a list again Well, it's confirmed the backup is gone and then Swift Swift list on the volume backups. You'll see there's no objects there So basically it makes illustration of how how that back how that works Within the kilo time frame. There's there's also support for a new there was a blueprint that was partially implemented and Incremental backups are now an option So you can do a sender backup create dash INCR and give it the the full backup container that already exists and it'll do a an incremental backup from that point in time There were also options to there were some other advanced options in that that boot print that didn't didn't make it in but and this is and as I said If you want to do an NFS backup you can set your backup drivers sender backup that drivers NFS and Then your backup share to any any NFS share that you want to Back up your files to as well Restart your center processes and that would be your target for that point Okay another valuable thing as support for encrypted volumes That that was a tricky thing And you know the junior release brought the capability in crypto volume, but there was no way to back it up So not not very useful. So now in kilo there is support for You know backup and store of encrypted volumes As if you know anything about a crypto, you know, you have to have a key to do encryption and decryption so basically what happens is the the UUID Key gets copied through the key manager And allows the source volume to be deleted and it's a encryption key UUID made in balance. So basically you could create The backup it would be encrypted you could delete your source volume And yet the key will go with it that corresponds to the backup so you could then restore if you need to One thing important to keep in mind is that restores must be made To the same volume type as a source So if you had an old backup and you deleted the volume type you need to restore it You're out of luck. So just something to keep in mind if you're an operator In that case the backup rest restoration would fail All right consistency groups This is a relatively new feature There's very sparse support within the center if you go and look at the sender matrix There's not very many drivers that support consistency groups Basically, it's a set of sender volumes that are grouped together And you would use them as a kind of a logical set mainly for the purpose of creating snapshots So where you have an application that has dependencies on you let's say a database with log volumes and various different pieces of it Spread around you might want to create a consistency group for those those volumes so you can treat them as a set for for you know archival restores, whatever It can support more than one volume type So if you have you know a consistency group with you know part of it's in a volume type It's flash base part of it's in you know spinning media whatever you can still create a consistency group around that and I'll show you that you can see the reference here if you're interested in that feature Example usage here Center consist create you give it the name of the consistency group and the the volume type You always have special specify the volume type and so You basically create a volume and add it to the consistency group at the time you're doing it So that you know the system you have to have a consistency group ID to do that There's some examples here not gonna deep dive into that it's as I said it's it is kind of sparsely supported these things are Moving there's active discussions on a lot of this stuff in the center community And if there's anybody here and we get to the QA session there's people on core team that have Comments or thoughts here. Well, feel free to chime in replicated volumes this is a Basically this is a key storage feature for you know high availability disaster recovery type scenarios for applications in open stack There's an existing v1 implementation of this. It's sort of a first take Go at it basically said well, let's get something done I'll get it out there and and you know to try to try to make some progress in this So basically, you know the driver would establish a replicated relationship for you You know it's create the primary to secondary a relationship And then be able to promote the secondary in the case of you know a switch over And then make that primary and then re-enable replication in the reverse direction It's it's it's been a tricky process though because you know in sender There's a lot of different hardware vendors. They have a lot of different Technologies different ways to implement things on the back end. So finding a common, you know approach that really makes everybody happy is a challenge. So What send you know people have done different things with net app We've done things with our store service catalog where we have replicated volumes that pre-exist and you just simply filter on those by having a You know an extra spec that says I want a mirrored volume Others want to create the relationship anew as as something's you know happening in real time So there's different ways to solve for it So basically if you see that the the volume type has the capabilities replication true Cinder can perform, you know a variety of operations on those create update extend elites You know as I said it'd be basically setting up the relationship being promoting the secondary falling back And so the the different things, you know the actions will depend on what the operation is Also v2 is the you know the next version of this is an active discussion now and in the sender dev team So if you guys have thoughts needs about that specific use cases It'd be good to bring that up now in the developer community So I want to shift a little bit and talk a little bit about resource pool management sort of describe how that works In the junior release of OpenStack Sender introduced this concept of storage pools and prior to that every back-end was basically this one monolithic, you know group of storage that didn't really have any distinguishing Characteristics amongst it so but now there's a concept of storage pool So you can have a storage pool that has you know Various different volumes in it that have div and this is an example based on that a net up capabilities, but these could be They can be very vendor specific, you know, whatever the vendor can enable Through those extra specs can be you know treated as different pools within that back in So basically the way it works is the you know The driver comes up. It's it tells the scheduler. Hey, I have three storage pools You know they have capacities of x y and z And they are you know these attributes these various attributes are specific to these pools and then the sender scheduler says okay I've got that you know, I'll make provisioning decisions based on that information So the driver kind of informs the scheduler What it's capable of and then the scheduler's job is to do the filtering and make the decisions based on that So if you think about it, I mean, you know a storage rake and as I said, you know can be subdivided in lots of pools within you know within our context we might have You know volume that has mirroring enabled. We might have a volume that has QoS enabled we might have Compressed volume or a duplicated volume or you know a volume. It's based on flash media or spinning disks out You know sass, what have you and so those can be put in different, you know Different pools within that back end And we can do do things with extra specs to allow the sender scheduler to make intelligent decisions on that and I'll get into actually some pretty cool Capabilities that have just come out recently with that regard In this example, you know, we're representing different media on the on the back end of the storage device So volume migration This is when I say administrative operation This means you need to be obviously it be need to be an admin user So this is something that would be available to the cloud operator would not be available to a standard tenant in the public cloud or private cloud You'd have to have special permissions to do this Basically what it is is a transparent move of data as you would imagine from the current back end To another target back in that can be I'm going to show you an example in a minute It's actually a pool to pool migration. You can do it within actually entire drivers I could migrate from you know, let's say I want to migrate from an NFS store to an ice scuzzy store or what have you you can You actually migrate between drivers as well as pools The only problem is there is one got you cannot have existing snapshots so You can do things like doing a workaround where you you know to create a volume from a snapshot and then migrate that volume Is kind of a workaround, but that's one of the limitations right now also migration of volumes attached to Nova instances is only supported Where the hypervisor runs, you know, it's capable of a lot of migration And it gives you a little brief example of the command line to do that Here you're specifying basically the it's kind of the host name But you would get this information here from a sender services list command It'll show you all of your all of your drivers. I'll get the questions in just a bit Sorry, okay. We'll go into the example. I do do an example in a bit So what I'm doing here is I'm just picking a sender volume I'm going to show it and I'm going to grep for just the the host field So basically where does it reside I'm trying to find out where this volume is and I'm going to migrate it from its Current location to a different location in this case. I've got two back in Locations vol sender one or a vol sender and vol sender two I'm telling the driver to migrate from sender to sender two Basically, it's doing the block move on the back end transparently and then See how it's progressing Actually, I may have waited for it to actually complete here so yes, it You can see that it's now instead of where it began out on vol sender It's now sitting in vol sender two. So it's a nice feature for the administrator If you need to do migrations due to load or things of that sort All right, moving on to sender manage This is a Had a discussion with the guys in our team to me I think of it as an import export manage also makes sense to it's kind of just a semantics thing It's a way to take pre-existing Lones or files that you want to bring in to sender and bring them in through the manage command It has it basically takes two options. You can either do it from the source name Here or the source ID It's very vendor specific If you give it the source ID, it may mean you know, it means a pretty generic thing But it's up to your driver to figure that out You know the source name in this case is obviously a file path or one path to ball ball one one and giving it the the pool Also talk a little bit about and in order to have this kind of features you need to have what we call multi back in You know if you want to move things around within your sender configuration file you're going to have the enabled back ends parameter and so these these blocks refer to This config here where the standardized LVMI scuzzy driver or in this case the C dot Which is really just a reference to that identifier there and you have you know, your drivers all have their own particular configuration parameters depending on the vendor and What's required to make that particular driver work and then when you? You know you'll see that You can use these types go ahead And here I'm becoming the admin user again rather than doing dash dash os username, which is kind of tedious on the command line So what I'm doing here is to create a sender type Create a center type named C dot I'm also going to create a key value on the C dot type, and I'm going to set volume back in Name the C dot NFS and then just show you that as a the extra specs list here So what this shows you is that the you know the LVM driver was pre-created it already existed And then I just created a volume a type of C dot with the value of C dot NFS We'll use that in just a bit So another thing that's fairly new There's been some optimizations of the scheduler The schedule now supports over subscription for thin provisioning of sand If you think about it in a thin provisioning case the provisioned capacity Parameter to might not actually mean a whole lot to you So basically what you have now is a way to over provision Through the max over subscription ratio and that defaults to one which is a safe setting But if you want to bump that up and oversubscribe to 1.5 or 2 or whatever whatever you decide as the cloud administrator That that option is now there Another very cool feature is that you can actually do pretty advanced filtering With these filter functions now So if you know I'll get into an example in a bit here while I show what the driver actually You know reports to this to the sender scheduler You can basically take those different parameters that are reported to the scheduler and do advanced math or regular expressions to say If you know stats dot involves is less than 1000 and volume size is less than five then deploy it You know here So you can get very fine-grained with the way you do that So there's a goodness function and then this this filter function There's more if you want to dig into that feature. There's more here at the bottom here But this could be a very powerful way to kind of do some intelligent Provisioning within skint center. I mean the scheduler does a good job of figuring out You know basically what what back into I have that has the most space and putting the volume there But in some cases that that's not enough. So you've got a little more power here through this new feature Basically what I just said on basically, you know, the free capacity in a thin provisioning profile Scenario doesn't really mean a whole lot. So Okay, another big thing that was a big thing for the sender project this time around was the ability to do rolling upgrades That's been a an ongoing pain point and not just in the center project That has been a painful part of running open stack in general And so with this release a lot of the projects are on board now with being able to do rolling upgrades where You know basically masking the complexity of schema changes and databases and things of that sort It's all handled much more dynamically This particular change came from code in the nova project That was modified to allow services to be independent of schema upgrades from this point forward So operators, you know if you're one of these operators I'm sure you know about this and you're you're very happy to have it in your tool set Okay, another thing that's this kind of a new thing within sender is what it's referred to as private volumes There's basically, you know, and it is public flag. It's very similar to the nova construct of his public so you can set a Type to be false public is false So only only the administrator can see if that's what all you do you set it to false Then you as the administrator can see this type nobody else can see it So it's good for testing scenarios where you want to validate that your config is good and that provisioning is working. Well So but if you wanted to add a tenant to it To that type then only that tenant would have access to it So let's say, you know, you might might use that where you had a High-value analytics project, you know, you need to grant a certain tenant or a group of users Access to that particular storage you could do it through the product volumes Construct Okay, so if you if your kung fu is weak you need to do some troubleshooting the first thing if you're running a Distribution of open stack that's you know vendor supported most of them will all Disable the verbose and the the debug in the logging So the first step is to turn that stuff on you need verbose you need debug because if there's trace Trace errors or debug logs that you need to get access those will be completely hidden for you until you do this and restart your sender services So that's step one You also need to have understanding of which log is going to have your error as I said You know the scheduler is the one that's making the decisions if the scheduler is aware of all the different sender back ends That might be distributed across multiple hosts or different arrays It'll know where the capacity is and make decisions based on that so you can see that You know here you need to make sure that your services are up as a first step That's obvious troubleshooting and also you need to figure out where your your center volume resides. You can do that with that Admin show command Okay, let's say you have two volume types one C dot one LVM one I created an extra pet extra spec of C dot NFS. I try to do a create and looks all good, right? But then you get the dreaded no valid host This is my personal pee with a This is a very common message that you see in the in the schedule log You'll typically get and you'll get it in Nova you'll get it in sender no valid host And it's it is exactly what it says, but unfortunately doesn't tell you what the next step is So you got to dig a little further to figure out. What is no valid host mean? so in this case Let me flip forward here So I'm looking at What my back-end driver is reporting volume backing name C mode NFS Does it match? No, the problem was I had C dot NFS in my volume type of what I created earlier Okay, so if you have a mismatch between what the the driver is reporting this would come from your center configuration block here So you can't just create extra specs and and hope that they're going to work If they don't match what the driver is actually reporting You're going to have a mismatch and things are going to fall apart very quickly So the fix would be either to change your configuration to your volume back-end name would be C dot NFS or You could update your volume back-end name equal C mode of NFS here with your extra specs And that this the reason I want to bring this out. This is kind of applies broadly It's not just with volume back-end names As I said if you have mismatch between what the what the sender driver drivers reporting and what the scheduler Knows about through extra specs It's going to be a problem so That's that's all I actually have I think we have a little bit of time for questions if anybody wants to dig in We have a lot of documentation here if you guys are interested in that So if anybody has questions, please come to the microphone 19 Yes, this will be a this will be available online And they're recording this as well. I'll be on YouTube. I can't go to 19, please We want to review the So that the example that I the demo that I showed you the migrate was the correct syntax I'm not sure what you're wanting to look at Yeah, you see some I think in the very the last line you have to add a Pong sign plus the post name otherwise it come I won't work. Oh, you're telling me at the pool Yeah, yeah, I pulled this from the docs online and that you're correct. The doc need to be opted definitely otherwise You're caught a lot of confusion. That was that was from doc. Just that's one point Yeah, I think my example had the pool but yeah, I know so that's why I pointed pointing out and the other thing gentlemen I think it's this command only works for Migrating between I think within the same back end if we want to migrate for example LVM to To not have for example, we have to use the command retype So this is a bit. This is a great confusion caused by water migration. Hmm. Okay. Thank you Those extra specs that you said on Cinder volume Let's say you said something wrong, you know, because it's all text through you want to change it, right? So question number one, can you change it on the fly? Yes Let's see if I can get to that slide real quick Basically when I do the Type set command it's essentially an update it looks Thanks this slide here So this command here you can run this command a hundred times with a hundred different values here And it'll just update update update every time. Okay, if it's your so that parameter will just get updated That makes sense. Do you have to restart sender services? No, okay. No, that's dynamic. Yeah. Thank you Okay, I have a question about the replicated volumes. Yes So basically I would like to understand more how you would make it work with two different data centers Do you mean that? For example in one zone you have one Cinder back end in the other zone You have another Cinder back end and then you can do it automatically or you will have need one back end across two That the centers that yeah, that gets to be very vendor specific I mean net app has a way we've solved that but the replicated volumes is You know I would advise you go look at the Cinder matrix and look at the drivers that actually support replicated volumes We can filter on volumes that are replicated so you can set those relationships up and data automatically gets you know Transfer to the remote volume we can filter on that through an extra spec There's a different approaches of time trying to enable this kind of functionality So yeah, go out and look at the matrix for the center drivers and you see yeah, and Cinder do placement of volumes based on IO Based on IO. I believe is anybody from center core here I believe that is one of the features that gets reported up through this the driver and the scheduler Anybody here know for sure? IO Don't anybody from core here. I think so would have to take that offline then All right. Thanks everybody. There's a question about the private volumes. I'm sorry Question about the private volumes private volumes. Yeah, so is it possible to assign a volume type to multiple projects? Or it's a single project assignment. Yes, I believe you can assign multiple tenets to the private volume. Yes Thanks


When my clients ask me What should I do? What should I do? There's an inner turmoil inside me. I Cannot cope anymore. It's too much. I'm overwhelmed What should I do? I give them the following advice Go to your desert Listen To your inner silence These are religious metaphors, and I'm not a religious man. I Don't believe in God. I don't believe in angels, and I don't adhere to any religion But I don't discard the baby with the path water There is unfathomable wisdom to be found in religious scriptures and God Is a useful metaphor for many internal psychological processes So in this video I'm going to make an exception and I'm going to talk to you about God and religion and going to the desert and Listening to your inner silence. I want to start with a quote From first Kings chapter 19 verses 11 to 13 He said Go out and Stand on the mountain before the Lord For the Lord is about to pass by now. There was a great wind So strong that it was splitting mountains this wind Breaking rocks in pieces before the Lord but the Lord Was not in the wind And after the wind an earthquake But the Lord was not in the earthquake And after the earthquake the fire But the Lord was not in the fire and after the fire a Sound of sheer silence When Elijah heard it He wrapped his face in his mantle And went out and stood at the entrance of the cave Then there came a voice to him that said What are you doing here Elijah? What are you all doing here? At the point that you stand in your life Next to the cave that is your past What are you doing here? Are you waiting for the wind? Are you waiting for the earthquake? the fire Are you waiting for your for this phenomena to split mountains and break rocks in pieces? Are you waiting for a transformation from the outside? Some kind of force of nature Benevolent of course who will take care of you and save you and guide you and transform you This is not a Solution the Lord is not in this phenomena You remember where the Lord is? I'll read it to you again and after the fire a sound of sheer silence When Elijah heard it he wrapped his face in his mantle and he went out And he stood at the entrance of the cave Because he realized of course that God was in the silence Caves and deserts and mountain tops these have always been the haunts of holy men of saints and of prophets People who had claimed to be Indirect contact with God or the way he manifests on earth Theophany and epiphany This is where they went They didn't go to casinos They didn't go to libraries They didn't go to the city the big city They went to mountain tops They went to deserts They went to caves because there They could listen to the sound of sheer silence Isn't it amazing that sheer silence silence? has a sound Sound all its own God speaks through time through space through being But above all God speaks through silence God said When he was asked by Moses to describe himself. He said I Shall be What I shall be it is often mistranslated. I am what I am. That's all what God said. God said a year a share a year I shall be What I shall be In other words, God is a process of becoming not of Presence not of existence not of being only not of stasis not of stagnation Not of inertia. No God is flow Is motion is change is transformation is Becoming I shall be what I shall be and what shall God be ultimately The theophany of God the manifestation of God the sound of sheer silence so Here's my advice Just stop acting for a while stop it sit still For one minute Close your eyes for one second Don't even contemplate Just be Monks go to mountaintops and the big religious figures Moses Jesus Muhammad When they sought self transformation They did not act They did not act They went to the desert They escaped the city and its numerous distractions It's numerous temptations It's numerous offerings and the constant push to be active the curse of modern civilization act or not be They went to the desert Because in the desert even if you want to you cannot act There's nothing to act upon So you cannot do anything in the desert but wait Wait and be yourself and Wait for some bush to burn for you For some devil to confront you From some angel to talk to you The bush burn for Moses Jesus bested the devil and Gabriel angel Gabriel spoke to Muhammad in a cave These are all transformational processes Moses who was a playboy Egyptian Prince learned humility God told him this is holy ground Take your shoes off humility Jesus Confronted his dark side his complexes possibly his insanity and overcame it the devil and Muhammad Was able to converse with angels In other words with the best part of himself and the best part of humanity and the best part of the world And to bring back a message of redemption and Absolution and hope There in deserts and caves and mountain tops with no one around You are faced with yourself There's no escape There's no excuse.  What do you want? Who are you Elijah? So if you want to heal If you want to transform If you want to hope again if you want to sing if you want to belong Create a mental desert around you in today's world It's difficult to find a real desert to be in all the caves are occupied and on the mountain tops They are television antenna the world has abandoned us Because we had abandoned nature But we can still create a mental desert our own private desert our own cave and Climb to the mountain top That is us our soul if you wish Close your eyes No social media No news No people No zoom conferences Nothing close your eyes and wait For this silence to talk to you Wait for what many of you call the voice of God you're in a God maybe Wait for the voice Wait for the voice don't drown the voice in action See who is talking to you Listen what you are being told with what is being said to you think of nothing else Think not at all actually Just let it flow through you We are not listening Even religion and I'm not an adherent of any religion as you know Even religion is a dialogue. It's not a monologue. It's a dialogue with God But all of us we're just talking all the time. We're not listening We're just talking Never ever receiving We are too busy being something even being religious. We're too busy being us And these us being us is a simulacrum. It's a simulation. It's fake It's a subtle called it inauthentic. We are not real We are renditions. We are phantasms We are figments of sick imagination And so we are not we are not listening we need to be passive for once Just be passive It's in Islam.   You need to let this flow through you And you do not ask for reciprocal action from God or any other universal principle There's no reciprocal action It is a gift given on unto you tomorrow. I will discuss the rejection of life as A form of blasphemy and sacrilege But this silence is a gift given unto you and This silence speaks louder than any words and conveys more than any therapist can You want to heal?


So we're going to start out with the skull. Now, the skull is actually, this whole thing is the skull. It's actually made of 22 fused bones. And then there are seven other bones that are associated with the skull. We're going to spend an entire lab talking about just the skull and its various bones, what the specific bones are named. In today's lab, we're just going to deal with major structures and regions. One of the things that we are going to do is we're going to make sure that we're comfortable with the mandible. It's basically your lower jaw. We've been exposed to the mandible already with the digestive system. And then our regions. I'm going to do my regions in blue. We have a facial region, and again, lots of bones in there. Lots of bones that we will know the names of. We have a frontal region. The frontal region is like forehead area. We have the parietal region. The parietal region is on top. And you can see that there is a frontal bone and a parietal bone. That's good to know. We also have an occipital region. And guess what? That's the occipital bone. And then we have the temporal region. And this is the temporal bone. Let's see. Those are my regions, cranium. The cranium, anything that touches brain is cranium. So the skull includes the cranium and the mandible. Cranium touches brain.  There's another bone, the hyoid bone. And I have an image of it for you, because the hyoid bone is actually kind of difficult to visualize. It's this little red guy. It's like a U, and it's tucked up under your mandible. And we can see it. We can see it on Carlos, the skeleton. We can see it on Esther, the cadaver. It's tucked up underneath your mandible, and it actually attaches to muscles. So that's good. That's your skull parts. Now we're going to move on to the next one.


What's up everybody, Jason here for jasperyskatesprac.com. 2022, Panini Flawless Baseball just sold out. This is a two box case break, Picatinny's number 11. And again guys, just sold out straight up. Didn't need a no randomizer, no filler, no nothing. Just straight up. And Chris got the Marlins last bomb ojo. Everybody else bought in straight up as well. Appreciate the fill, guys. So like I mentioned before I started this video, we might have another one available. I'm not gonna post it until I get a confirmation if I don't get any confirmation tonight, maybe for tomorrow, but we are selling this on the personal side, so if you're interested in personal, it is available over there, but until I figure out, I can't really take one, and I don't want like what happened last time too, over so and we don't have a case so. TBD on that. We have this one here though, so good luck. All right, here we go. Showcase Relics, three out of 20 Nolan Gorman. That is for the St. Louis Cardinals Mark. Tony Gwinn Relic to 15. Padre's going to David Chan. Then we have a Dual Patch Autos. Look at that. Five out of five. Laundry Tags, little Nike Swooster, Jackson Core, and Daniel Lynch, both for Kansas City. Very nice, who has Kansas City? Brandon Hall, there you go, man. It's a pretty sweet one right there. I think this actually might be too big for Carl and Office. Jake Berger to 20. Chicago White Sox going to Kev. And again, another Laundry Tag there. Josiah Gray, three out of 10 for the Washington Nationals. Matt Smith with that one. You got a tail scar Hernandez to 15. That's 14 of 15. Toronto going to Carl. Jumbo, Star, Swatches, Patch Autos, 10 out of 15. Riley, Ryan Zimmerman starting out Riley. And that's for the Washington Nationals, Matt Smith. The last one here is a little Tom Glavin. One out of two. New York Mets going ahead. All right, let's see what's in the encase cards. First one. Walter Johnson for Washington. Three out of 20. And let me double check which Washington it is.  Yeah, Washington Senators, but that's still the Washington Nationals. No, hold on. Yeah, I think so, but I'm seeing something else on here. I mean, Wikipedia is saying it's national. Well, you know what? Why am I doing this for? Duh. Let's go to a group break checklist. Remember we had that little, that whole card drama? Whatever a group break checklist says basically, because they usually do it the right way too. Uh, Walter Johnson. You're right, yes. They do have it as a Minnesota Twins. Okay. That was weird though, because Wikipedia was giving me other answers. And let me show you guys. I forgot I could just use group break. There you go. Minnesota Twins. So let me mark that really quick. Minnesota Raymond. So there you go, Minnie. Let's see what we got here. Nice O'Neill Cruz. Look at that RPA. 10 out of 15. Three color patch. Who has the Pittsburgh Pirates mark? There you go, buddy. That's a nice hit right there. I sure covered it, right? I hit two of them on Instagram, back to back, Steve, on personal. I hit that a five and that a seven. It was wild. But have we hit one of those over here yet? All right, we got seven out of 25, Tim Anderson. That's Chicago White Sox. White Sox is Kev. How about this one of one? Look at that patch. Mark McGuire. Very nice right there. Oakland A's Robert Runkel. That's pretty sweet. We have a Russiny Contreras autograph there. Six of 10 for Pittsburgh. Gonna mark Aaron Ashby for Milwaukee. Nice RPA there to 25. Josh Lowe to 20. Six out of 20. Sam Bebe Ray is gonna Carl. Bobby and Ray won a 15. Little inscription there, 30-30. Philly's gonna Gregory. Then we have a little Diamond Gem, or not Diamond Gem, that's a Ruby and autograph. Will Clark, USA Baseball out of 20. That of course should be Giants, right? Yes. San Francisco Giants, that's going to David H. And last winner is Charlie Blackman. Nice patch autograph there to 20. 13 out of 20. Rocky's is David Chan. All right, here we go, guys. What is this? Green. Francisco Lendor, one out of five USA Baseball. Now should we go to New York Mets? Ed. All right, here we go. Don't know what this is. It has to be an autograph, right? It is. Arizona edition of Randy Johnson. Four out of five. And look at that signature, dude. Randy Johnson, four out of five. Zona, going to Ryan. Congratulations, buddy. Nice one right there. So big RP in the first half, second half had the vets. And there you go, guys. Another case of flows down. You can look out for potentially some more this weekend. Jaspyscasebreaks.com.


Why are you like this? Let's get you prepared for your new life in my room. He just wants to fly around the room. He's like, this lady, can you just get me Brando? Right now, in fact, I get a call from George. He noticed that Blue's beak was split and possibly cracked. I heard the fear in his voice. I feel like what must have happened is this. I bet Merlin went for Brando and Blue went in the middle to protect Brando and just was able to grab onto Blue's beak. I knew that I wanted him to have a surgery. Right now, Blue is at the vet. I'm going to be picking him up today. Hello, my fellow snippers, flighters and hatchlings. My name is Marlene McCohen and welcome to our channel. This is Meg Hanne. Hello. We are outside of all animals veterinary hospital where my bird has just gone through surgery and we are about to pick him up. So let's make a call and get our little baby. I need to see my baby. It's been a minute. They're very concerned about Blue's last name. I'm like, do you want the name on my card or do you specifically need Blue's last name? Because we haven't chosen one yet. So if you have one, then I don't know about it. Are we enrolling him in school? Then I tell her my last name and she repeats, we need to know Blue's last name. Yeah, she did. I'm like, wait, what? I don't know what kind of paperwork George and I have together. Is it George's last name, my last name? She's like the bird's last name, please. Apparently it's on file. Well, never mind.  Will Blue McCohen, please call? If that isn't his last name. We don't even know. I have no idea what his last name is. It's like the bird, the blue bird named Blue. They just had surgery. But we need his last name. Do you have it on file? Cause I'd like to know what you guys named him. I guess we'll see in the receipt. It looks like, oh, okay, we're back here. This is good news. Half mark there. Okay, so this is, what is this? Is this a soap art? What is medicine? Yeah, it's a bird. What can I give him to eat? Well, he can go back to his normal food, really. Maybe soft veggies, stuff to store. Oh my God. Yeah. What is that? Is that like a cast? It's like an acrylic, basically. Just to hold it in place? Yeah.  And will some of the beet grow back where there's a hole?  Oh. Baby. He's got a prosthetic. He's got an acrylic nail for a beet. Oh.  My Ganny has like a prosthetic. Oh, I never saw him. Did you see him before? He went, do you see him now? I couldn't do the camera because it got really fuzzy. Look at him. My poor baby. Oh, he's gonna sleep in mommy's room. Why, my baby?  Like, if this is what it looks like with the acrylic, I don't know what it looked like without it. Oh, can you imagine he survived the whole surgery? Is he like standing? Like, yeah. He is definitely like, you know, in good spirits. He's like obviously getting ready. Thinks he's gonna see Brandon. Yeah, if you can come look at him and show me how to do the medicine. I mean, because looking at him has given me such a fright. Okay, so I'm gonna finish up this project. I'm talking to Dad about how to finish it. And then I'll come, I'll head over there. Okay. Are you crying? Why are you crying? Because it looks scary. Can you send me a picture? Let me a picture. Okay, it's hard to get a picture of. Oh yeah. Oh my God, he looks like Freddy Krueger. Oh, that's a new added thing. We have Freddy Krueger. Oh, he's his job. This is where you're laughing. Oh, he's so cute though. That's not funny. It's actually not about you to put your job. Like, what's Brandon gonna think? This is like a cast. So there's a wire that's underneath that colored cast, right? And he's hoping it'll fuse together. And then as the thing grows like a nail, it'll like grow well.  Okay, I love you. Love you too. You went to the hospital, that's all.   But he knows, like he's looking like, he must watch the outside of our house on YouTube. He's like, are we here? I'm like, okay. Life in my room. So this side's completely normal, but it still needs the prosthetic on both sides. He just wants to fly around the room. He's like, this lady, can you just get me, Brando? What do you think you should relax? Like I've been relaxing for two days. You see it? You're not acting like you've got a problem. Hey, I need to process this. Give me a minute. Look, it's itchy. And the itchy can imply that it's healing. True.  He's got no problems. You wanna make use? Well guys, I'm so glad that Blue is home and that Blue has energy and that he's in good spirits. I would prefer if he relaxed a little bit more, but I guess it's been two days since his surgery and they've been hand feeding him and keeping him contained. So I think he just wants to get out and spread his wings. I'm so happy that he's okay, that he's alive. Like that there was no trouble with the surgery that he survived the bite that this was fixable, you know? But yeah, it's kind of hard to take in, it's alive. I can't wait for him to see Brando. I mean, it looks like he's over there getting ready for her as we speak. Generally, I would say if your bird's coming home from the vet, you know, try to take it easy with your bird, but this bird, he's got a mind of his own right now. So I'm letting him take the lead a little bit and then I'll get him to bed a little bit early. I'll let him see Brando, but I won't let him, you know, hang out with the other birds. But I do enjoy that he's flying around and hanging out right now. I was a little bit scared about like, okay, you're a beak, but I mean, it has been in two days of bonding right now at this point. So I mean, today would be the same as tomorrow. It'd be the same as next month, I think. I'm exhausted from the experience. I did the shock, you know, because I didn't see this happen. I didn't see anything before he went to the vet. Probably better be seeing it like this than how it was. And if you would like, comment some well wishes from Blue because definitely he would appreciate them and he definitely, you know, went out on a limb for Brando. So I think he's a very special bird. Any of the comments that you guys write to him, I will read to him and I do believe he will enjoy it. He looks happy to be back, although a bit itchy. So, so that's the update here. I'll see you guys later. Bye. Also, I don't know if you noticed, but I put some vegetables for you, which I can just peas, your favorite. I need a little bit of nice and cold. Get another dog. I walked Harry today, saw this other crock-a-boo. Love to me more than Harry, loves me. Don't love me, he doesn't love me. He's like, he hears me wake up in the morning and he goes, ah, great, she's the way she can walk me. And then he stretches and then he comes and he's like, riff, riff. And I'm like, ugh. He kisses to get out of this. Come here, you have to go in your cage. You have to, come here. Oh, you good boy. I love you. Oh, you're such a good birdie. Okay, you ready for a seat?


And one day we hope to recognize a little pinging inside. Ladies and gentlemen, I present you Mr. Ron Minich. Now it's the microphone working. I wanted to thank Faust then for having me here and obviously that it talks about Linux BIOS. Now sometimes people ask you what you're going to talk about and then you send them a paragraph and the paragraph is in your program and I realize this talk bears some small resemblance to that paragraph but it's not actually the same thing. So I'm just going to talk about what it is, why we're doing it, give you some idea of how it works and then talk about what the Google supported work is all about and then where we're going to go with it. So what it is is really exactly what it says it is that the plan starting eight years ago was that Linux would be your BIOS. And we need it. So we're going to take Linux completely out of the flash, put Linux in the flash, throw the PC BIOS away. How many people here like the PC BIOS anyway? I hope no one's going to raise their hand. We all hate its guts. So the goal here is to get rid of the PC BIOS completely. It's a 1981 piece of technology who needs it. And in theory it should be easy as Jim mentioned this morning. These computers are actually fast.  So getting Linux running in a few seconds is really something that is pretty simple to do. Who uses it? Here's a couple of supercomputers that use it. There are about 8,000 various types of computers that ladle the run Linux BIOS. This is the Kray XD1. Kray has been using it for some time. This is of course the OLPC, the iRobot Packbot. That's kind of a neat application. That thing actually is kind of a mind detector.  So that's kind of a nice application. This is kind of neat. This is an FPGA card that plugs into a hyper-transport socket. So there are several companies now that take a dual optoron motherboard, take one optoron out and put an FPGA in. Or they have a card like that. That card was designed at Mannheim. Well what happened when Extreme Data tried to do that? They got a Sun Ultra 40 dual optoron machine. They took one optoron out and the machine would not boot anymore. So the super different proprietary BIOS was too dumb to boot the machine with one processor removed. So I did a port last summer of Linux BIOS to that machine and it comes up just fine with Linux BIOS. But the other really neat thing is it turns out, according to A&D, that the code in Linux BIOS that turns on hyper-transport is better than any proprietary BIOS that exists. If you get the open source BIOS, you're not going to get the proprietary BIOS. That's kind of neat, right? So we're really at the point where we're doing a better job than the closed source junk. So how many people use it? So this is kind of funny. We bought a 1000 node cluster here and we thought we'd kind of hit the big time with Linux BIOS. But compared to where we're going, and I mean it literally is a growth of 10. So that year we had 1, then 10, well I'm sorry, 10 and 100 and 1000 and it's a 10x growth every year. And you don't really realize what that means until you get about the year. Then you realize a 10x growth a year kind of adds up after a while. But the funny thing is the point here where we thought we hit the big time isn't even visible in this graph. Why would you bother using Linux as a BIOS? So the motivation for me is this column right here, Drivers. So you can use something like EFI, which is proprietary as we all know. That's Intel's BIOS that you're trying to get everyone to use. Yes it's extensible, you can do it in C, but the way you extend it is you get the 1,500 page EFI spec. Attempt to understand it, which I can't yet. It's a totally new API, a new program environment, you need a special GCC. You've got a DOS-like file system, you've got to live in that weird environment with a DOS-like command prompt and do a full custom driver. Or you've got Open Firmware, which is a lot better, it was proprietary, it is no longer proprietary. It is extensible and you've got to figure out how to write these drivers in force. Some people can do that well, I can. There's the PC BIOS proprietary, but if you're good at assembly you've got it made. So all you good assembly language programmers out there are set. I don't really like writing an assembly. The big thing here is, Linux, if it's my BIOS, it's not proprietary, it is extensible and we all know how to do that stuff, right? It's a driver. And most of the time somebody else writes the driver for you, so you don't even have to think about it. So do you like writing non-Linux drivers when working Linux drivers exist? I think all of us would just as soon use someone else's driver. Or this is the fun one. You know, you have a piece of hardware that works fine in Linux and fails in the BIOS. That gets really old really fast. Or if you write a driver for one of the first three, guess who gets to maintain it forever? And this is a nice thing, you know, we've been working on getting some of our code into Linux kernel. The neat thing is you're on a mailing list and you start seeing patches come in from all over the world to fix bugs in your stuff that you submitted. So it's a lot simpler and a lot cheaper to have the same drivers in the BIOS and in the OS. And that's part of the argument for having Linux as your BIOS. I should mention the DRM BIOS issue. This is a funny phrase that was coined by a guy I know at Google. We were talking about Linux BIOS and he said, well you know Intel EFI is just a DRM BIOS. And we talked a little more about it and here's the thing. The newer BIOSes have unlimited control of your memory and IO cycles. What's that mean? What that means is that in the chip sets, the chip and all your machines out there nowadays, the BIOS can set a bit and set an address range and any IO you do in a certain address range will get trapped to a system memory management node handler. At that point the BIOS can decide to veto your IO. But what's really fun is they also implement a full network stack. So how paranoid do you want to get? It is actually in the range of possibility that the BIOS could look at the kinds of things you're writing to disk from the operating system using the IDE IO ports. And the BIOS might get upset about the kind of data you're writing to the disk and send network packets somewhere. It's all possible. It's all possible today. How do you know what it's doing? There's nothing you can't by design. So that's a little worrisome, right? I've noticed some people with some banks that are very upset about this because what they've realized is they may be running Linux, but they don't know what's happening underneath. We had a long telecom with Intel about this and they were explaining the really cool aspects of VFI, the fact that things could get intercepted and EFI could take action for us. In fact, EFI could go out, find a server, download a new version of EFI and burn it to flash all without us knowing or intervening. And panic ensued at the end of that conference call because we had just been told that the BIOS was kind of able to go off and do things without any direction from us. So what we've required for the last six years when we buy a supercomputer, it either run Linux BIOS or more recently, they have to give us full and buildable BIOS source because it turned out IBM didn't want to let us run Linux BIOS. So what IBM did on the most recent machine we bought, they shipped us buildable BIOS source for that machine, which I'm never going to look at because I don't want to get contaminated, but this is the way we work and we did it for security reasons. BIOSes that can take action on their own are really a bad deal. More people should be worried about them than are worried about them. Linux isn't a BIOS only though. It's a bootloader too, right? Since Linux can mount and use all the Linux file systems, it can do everything you need done. It can do the Kexac, it can mount things, it's got network protocol stacks, it's got all the drivers you need to get the things. So Linux is actually a great bootstrap loader. What if you need a new file system? Well, it's always there in Linux. Do you really want to write a whole new set of file systems in the EFI environment that are enforced? And again, I don't. Somebody already wrote them. I don't need to do that again. So if you remember all those custom entries for drivers, we've got the same entries for file systems, network file systems, network protocols, wireless, you know, the list goes on and on. It's a long list. So the way we use Linux is Linux as BIOS and Linux as bootloader. We've measured this. It's the most reliable and flexible and fastest way to boot your machine. This is interesting. Until about three or four years ago, Linux was always a really fast way to boot a machine. And if we put Linux in Flash and Etherboot in Flash, Etherboot was always a little bit faster than Linux. At some point when the CPUs hit about two gigahertz, we discovered that Linux was always faster than Etherboot. And it's not that intuitive. Etherboot is this sort of direct, small kernel, polling I.O., you sort of intuitively think, okay, Etherboot has less overhead than Linux, it'll be faster. But what in fact happens is Linux has so much internal concurrency and overlapping of operations, it's actually the fastest bootstrap we've used. And the other fun part is, of course, these non-Linux systems tend to fail in unforeseeable and unpleasant ways, okay? And I have a friend who has 4,000 node clustered.  So they went up to one node and found out that the node said no keyboard present, hit F1 to continue. It turns out you can fix the problem in parallel. The parallelism is limited by either the number of people you have or the number of keyboards you have, okay? So if you can imagine 20 people fanning out, walking up to machines, bam, F1, bam, F1, bam, F1. It only takes a couple hours. So I'm not allowed to name the lab or the company because they're so embarrassed about it, even though it's been almost a year since it happened. What do you want to see if things go wrong? A lot of biases will do something really friendly like this. This is my favorite, actually. Can't boot disk, okay. And my reaction is, what is okay about this? I've never known why it's telling me things are okay because they're really not okay. And in fact, this is even worse than hit F1 to continue because if I know about F1, I hit F1, what am I going to do here? I've got to interact with this thing. In a supercomputer cluster, this is about the worst possible thing you can ever have happened. There are a few things worse than the okay prompt in a cluster. What you really want the node to do is always go over the network and talk to you over the network about what's going on, right? A lot of companies, and actually, I'll just say the generic big search engine company don't even hook consoles up to their clusters. I don't either, right? It's been years since I hooked up the consoles on a cluster. That's what the network is for. But what do you want to see if things fail? Well, if I have to have something that's going to give me this awful prompt, it really should give it to me over the network. But I want a shell prompt. I want a Linux shell prompt. I don't want this. I really don't want that. I had it for years on Spark Stations. That was enough. What if you want to extend the BIOS? Do you need to extend the BIOS temporarily for debugging purposes? Really easy on Linux, right? Mount a disk or a network file system, change your path, insert a module, whatever you want to do. Knock yourself out. It's Linux, okay? So you can trivially extend your BIOS if Linux is your BIOS with new commands or whatever you want to do. That's just really not easy with a standard BIOS. Sometimes you don't have any other options. So anybody who's got the new Intel chipset will find there's a new feature on that chipset. It's on that laptop which Stefan loaned me. It's called the Regulatory Demon. Has anyone seen this thing? You cannot run that wireless card without the Regulatory Demon. You can only get it as a binary. Yeah. You can now. Thank God. Okay. So there is a way out of this hell. That's good to hear. Nevertheless, if you need a WPA supplicant or if you're running something really complicated like Infiniband, that gets a little messy, right? In something like EFI or open firmware. And at some point, should the BIOS or the bootloader have modules and processes? Some people say, yeah, but at some point you're just building a bad OS. Why have a bad OS as your BIOS when you can have Linux? There's no reason to recreate bad OSes at this point. We've got a good OS. We know it can fit in flash. Problem solved. So for the future and more complex BIOS I feel Linux wins again. The newer drivers are getting really complicated. It turns out we need one less binary component than I thought. But still, you know, what's happening is people try and stretch the BIOS to fit the hardware and the other things that are coming down the pipe. They're turning into bad OSes. And EFI is just a great example.  I actually don't know. So what we say is go with a simple BIOS idea. Linux is BIOS, Linux is bootloader. Okay, how does this work? And this is where the real fun begins. This is kind of a summary of some of what we've learned over eight years of trying to make this thing go. First step is you've got to enable and bug fix the CPUs. Now, if CPUs is floral, I'll get onto this, but you've got to enable and bug fix the CPUs and very quickly stop all but one. For hardware, you've got to discover, configure and enable it. Pre-PCI, you really couldn't do this stuff, right? You had things like ISA, you had funky, you know, little ROMs and stuff that had configuration information, but once we got PCI and we got self-configuring, self-defining hardware, life got less than possible, right? I won't claim that PCI is easy even now, but it's pretty straightforward. As an aside, I'd like to say, I think OS is our original thought and hope was that the OS, especially when 2.4 came along, the OS could configure all the hardware with no help from firmware. That has not actually proven to be true. I don't know if we're ever going to escape the need to have the firmware do a lot of pre-work for Linux before Linux and the other OS can figure out what has to happen. What's the boot software have to do? They've got to pick a device. That you can do in a Linux RC. You can do further configuration. Again, this is boot software running out of firmware, out of Flash ROM. If config, SF disk or whatever, can launch DHCP or mount an equivalent and get some more information on the disk, maybe do a mount, validate it, that's a file command, boot it, that's kexx. This is kind of the steps of a bootloader. Everything is there in Linux you need to do to do that. We actually demonstrated that on OLPC. Before OLPC went to open firmware, the way it actually worked is it came up. We had busybox and Flash, that was in the NIT RD, in some versions we had the kernel and Flash. It came up, it had a Linux RC, it had a timeout, it would look around and see what was there, optionally boot something over a network or from a local Flash. I actually, the system I delivered to Extreme Data, which again was a Sunnel for 40, that also did the same trick. It came up with an Ashtrump and you could run a script to tell it what to boot. If you know how to make Linux do things, you know how to make Linux as a BIOS do things. The initial key idea was we thought Linux could do everything save to the lowest level hardware. This is the first version of Linux BIOS. It's pretty simple, it's four lines of code. It works like this, I hope I'm not going to move too fast. Mem copy jump. That's Linux BIOS version 0. It didn't work at all. It didn't have a prayer working as it turned out. The first problem was what I called a DRAM problem. Here's a DRAM module So here's a DRAM module and I don't know how many people know this but of course the DRAM module has a DRAM but the thing that's on the DRAM module that's not always apparent is there's this serial EE problem. Why is there a serial EE problem on a DRAM module? Here's the pants they're used to. C-Pute at North Bridge to SDRAM bus to DRAM. That's pretty straightforward. Here's the pants that not a lot of people are used to. C-Pute at North Bridge to South Bridge to Super IO in some cases or over here to the SN bus to the serial EE problem. Why are we doing that? The reason you do that is that DRAM module is a synchronous clocked machine. So there's actually a state machine here that communicates with the state machine here for memory cycles while there's timing in these state machines and the serial EE problem defines the timing in the cases which are common that it's correct but it defines the timing for this module. And so what you've got to do is configure up that bus all the way down turn to Super IO on or South Bridge SN bus hardware on talk to this, read the values in compute from those values and your knowledge of the North Bridge timing how to set up the North Bridge. Well what's interesting is you've got to do that with no stack. So you have no C function calls until you have DRAM and you have no DRAM until you do all that work. So you've got to do all this stuff DO over the I2C you've got maybe 17 fairly complicated parameters all of them interact. It's like having 17 springs that you can stretch. And then you've got to match them to the North Bridge so some of the strings stretch them too far because then the timing won't match. Then you have to program the DRAM and program the North Bridge. So that code is the single hardest piece of Linux BIOS and to give you an idea how messy it can get we had a project to do ported this to the G5 and we never got it because there were some bugs in the G5 chipset that we weren't told about. So this code though is just about the hardest thing there is to do on any machine and it's actually getting harder as time goes by, if anybody knows about DRAM training and the new DRAMs you actually kind of sit there and you actually train it and it essentially tunes the delay lines. There's a delay line for every bit that goes out to DRAM and that all have to be tuned slightly differently. So then there's a little bit more we have to do so it ends up looking a lot like this and some of these are empty on some things you did the display you set the PCI method there's methods one and two so you really want serial output as soon as you can possibly get it because it just makes your life a lot easier when you're debugging then you enumerate PCI turn the frame buffer on but you can't really do this until you have some idea what the structure of PCI is then you enable PCI initialize PCI initialize the CPUs and then this is essentially that mem copy more or less that I showed you earlier so there's a fair amount of work to do here and in fact some of this PCI code was ripped right out of 240 that's where we got it modified it quite a bit but that's where that initially came from SNP support is part of the let linux do it philosophy we actually had a guy in who in 2000 wrote SNP startup code that ran in linux so the idea was your motherboard had come up and linux kind of thought it was on a single processor board but it would do a little looking around and figure out that it in fact was on a multi processor board and start up all the other attached processors with the standard IPI mechanism and all that what was kind of neat about this is that nobody had SNP startup code in C it was all hidden in the bios as an assembly so we finally were able to create this code and have a reference essentially for how to bring up an SNP which was really nice to have then we got to the K7 and here's my theory on Panium SNPs there's hardware voting that goes on so they all come up and there's some kind of hardware voting mechanism and they all stop except one so there's probably a patent on that speaking of patents and so on the K7 it turned out there was no hardware voting they all just came up and started running and actually had to put the voting in software in the bios because we didn't want them all running the bios at the same time and that meant that we had to take that code that we had started to put in the Linux kernel and make it descend down into very early code in Linux I mean in Linux bios in fact assembly code so there's sort of for K7 is the sort of initial early vote step for who gets to run the bios in that what that really means it's an SNP bios at the time I kind of thought well an SNP bios seems like a fairly useless piece of software but then the optorons came along every optoron has memory attached to every optoron CPU you can now do memory initialization in parallel on optorons so the memory in it in an optoron is SNP on Linux bios so that was all going really well we had gotten to the point things were okay we had a half a megabyte flash 2.2 just fit right in there no problem but this became a big problem and Jim alluded to this this morning why is OLPC running open firmware when our initial efforts were all Linux's bootloader Linux got too big and it continues to get too big and you know it's been a problem down the problem motherboards went to a new form factor they went from the nice big dip parts it seemed to have about a half meg easily available to these little small parts that at the time were just quarter meg parts they were limited by pins so at the same time this guy was getting a little big the flash parts were shrinking so we kind of had to change things a little so this is what we this was the vision in 99 right this little Linux loader and Linux kernel this is what it meant in 2002 well a simple payload loader is now called Linux Bios we're doing things like loading plan 9 kernels and plan 9 loaders and Filo which is grub without Bios calls Mentest, Linux kernels and etherboot right so this changed a lot you know I'm from New Mexico we've got this t-shirt the t-shirt says New Mexico it isn't new and it isn't Mexico so I told Stefan about that and he said huh well this isn't Linux and it isn't a Bios because we don't support callbacks but you know we've got the name Linux Bios and it's kind of stuck and it does incorporate chunks of Linux kernel codes so I guess we can stick with the name but in 2006 we're actually getting back to where we began so I mentioned that as little square flash parts that only hold a quarter meg because they're pin limited the newer parts are called LPC and what they do is they multiplex the address in chunks and they get a big memory address it just sends it a bit at a time and you know the 2 megabyte part is not a big deal 16 megabyte part is not a big deal I saw this headline in electronics newspaper recently Nan Flash is free so the parts are getting big again and we know we can easily fit a kernel in a busybox a lot of busybox in 2 megabytes that's not a big deal and what we found when I talk to vendors and say I can give you a couple of choices we can put a grub like thing in there we can put an open firmware thing in there we can put a Linux kernel in there with a shell prompt response is always the same we want Linux people want Linux in the BIOS it's just the easiest thing for them to deal with then the next question to ask is is it working so that's where the Google work comes in what Google has done to support us with some money which is always nice and so Steph and Ryan are our core systems decided one of the things we can do is build this quality assurance system why do we need it what happens with commercial BIOS is when they go to a new motherboard they essentially fork the BIOS so you've got this code base this is what I've been told and the commercial BIOS company will take their BIOS and send it off to someone at that point it's a fork we don't do that in Linux BIOS we maintain a common code repository but what that means is that as people do things and change things at any given time we may have a motherboard that's not working and we'd like to know that as soon as we can but we can't afford to keep every single motherboard that's supported in house so what we need it is a way to have sort of distributed systems that report to a central system that we can test out new versions of Linux BIOS on so I'm not going to really talk about this much this is kind of a rough view of the test cycle but a lot of times what kicks this thing off is a commit from someone and the first thing kicked off by the commit is a complete build of every single Linux BIOS target so that's part of what goes on if any build fails then email gets sent out to the Linux BIOS list saying this thing just broke the test setup so there's an automatic build system run by this test server the test server in turn can control motherboards up to 16 and basically the test server can reflash the flash on that motherboard and then run it and look at the serial output and test it and see if the thing came back up now the neat thing here is every once in a while you'll flash a BIOS and the potential is to turn the motherboard into a brick at that point you flash a BIOS and the motherboard's gone but the hardware that Stefan set up allows us to recover from that because there's actually two flash parts the flash part you can fall back to if you really mess things up and recover so we start with the Linux BIOS tree we do a full A build for the A builds to succeed we control those motherboards and then do the testing and see if it works on a given board so this is kind of neat this is called a BIOS savior to solve that problem this plugs in, there's a little tiny flash part here, this is the part you re-program and potentially program with a bad BIOS and over here is the one you recover from and this little USB module essentially electronically controls the switch that flips between the two so that's how you can take a chance where you might brick a board and still recover from totally trashing the flash part being able to get the board back so here would be the central repository at openbios.org these would be different vendors so here's a vendor, this guy runs two systems under test these are different guys running multiple systems under test and so on so the idea is that we can have MSI and tie-in and other vendors set up these things centralize the reports and make them accessible to everyone I don't think anything like this has existed before for BIOS because the vendors tend not to care about an older board or someone else's board fails but we have a lot of cases now where this vendor will find a problem in a chip and that will actually fix this vendor's problem and we're kind of hoping the vendors are going to pick up on the fact that they're really somewhat in the same boat here that they can actually benefit from each other's knowledge of say the NVIDIA CK804 the ins and outs of that chip and if you fix it for one it's all it is sort of happening MSI has a full-time Linux BIOS person now and so does AMD so people are kind of picking up that this is exactly the Linux dynamic here applied to the BIOS so these BIOS's image are built by the AutoBuild tool and then we've got these daily snapshots of the images this is kind of neat, I like this I'm terrible at this but stuff is very good at it there's a test name here it's very hard to read but this says banner, copy to RAM jumping to Linux BIOS Linux BIOS and RAM and so on and this is pass, pass, pass, pass failed, failed so you can see your SVN rev number the target board the failed and passed you get a really nice test summary and the idea here is say MSI might only have two or three of these pages they're monitoring and they will get an email kicked off to them when a commit happens and anything goes wrong so Stefan built this thing he's using deja vu this is kind of nice you can really test it all the way up to Linux Timer interrupts and do your IDE drives work and that kind of thing so it's a really good assurance that you've probably got things right it's kind of interesting but if you've messed up the interrupt tables the first thing you see it sort of says I always forget the message calculating Bogo MIPS or something and then it never comes back that means Timer interrupts is just gone so I don't know if this applies to anyone here but core systems is looking for people to join in with this thing so the future work what I've talked about here is version two what happened was everything was version one and the Opturon came in and the Opturon was so complex and so far beyond what we were used to because every Opturon essentially had three processor buses which we weren't prepared for a number of things came along and kicked off version two so version two ended up being very complex to deal with systems like the Opturon what we're doing with version three now that we kind of know how it's got to look is restructuring the code and it's for casual users but it doesn't really mean guys who just run AVI Word it's kind of casual BIOS users what we're really talking about is an engineer who might spend one day every two weeks working with the software doesn't have time to really learn it in depth but wants to use it so that's what version three is oriented to we've moved to Kconfig that we hope will make a more familiar environment for people we've greatly simplified the code pass because after a couple of years you may have a lot of flexibility in how you construct an assembly code file but if every component of that file is not changing more than about every three years there's no reason not to just put it all in line in one file we had complaints about the amount of usage of include people don't generally like include load scripts we had made use of load scripts but I think we were doing a little more with load scripts than people are used to so we kept tickling bugs in the Canoe Benutils it just seemed like we did that about once a year or twice a year you'd start building images and they wouldn't boot at all and you trace it all the way back and find out well Canoe LD GCC is generating new segment names or Canoe LD is somehow not doing the right thing so we're really minimizing our use of features in Canoe LD we actually had a C compiler in the tree and that was done to minimize the assembly code this was written by Eric Biederman ROMCC was quite a trick because what it actually did is it could compile an ANSI standard C program in a way that it used no memory so it's 25,000 lines in one file optimizing C compiler there are a couple bugs but it was really an incredible piece of work because it basically could tell it oh you're in a penny and whatever use MMX registers or you're on this use streaming CND registers or whatever and it would basically use the registers and nothing else and that's what you've got to do in that early step where RAM is not up so you could actually write C code and compile it and then you'd have this assembly that used no memory but the problem was that that's a big maintenance effort and we've just learned how to use a newer or better thing called caches RAM and basically on the paniums and power PCs just about everything nowadays you can pretty much play this little trick and all of a sudden the cache memory is kind of acting like RAM for some period of time so we're moving to all caches RAM and that means the only C compiler will be GCC and we'll still have very minimal amount of assembly code. There's about 100 lines of assembly code in a given Linux BIOS and politics so you can't escape politics when the opposition to what you're doing is non-technical you know I don't know we started this project and we thought that the benefits were so obvious that after a year we'd get companies in working with us but what actually tends to happen is you get a lot of opposition because you're pretty much selling the BIOS vendors they can all go out of business now and you're kind of telling Intel we'd really like you to continue to release documents about chipsets which they don't release anymore by the way so what's happened in the last 8 years when anyone knows about developer.intel.com I mean the 440LX chipset you pull the documents there's a tutorial in that document about how to program SD RAM okay 3 years later the newer chipsets there is nothing in those books about how to program SD RAM and in fact there are notes about there are certain registers we're just not going to tell you about unless you get an NDA so you know once we started using the information that was posted the information in the case of some vendors just vanished so we kind of have to deal with this political stuff the other thing is we're really trying to show vendors they can make money even if they let us do Linux BIOS the cool thing that just happened about 3 weeks ago is NVIDIA let AMD release all the programming information needed to do the Linux BIOS port for the NVIDIA chipsets that covered 4 vendors so there was an MSI, a Gigabyte a tie-in and a super micro motherboard using this particular chipset so about the same week probably the same hour for all I know that NVIDIA okayed the release Yengai Liu at AMD released Linux BIOS for all these boards it was a wonderful day the discussions with NVIDIA began from my end in 2002 and what they told us is tell us how many million chips you will increase our sales by in a year and we will release that information so if we can show vendors there's a business case and that's what happened with OLPC then they'll be on board so you can't really demonize them but they can be very frustrating people to deal with sometimes and of course continue to evangelize so a lot of people will maybe remember the IBM PC it came out and the BIOS listing was part of the package you got there was this sheet of paper that was the BIOS and what people didn't realize it was closed source that was copyrighted people had to do clean room re-engineering of the BIOS so what we've shown is the highest quality BIOS can be built using free software and it really is it is a good BIOS it was interesting to have AMD tell us in an open meeting that the hyper-transport software and Linux BIOS was better than their hyper-transport software and in addition better than anybody's hyper-transport software that was just a shock but we've seen other cases it's so hard to set up DRAM that a lot of times what the BIOS vendors do is they don't do a very good job so you get fast DRAM they run it just slow because we've found in a lot of cases we do a better job of programming DRAM than the proprietary BIOS we've had the millions mark about a year ago it's in a lot of systems you and we have never heard of I get a phone call once in a while and somebody says ah one of the ones I got was really funny there's a telegram that made a turkey and they make 15,000 of these things I forget if it's a month or whatever and it's digital and they found out that when they ran it with a PC and people turned it on they didn't like waiting 90 seconds for the image to come up so they put Linux BIOS in it and then the image came up in a couple of seconds and then people were no longer angry with them so it's just weird things come out from nowhere about usage of this stuff one of the great remaining things is to get it into your laptop and we don't know how to do that right now there was a DARPA project that was founded a couple of years ago DARPA gave IBM money to put Linux BIOS in a laptop we were working with them and the part of IBM that did laptops did not want to tell the part of IBM that got this money how to do it they told them to go pound sand so IBM isn't always very friendly to IBM of course the laptop part of IBM got sold so maybe there's some justice there but it wouldn't have been really cool so if anybody has clever ideas about who we can talk to or how we can get this in a laptop we're willing to listen and talk so let me just show you one more thing if I can this was pretty exciting first off we don't know the guy who posted this so this is just an announcement first desktop motherboard supported by Linux BIOS Gigabyte blah blah blah but what was really cool is AMD essentially said we're supporting this this is where we we've been trying to get for eight years now get to the point where someone at the level of AMD and Gigabyte would say we're supporting this now where we're not okay notice this is like a part number right but that part number means I don't know Phoenix or something what we wanted to have is M57SLI-S4-LB we wanted to be the case that you go to New Egg or something and you order this part and it comes and Linux BIOS is on it that's the next hurdle but actually getting AMD I mean AMD has just been great for two years now they've just been absolutely incredible but having AMD sort of say oh yeah we hired this guy he does nothing but Linux BIOS full time and by the way we're releasing support for four motherboards that was actually a really big deal and that was just February 22nd of this year so I guess that's it do we have time for questions or did I run over? Will it boot my operating system of choice CG OpenSolarOS sorry will it boot up other OSes like OpenSolarOS once it's done all the hardware initialization? What was the first part? Will it boot any OS I like? Oh okay so there are problems with different OSes so years ago I tried to do FreeBSD because I used to do FreeBSD a lot FreeBSD still wants to make BIOS calls if your OS makes BIOS calls you're out of luck your OS shouldn't make BIOS calls it's a bad idea but some of them do I'm sorry somebody told me they've done it okay the OpenSolarOS guys are here we could walk on down and say hey do you guys make BIOS calls or not if they say no and the other thing we can do I've got it running an emulation on QMU we could just do it just test it and see what happens so that would be kind of interesting what's this? Windows Windows we actually had Windows booting under Box BIOS years ago but it was pretty painful what we'd like to do is try to react to OS guy again and have him talk to us about freeloader so is that what you were asking? I'll just second everything but will this in theory do you need to do some hardware do you need to do some hardware changes to motherboards to be able to run Linux BIOS or any other software Flash ROM you just reflash the flash and you can do that in place you have a program called Flash ROM that lets you do it so from the web page you can see from long ago I used to add one address line for making the flash parts bigger but I haven't done that for years so yeah I'm using some clusters I like to the idea of the Linux BIOS project so getting started I can get for making my system compatible with your list today you have a list of hardware that is working with Linux BIOS I can help you to having some more system available meaning I can help you to make my system working with Linux BIOS what should I need to start you should send mail to me sorry you want to know how to make it work with your system I didn't quite understand I'm sorry I have to send you an email for starting so my main question is I have some bugs I want to be supported to Linux BIOS I have some basic knowledge of other hardware works so I can get started in making my system compatible with Linux BIOS what should I need maybe an EEPROM programmer actually the main thing you need is knowledge of the chipset so I haven't used an EEPROM programmer since about 2001 we do all our programming on the motherboards and a chipset guy pioneered this and told me it's safe we actually unplugged the flash part while the board is live plug and do one in and program it the main thing you need to do is get on a mailing list and send us an LSPCI that's always the first point LSPCI and then we can talk about whether it's possible so it would be great to have you involved install Linux BIOS okay so the question is how do you install Linux BIOS there's a couple ways you can go at this it depends on how daring you are I get on the mailing list to start and ask if anyone else has had experience with the motherboard you're using and sometimes I just send people an image Jordan Krasen AMD has built a utility build ROM and build ROM will allow you to basically lay out your motherboard and you say make it pulls down the kernel it needs it pulls down the Linux BIOS it builds the right version of Linux BIOS for that motherboard and when you're done you've got an image that has a kernel and an NRT and a Linux BIOS image which you can just flash but the mailing list is usually the best place to start to make sure you know what you're going to do not everybody wants Linux and flash some people just want the grub like loader so it would be good to talk to the mailing list and see what's possible alright thanks


It's amazing what real-time leads can do and how you can increase your percentage when you're setting appointments of actually getting appointments Way higher. Joe set like 80 90% of appointments. Oh, yeah when he gets them in real-time Yeah, we've had completely different results from when we didn't do real-time to now doing real-time We've got agents telling us that they're getting ahold of people almost every call now Legitly, which is crazy because but when you have to wait Days or with direct mail weeks a lot of times won't remember they won't pick up the phone when they do it immediately 7 p.m. At night and you call them you have a much higher chance I would say double the chance of reaching them in setting appointments So when you're setting appointments and you want to be setting 80 to 90 percent of the leads that you talk to or at least Talking to 80 to 90 percent of the leads you purchase super rare Real-time allows you do that when you're setting appointments