Okay, hi all, welcome, it's about the hour, so time scheduled, so we have to start. My name is Jeffrey, I am with the Open EDF project today. And I'm hoping to explain a little bit about what we are trying to do, and mostly also get your opinion on things. So, the genesis of the Open EDF project was, the fundamental question, and that was, what if storage for containers, storage system itself, was actually as a container? So, basically what we do is we try to distinguish the complete storage system and store it in containers so that it can be a resource-class citizen. And they just said, no matter how flexibility around this to do what we do with this particular workload, they can say, oh, we're just doing that, right? Just an NVE device doesn't do any faster than that, and we're done. So, this basically, this is how we come up with the idea in a term, container attached to storage, because we thought that we could augment the actual gas pipeline in our storage systems. And another fundamental thing that we do, or we do, we still do, is that typically storage developers, because I'm trying to comment on the storage background, storage developers typically reason for this to happen tonight. So, this is my guess. And I have to break that somehow so I can get the application. We basically converted that. How does the end-user, you see there, you get my persona, how does he work, and how can he make his life easier by giving him tools, fresh and intense, out of storage? Because that, originally, in storage consolidation, was done for efficiency.  And back then, storage devices were very small, so they were in a megabyte range. So, they were just sitting there, not going to utilize in terms of passive performance. So, they consolidated all that access storage into one storage system. And then, obviously, also, I made data management in particular for take-back-ups, which is take-back in the day. So, it originated in the local area network, and Morrell was a pioneer in this area. And he actually had to have a special TSR, Terminate and Stay Residence tool, or Daemon. It wasn't called Daemon, because it was Doste. But you logged into the system, and then the volume became available to you. And this is also the time where TCPIP was not established yet. It was IPX, SPX was still very much used back then. And still, even in Windows 95 and Windows NT, they were adding additional drivers, not shipped by Windows, by the way, but that would need to be installed in order to attach to these storage systems. So, fast forward 20 years, then we had a dedicated network, a real storage area network, or a network-attached storage system. These were very specialized hardware appliances. So, they had very specific hardware that was not otherwise available to the average computer shop. So, also, it was a separate network. So, they segregated the network. It was a different network fabric. It was fiber-channel. There were no IP packets moving back then. It was all fibers. It was very specialized and also very special switches, with special interconnects between data centers so that you could do federal over-across data centers as well. So, how does this look in a picture? The picture says more than 1,000 words, I suppose. So, this is just a random storage picture that I googled. And if you look at the whole picture, what it starts out with at the bottom is the shared storage. And the shared storage is not in the sense of, hey, I'm having this system and sharing this out through an event or whatever, but this was shared on a fiber-channel level. So, the actual disks were visible on both controllers. So, when you do LSPOK or something, both machines would see that disk, right? So, the shared storage applies to the actual disks. And then they were connected to the fiber-channel switches all the way up to the controller. This is basically a front-end, back-end system. So, we have the front-end observing out the protocols that we all love and hate, supposedly. Ice fuzzy in the vests, SMB, or what have you. And then, at the lower end, we actually have the storage-related protocols. And these front-end systems that we ran, for example, to accelerate white caches, that alludes a little bit to the special hardware, right? And you did not have, for yourself, PCI caches. Not even v-devices looked a lot like it, but it was not the same as an NV device today. And the CPUs were relatively small. I think it was around 2007. They were still using pens from the CPUs in this type of system, because it wasn't really CPU-tested at all. So, another fundamental property of the system was HFAO, right? Storage can never go down, because if storage goes down, we can all go on. So, HFAO would typically need to be done in 180 seconds. And this was a heart limit, so to speak, that was configured on the actual clients running on them. So, you have to install special drivers, right, to set up the settings in the Windows registry or in the SIGMS or whatever operating system you were using. So, they were fine-tuned for every operating system that you were using. Unified, so block-and-file servers. Later, also, VTL, virtual tape library, and back of today's. And when all your data was on that particular storage system, then you can do some interesting stuff with the data in terms of storage features, like snapshot clones, dedu, and replication. The significant downside of this is that these systems were huge, huge, expensive, right? There were millions of dollars, euros, and euro, after all. And not only that, they were really imposing the way that you operate storage to the upper layer. So, the specific features of the storage system could or could not do bubbled all the way up to the end user. And that was, you know, I still think, a significant downside. So, it had a profound impact on what you could do with the software. So, software-defined storage. Software is eating the world. Software-defined storage happened. The hardware, you know, commodity hardware, as it evolved, became more powerful, quicker, and what have you. And systems like Luster, and SheepDoc, and all these open-source, software-defined systems came along. All built on commodity hardware. So, Fibre Channel luckily died. They tried, the industry tried Fibre Channel over the internet. And luckily, it was born dead because, you know, it was a very, very expensive way to keep people on the Fibre Channel coolant so that they could charge you a lot of money to do so. So, luckily, for software-defined systems. Devices became faster, obviously. So, all the state devices and NDRAM was not needed anymore because now we had SSD devices that did, like a Zeus RAM. It was like a SSD device, connected to the SAS bus, but it was actually around it. So, you could really have right-optimized workloads at certain aspects. But, the architecture, however, did not change. The way that we had the separation between computer storage is still there. And this is where fiction happens when we look further down the road. And that is when the containers re-happened. And I emphasize that we, because containers are not new at all, originated, I think, in the BSD space with jails and later on, so I have zones in our Linux with containers and one fundamental thing is that the way we develop, build, and deploy and run applications has changed a lot. Like I said, we invert it. So, we look from the app up all the way down to disk. So, what can we do there? And what are the observations that we're seeing? So, modern day apps have native standard features. They have load balancers, they have database sharding. They use sophisticated consensus algorithms like AppSource and Graph. So, you don't really need to have a whole lot of storage complexity in order to achieve scalability. The way that we redeploy them, Kubernetes is based on a Google Workpaper. If you haven't read it, it's a very interesting paper to read and explain where Kubernetes comes from. But if you look at how Kubernetes operates with workloads that can come and go at any given time, persistent or not, you kind of wonder, okay, so how do I make that work with my storage system, let alone with different storage systems at different regions and different locations and different vestiges. So, it's a big pain point. So, with Kubernetes, we have basically tar balls on storage, right? It's a computer, it's a defined compute storage networking. Everything is confined in a tar ball, but it's actually a bit of a shocker, because it's something to that extent. People also got smarter, right? Developers, they took they accounted for failure. You should always write your software such that it can handle failures. But people also start to write software that are DC aware and even regional. This was very much fueled by Amazon, AWS services with these regions and stuff like that. But people now develop their software and if I can't get it, I'll try to get it somewhere else and don't mind how they get there, but we'll see about that later. So, the mindset of developing applications, high availability has been built into applications. Not as much as, here's my app, I don't know how it works, you just want it on your storage system and you better hope that your storage system never fades. So, the friction between teams is still there, probably even more so, because the workload is coming though, right? So, now typically you have one VM with one mount point that's okay, but now if you have a hundred thousand containers spinning up and two weeks later they're all gone and then two weeks later they come up again, you have all these mount points and those systems have a limit on the amount of mount points they conserve, right?  So, typically what happens is, you know, the legacy storage system in particular they try to duct tape this stuff with the plugins as they call it, right? So, instead of doing an API call to something that is sensible, they have this interceptor they intercept the API call for vision along the legacy storage system or whatever. But people generally don't really like that, so the friction is still there. And the other thing is that although you are able to mount something from your storage system they did not unlock the storage feature that you might want to use, for example this dataset is used to take that file. So, don't bother to compress it so in my Ammo I say no to that because the developer knows that. So, how does this work a little bit? So, this is a small animation I didn't make it myself, but I think it was a pretty good one. So, the idea is that you have your operator who expresses his intent in the Ammo.  Yes, does it not make sense, but it's actually in the file and maybe not, but it's human readable. This gets executed by the environment and below you have your content. It's magical, right? But, these systems are not stable. So, it's easy to understand how do we do these mistakes. So, if we take into account the stateful aspect we have our storage systems and then we try to somehow, some way, volume plug in our way to make this work again with storage, but the problem is there's a very tight plug-in. You lose the flexibility of moving systems around, spinning them around fashion one and moving, taking them back out and all these cool attributes that we are using. So, the other thing is that in this style blend you have all these workloads and you don't really know how your workload behaves on the storage system because, you know, it gets mashed and becomes unclear water. And with all the observability and tracing tools that are available these days you really don't like that. So, an alternative approach is kind of the same idea, but now, instead of doing it with an eye of blender we actually spin up the gaming container with your workload, right? It's very simple. It's a little bit more complex to implement, but the genesis of the idea, like I mentioned, is that, you know, each container workload of home ties or micro-services or control. So, practical things. So, how do you do that? And so, you have this queue utility that you can run and you add to your OpenEBS operator. This is a database that we want to spin up, for example, so we apply that to the Kubernetes system. And then, the Kubernetes does its work and all the people, we have those persistent volume claims, as they call it. And that's it. So, what we really want to achieve is that storage itself just fades away as a concern. It becomes, you know, DevOps way, agile. Data ability is really important for various people. So, that's a little bit about the control plane, right?  There's a lot more to it in the way that we integrate the Kubernetes and all that stuff and talk about its own. But I also want to talk now about what OpenEBS is about and to change these conclusions because we did our research looking at the system of applications and how they develop. OpenEBS is not a distributed file system. And there are various reasons of why it's not a distributed file system. One of the reasons that distributed systems are generally are relatively hard to manage in production. I have worked on two proprietary distributed systems and, you know, you have to hand hold them a little bit. And depending on how big it is, you probably need a small team that is at least to keep them operational on a day-to-day basis. And to really unleash the full potential of distributed files system, you need special drivers, right? You cannot just, you know, have your petabyte scale multi-gigabyte throughput system and just connect with SIFs and just expect that data to come out of the pipeline. You need to have special clients to unleash the basis of such data.  The sum is pretty big, but individually there are relatively small, right? So, talking about a Mongo database with, I don't know, 300 gigs, there's relatively small. If you keep it that small, it becomes very simple to manage. It becomes easy, of course, to replicate it. Replicating half a petabyte and if you can control this on a workload system, it really becomes feasible. So, another aspect is that hardware demands a change in the way that we do. And, you know, we finally can change the actual architecture because we still have this same model of segregation from storage to computing. And the single end of the e-device can do up to 450, or half a million octaves. It's just one device. So do I really need a distributed system to gain higher power? Likewise, do I need to scale the petabytes like I mentioned, but I'm not built because the individual workload is relatively small. So, and one final aspect and this is really, really fundamental is that when you're doing containerized applications you are already working on distributed systems, right? And all these microservices working together to achieve one goal, and I feel given the complexity of distributed systems adding another distributed system to that to make it a match of two different distributed systems I don't think that that is the right approach or not saying that it's not by definition the right approach, but in general, you might want to try something else. So, the comparison and, you know, kudos to the graphics on the right side is actually mine. You probably can see that, but it kind of looks the same, but for a specific application, right? So that's the fundamental idea. So I'll go over the components a little bit that are in there. So first of all, there's the controller, like our front-end. This is where you connect to, right? And we are starting out with block. We can do file-based if we want to but it seems to block is a lot of simplistic a little bit a lot easier to employ because you don't have to set masks for the amount of forms and things like that. It's more one-to-one mapping to the container. ICERD, that's basically ICSI over a v-band, obviously, NMD over Fabric. They're actually working on NMD over Fabric for TCP. I'm not sure what to deal with that is, but if that actually gets released, then that would be really, really great. So this controller is pluggable, right? So we can just swap out those services because there are containers out of the wall. So that allows us to have a pluggable back-end so that we can change the way or change the logic that actually persists the data on this, right? So if you have a database workload, right, you might not want to have a copy on white file system or anything because your database inherently already works with white-hand lobbying and things like that. So it's kind of doubling up. So depending on your workload, you may want to define your gamma a little bit different. Right? So on declarative state, a number of replicas, snapshots, schedules, the retention of your snapshots if you want to replicate it, and the consistency levels are also important. So with that, I mean, if you have a T-band replica, you can say I want to have a quorum and I acknowledge right. I have a quorum and I acknowledge back to the client. So to speed up right-end, obviously. And obviously this also needs to be DC aware. Region aware is a little bit more difficult because regions typically have a big distance between them. But DC aware is certainly a possibility. So now we get a little bit more into the meat of storage systems themselves. And like I said, the replicas are pluggable and we have several. And today I would like to talk about one that we are working on or at least trying to figure out if it would be a good idea. That is reusing the DMU layer of CMS. I don't know if people are familiar with CMS. I think you should. If you're not, you definitely should need about it. But it's a well proven battle-tested file system. But it has a very interesting design. It's like models like a database. So we basically took that in a component of CMS and run that in user space. And one of the properties that it has is that each right is a sign of transaction. And these transactions are matched into transaction groups. And these transaction groups are flushed to stable storage. This is done atomic. So either all right succeed or none of them. So this also means that there is a file system check. More often than not, when my laptop dies, I put up Linux and it doesn't I know checks or whatever it says. But that's not necessary with this system. This is important again because if you replicate data you want to make sure that you replicate a consistent data. That's one.  That's a new file system. And say, oh I can't repair it. Well thank you file system. That doesn't help. So pool storage model. This is a virtual memory management type like allocation scheme. So you assign devices to that pool and the system allocates logs from that pool. And this was considered a rapid layer violation in 2006 by the Linux community. Because unlike any other system right now it combines several pieces together. So Linux, you need to partition drives or LLVM things and then put a file system on that. That's all taken care of by the system right now. So it has a very easy way to consolidate storage devices and basically allocate storage devices. Another thing is end-to-end data integrity. Very, very important in a file. It does this by checking all rights. It sounds very simple, but actually it's not because it's also a public right file system. But we all know that because CFS has been there for I don't know 10, 15 years. But now it runs in a container and in use. And the upside of this is if you move your system from cloud to cloud because my view of CFS is not native in the Linux kernel. It has a different license. It's CDL versus TPL. That won't make so you can't have it. You can, but they have to install it yourself. Now when you move from clouds and you want to scale up and scale down like that, you cannot afford yourself to first compile a kernel module across your fingers and load it and then actually try to import the data. So kernel dependencies or this dynamic kernel module system or whatever it is. So the other thing is that it does not take the kernel, right? So if you load a non-GPL thing and your Linux kernel will say, hey, kernel take it. Maybe that's not a big issue, but if you have a support contract with I don't know, let's say a Rathad or whatever, that actually might become a problem. So with the capabilities we have asked what it allows us to do is do things like this, right? So the logo says it all, right? That's our mule. It's a combination of a horse and a donkey. I didn't know what that was either, but they can't make babies when they do this machine. So you have your workload in US Central and it's running CentOS, right? CentOS and I'm not going to pronounce it. Basically you move your data or your compute.  And normally this was really hard because you would either be tied to a compute volume or a Amazon volume. Now we have this additional abstraction running in the container that makes this possible, right? So again, your data mobility becomes a real feature today. So but there is a problem, right? A big problem, actually perhaps it's the elephant in the moment. So anybody guess who said people who think that user space files systems are unrealistic for anything, but toys are just misguided? Anyone have an idea who said that? Yes, yes. He kind of knows a thing or two about computers, right? But like I said, this is 2011, so I purposely put the gate there fast forward. It's 2018, so seven years down the road in the hardware, like I said forces of change in the way we do things. Good for us. Microsoft could be good for you. NTFS. Yeah, okay. Yeah, could be. I do not have the finances to acquire a license. So how do we achieve high performance numbers in user space because we have a lot of context which is every time when I do a mutex lock in user space, the kernel comes in, right? Context switch. We lose performance. The other thing is how do we do direct memory access trends for user space because we don't want to copy the data and we want to immediately tell the CPU hey, there's the data, just grab it and move it to the device and we're happy. The other aspect is besides that, the other thing is that the kernel actually becomes a bottle, right? So we are moving towards 100 big devices that work and if this is interrupt driven, the CPU can't really make or the kernel can't really make forward progress. It gets interrupted every time and there is, although Linux is very concurrent, there are still co-pads that have to be sequential, right? So the kernel really can't keep up. Same thing goes for NDE devices, right? And we're moving to a 3-day cross-point and the problem only gets worse. The other observation is frequency remains relatively stable, right? The core count goes up. If you buy a Mac and a laptop thing, they have 19 cores, right? It's just amazing, right? So I see a lot of idle cores somewhere. So, you know, this is not just me claiming this. This is a graph from Intel. On the lower side, you see the number of SSDs and then you see the number of I-outs, right? So, like I mentioned, the single device around 450, half a million I-outs and then when you add more devices, the total number of I-outs does not increase, right? So, that's not getting the biggest bang for your buck, to be serious. On the right side, this is a very contrived example, so I have to admit but this is a simple web server because you can write web servers with, I don't know, three, four lines of code these days. One is using the kernel network stack and the other is using a user-level networking so it completely bypasses the kernel to process the package. You can't do anything else on the network card either, so the network card is lost, right? But it does show you that if you do things differently, you can get more performance higher. So, how do we do that? So, as I mentioned, we bypass the kernel and it's a paradox but in order to do that, we actually need to kernel, right? So, we instruct the kernel, famous the kernel and those devices go, give me those addresses and I'll work with them. So, there's a trust solution there. And then we use a framework called SPDK and with those systems, we can create containers that do IO call work, right? So, I don't know if this is a clear picture but we map the register to user space and then SPDK does its IO directly into these addresses. So, Polmo drivers, what that means is that we take a CPU instead of using interrupts, we constantly are pulling forward. So, if you look in your system, you will have a core that is 100 percent C.  But, you can imagine that if I have a Kubernetes system that I would run, I don't know, 100 containers on that particular machine and all of them would try to do 100 percent CPU, that's a problem, right? So, what we did is we containerized all the IO devices and we called this an IOC, the IO container, and instead of submitting the IO to the kernel, we submit the IO to the container, right? So, it's a user space DevFS. So, but then we have this problem, okay, so how do I tell my MongoDB instance to do IO to your IO company, right? The rule is not going to change because we had this great idea even though I say so myself. So, picture, how does this look? And so, this is the IO scene, so we put their mix, the SSDs and CPU, and through the CPU mechanism, we can actually control a little bit what CPU it takes, so there are some tunables there left still, but the idea is that I allocate these resources for storage on my node, right? And, like I said, we have on the course these days in those high power shows, and then we have this plus, the short speed, that we communicate on. It actually really looks like a micro-show, right? So, I did why. So, how do we communicate, right? So, the idea here is that we reuse proven technology from the virtualization world called V-host and Verti-Oscusi. This is done entirely in the user space. So, the IO container basically serves out through this V-host protocol the locked devices. Just like you would connect, you move to your, you connect it and you say use this device, you kind of just basically specify, use this socket, because what we will do is we will use a shared memory and basically you connect to that shared memory system and that's how we exchange data. And, because this is a V-host interface, and it is consisting, we do not have to change any of the software that's already there because completely abstract. So, the replica containers connect using this Verti-Oscusi system. They expose sockets, like I mentioned, but there is no lit Verti-Oscusi available. The protocols are embedded into the hypervisor. So, there is no standalone library that you can check out and just run. So, we are currently developing that. Looks good. We moved some bits already so that's really good. Another important aspect is memory management. So, like I said, we outweigh the memories that we stick to in the IO container. But we use huge pages for this. And it's an undocumented feature of a Linux kernel. But these huge pages are in memory so that means that the kernel doesn't move on the map. And, because they're kernel-so, we can actually DMA straight into the NV device from the Verti-Oscusi IO and that's what how we can put on. There's also some future work going on. I actually checked the fast-time schedule from last year and FDIO or FIDO, as they said. They basically take this user space networking to the next level with something called VPP. And they can achieve on this bus, this will just be a tier of it per second. So, bandwidth side of things because when we have these high million 500,000 IO SSD devices, we also need a lot of bandwidth. And that's why FIDO can deliver as a universal additive. It's from Cisco and in particular what we are interested in is VPP-VCL. So, VPP-VCL stands for Vector Factor Processing Virtual Communication Layer or something. So, it becomes a little bit complex but the idea is and this pretty cool is that they emulate the native PSD sockets, right, like sockets and we see that kind of stuff. And you can help you preload it with your existing software and automatically getting the benefits of the Vector Factor Processing. It won't go as fast if you would rewrite it from scratch. But, you know, you know when you rewrite everything from scratch obviously. So, to make it a little bit more clearer on the host side, the host, user thing, just to make sure. So, we have the host. We have some huge shared pages which are containerized as what the second blue line is supposed to indicate. And then we build a virtual queue and then we expose that virtual queue through an IPC mechanism just to give memory on regular video specifications. Nothing magical to that sense. But it allows us to communicate as fast as you can. Final slide and summary Open EBS. We will have to bring the advanced storage features to individual containers and with advanced storage features I mean things like copy and write, dating integrity, because I truly believe that you need to make sure that the dating is actually the dating you need. Cloud native, using the same software development paradigm instead of cloud washing and existing system duct tape it, we actually rebuilt the whole thing from scratch. Implemented fully in user space so that we avoid congestion in the kernel like I mentioned and because of that it's also multi cloud. So move your data in and out of different cloud providers, not just cloud providers but also on your own storage system or whatever you have in the office. Declarative provisioning and protection policies that remove the friction between the teams. So there is no more, hey I need you one and things like that. And the other thing is that we are not a cluster of storage instance but instead we are a cluster of storage instance and that means a fundamental difference. Okay, so with that I'm done, thank you and if there are any questions please feel free. The data in the containers when we reschedule the container for whatever reason is crashed and so I actually mentioned it but we want to deploy the replicas or the only deployed storage system we actually make sure that the application lives in the same pod as the storage pod and when we when the application dies we have both the interviewer and the scheduler and we have this affinity to reschedule it there. If for whatever reason you can't do that we reschedule the application to another right and if you can't do that for whatever reason I'm not sure if you implemented that already but then we have to copy the data to another machine that is taking off. But that controller running in the cluster takes care of it. Yes, yes, yes Right, so the question is if we can do multi read write many so the sheer nature of iSCSI and block based protocols is that they are not in share. In that picture that I have with the big box with the shared storage both controls can see them but only one controller can actually access at the same time and they do this with SCSI reservations systems SCSI reservations shoot the other node in the head like paradigms to avoid that but in essence you could if you want to work with SCSI reservations now most of the people ehh not really so ideally what we're looking for that would be a file back end that you can share a little bit better so we don't have right now a file based a few points so to speak so we only have a block based controller but we could relatively easy with I don't know does that answer your question? Yeah. But you could for example just run in your container and that's what we're just doing like that how does it develop? On top of CNS servers and I have some level virtual machines using the GMP and Linux containers using fuels will I be able to design this paradigm using open ES? Well so it depends well of course everything doable so one of the things with SPDK is that it has different IO back ends so I write the code once but I can select during the configuration you are writing to a regular rotational device in that case it would be the A and the back end so then there is the kernel is still involved then there is the way to do NMV and there is even the set plug and we could even augment it so that would be one way the other way is that you would I don't know just point your V-host to an open ES V-host. Yes I would like to get direct of the Nostral and maybe even of the ZFAS and do it in user space. Yes. Well that's the trend right? Yes. So user is a new kernel so thank you my time is up


Well, Jim, your father and LBJ were old, old friends going back to the early 50s, I guess. Even before, Dad got to know President Johnson when he was a senatorial candidate in 1948. Dad was Air Force Secretary. So he went charging down to Texas and campaigned like the dickens for him. I think he helped a little because that was a rather close race, as you know, and every vote counted, maybe twice some of them. But anyway, they became fast friends at that time. And 52, four years later, after Dad had three or four other jobs under Truman, Chairman of the National Security Resources Board and RFC Chairman, he ran for the Senate in 52 and joined the President there, and the President became leader. And their friendship continued.  They got over that, and then I think it was hard for the President when Dad came out against the Vietnam War that he did it. From a Hawks perspective, he'd been on a carrier. Young pilots said, why does my government worry more about North Vietnamese lives and my life? And Dad said, what are you talking about? And then he showed him the kind of targets they were hitting that they didn't think were that important but well defended. And then the captain came down and said, yeah, that's it. So Dad said, I'm against the war and made a speech about it. That was 1966 or 7. And I think that hurt a little bit because I know it did because I remember I was chief of protocol at the time, and we'll get back to that in a minute, but Hubert Humphrey, I took the Prime Minister Lee of Singapore, who was one of the few Asians who were totally supporting President Johnson and a very anti-communist kind of guy, a tough guy. Double first at Oxford and that kind of stuff. And I took him in to see Humphrey, Vice President Humphrey, who had no idea that he'd be a candidate for that office at that point later on and said, Humphrey said, well, Mr. Prime Minister, even though Jimmy's dad doesn't think this is a good idea, we know it's the right thing. It was a little awkward for me because I wasn't there to defend my father but to present the Prime Minister. And as we know, Humphrey himself had to carry that load through his own campaign and began to switch toward the end. But anyway, all that's over the dam. I would like to put a marker down that I think the President was enormously strong to carry all that he did all those years and get so much legislation passed, lean on the Southern senators, get the civil rights through. I don't know anyone could have done it. Anyone else could have done it in that time frame. I don't know if we've ever had a President could have done it that way. So of course the war wore down his strengths a lot and his serenity. Mrs. Johnson, of course, was always such an important person in his life and kept him going beautifully well. So I first met the President myself right after he came to the Senate. They lived in Chevy Chase and the girls were little and dad took me out and mother to have tea with the Johnsons on Sunday, a couple of them sat on that. They had a sort of veranda porch and it was kind of fun to see these two boys talking about the country and the future. And I don't know that they golfed together. I don't know the President that much of a golfer but dad played a lot in those days with Clifford and that crowd and Smathers, who was a golfer, but those were the days when they talked about the committee which would meet and with Rayburn and all they'd sit around and talk over a little bourbon about things. It was a congenial time for the Senate. There was a lot of mutual respect and mellow discussion. Friendship, parenthetically, I don't quite see that today and people all seem to be on a slightly different track and they're carving out their own careers. There seem to be a symbiotic relationship in those days between some of these titanic individuals, Russell and Irvin and Senator Johnson and dad and all. Those were pretty good days from that perspective before he became President. They got it much harder for him. The presidential race, dad didn't go into the primaries, I think he felt and was advised by Clifford that it's very costly, you might be perceived as a Protestant alternative to Kennedy as your friend, you don't want that to happen and the primaries are expensive and there's no need because Johnson and Kennedy are so strong they'll cancel each other out and you're everybody's second choice. That wasn't bad advice and it just failed by one state. He got all the way to Wyoming on the first roll call and they put Kennedy across. So then the question on the vice presidential thing, dad, I don't know how interested he really was in the vice presidency but it seemed logical that everybody seemed to want him and I remember he called us up to his room and right after the nomination of Kennedy and said, boys, we got a question now, do I accept the vice presidency? We said, no, you don't want the vice presidency, you didn't come here to carry a guy's coat around for four to eight years, you're prime of life, you go to go back to Washington and get to work and make your own views known and felt. So you don't want to do that. So that's very interesting. Now Clark, what do you have to say? And if you've ever heard Clark as a barrister, you'll know what we went through at that point. He said, well, let me see now. We've nominated this young Irishman to head up the ticket, the Democratic ticket. He's their standard bearer. And let's see, he'll be running against Richard Milhouse, Nixon, is that right? Would you fellows agree with that? Yes, of course. So he'll have many questions for his fellow Democrats and how they can help him. I'm just turning over my mind. What question could he have that's more important than will you go with me? Will you be my running mate in this challenge? And what I hear you boys saying is, no, you don't do that. Give us something else to do. We're not going to do that. Is that what you're saying? So we said, OK, we give up. So I remember going to bed that night thinking, well, that's the way it's going to be. In the morning, my wife and I were in the hotel room in early morning and turned on the TV and there was Kennedy. He clearly looked like an unmade bed. He hadn't been to sleep for 24 hours. And he got all the names and titles wrong. He said, we were thinking of Senator Loveless of Iowa. Loveless was the governor of Iowa. He said, we gave some consideration to Senator Docking of Kansas. Docking was the governor of Kansas. So I thought, boy, this guy's really coming apart. And then he said, OK, and we certainly gave consideration to my good friend, Senator Signington. So we got the title right. And that's what we call mixed emotions because in a way we were not sure he should be a vice president candidate, but totally convinced he had to do it if he was asked and probably should do it. But that's when we learned that Johnson, because of the Texas and all the important reasons for taking. So my dad was very, very calm about it. I think he felt maybe even slightly relieved. If you saw the way they each spoke after Kennedy's announcement, and dad threw away his notes and said some lovely things about Kennedy as being the reason he's up there, not any of us. He worked harder. We got the issue straight and everything. So he was very noble about it and correct and went about his business as we had expected and hoped he might do in the Senate. And that took us in, well, it was another few years before, well, let's see, that was 1960-61. And that's when dad got involved in the, what we thought was the missile gap, that national security was his most important issue, although we took a lot of interest in other things like farming from Missouri, that's an important issue too. So I didn't really catch up too much with the president again myself until, let's see, 1964, he ran on his own. And that's when I took my guitar and sang all over the place, folk songs and stuff. And I spent some time with Linda and Lucy making barbecues come together and maybe they came out to St. Louis and Bush's Grove, Gussie Bush's marvelous farm out there and they all came out for a terrific barbecue fundraising thing. And on the strength of that, because of all the guitar playing, I think the president thought I knew something about music because when he was getting ready for the inaugural, he named me Chairman of the Music. I knew absolutely nothing about bands and orchestras. I remember I sang with one once when I was in New York in law school, but that's about it. So I got hold of Peter Duchin of New York, an old friend of mine, and he helped me put these bands together. And we were very good. I think we got a lot of bands from around the country, lesser known ones, but we had, of course, Lionel Hampton and some of the great bands. We had to have Meyer Davis and, I forget that other guy, but we had a good inaugural. And in order to go to all the sites because of the traffic, I rode a bicycle all over the place. I just got on a bike and went from one hotel to another and to see that everything was going smoothly. It was fun. And then I was asked to come back. I had been, you know, in the Kennedy administration as a food for peace deputy director under my government, which was a great experience to travel around and get people eating American wheat and drinking the milk. It was a wonderful thing. And then I had been Bob Kennedy's AA in justice from 62 to 3 during the Cuban Missile Crisis. And this was going back before Johnson was present, Kennedy was present. But I'm just going back over the years. And then I went back into law in Arnold Fortes and Porter in 63 just before the end of the President Kennedy's life and term. And was asked to come back into the Justice Department as I think they called it executive director of the President's Committee on Juvenile Delinquency, which was a committee consisting of Katzenbach as chairman, he being the AG John Gardner, who was this marvelous fellow that was Secretary of HEW, I think they called it at that time. And Willard Wirt, Secretary of Labor, because those are the three agencies whose policies impacted on young people one way or the other. And I really enjoyed that job working with judges and juvenile courts and probation officers, trying to think of ways to get kids out of trouble and keep them out of trouble. And that's when the President in 1966, out of the blue, I was at a conference of judges in New York and the phone rang. And they said, it's the White House, it's the President. I said, yeah, sure, it's the President. And I went into the phone and I heard the voice, Jimmy, yes, Mr. President, Lloyd Han has quit. Will you do it? Quit what, Mr. President? He was chief of protocol, and I'm asking to do it. Will you do it? Will you know? When the President of the United States says, will you do it? Even though I wasn't quite sure what it was all about, you say yes, and then you run to the dictionary and try to find out what it is you've agreed to do. But I said, yes, sir, Mr. President, good, I'm glad to hear it. And so I remember calling up my wife, Sylvia, and say, get ready for this one.  And when Dean Rusk administered the oath of office, he said a funny thing. He said, you've now graduated from the world of juvenile delinquency to the world of adult delinquency. And so it was. I remember I had to go running off to the White House to get to meet the ambassador of the Sudan. And this was my, within the first half hour of my stewardship of the protocol office, I meet this fellow who speaks French only. We got through that all right. We went inside the house, the diplomatic entrance, you know, the back side. And I had no idea where to take him. And I looked at the ushers like, you know, give me a hint. They just backed away and smiled. They figured, you're the chief, you know, so I walked down this corridor. I finally decided to open a door and we walked in, it was a broom closet. And he said to me, same old premier, it was my first time in your country, as if to say, smoke me out. What is this ritual? Did it happen to George Washington or something? So I said, it's my first day on the job. So we went up to, we all laughed and went up to the president and received this fellow. He was the, did I tell you, ambassador of Sudan? I think he was. And so we had a great run in that, I say we because it's kind of a team thing. If it's a male protocol chief but there have been a lot of very good female ones and I think their husbands have ducked the duty but the wives don't get to do that. And you know, the president was, he was very generous and good to us. He could only, he got sore a couple of times. And I disappointed him once or twice. I know. I'm thinking of one time when Lucy was getting married, he said absolutely no gifts from the world. I don't want the country saying the president's getting his daughter married and getting all this. These treasures from 117 other countries. So I let the dean of the diplomatic corps know that we will not countenance gifts. So we want, we were trying to forestall the possibility of any gifts. And he understood and he produced for Lucy, the entire diplomatic corps put some money into a kind of a silver tea set or something. It was a nice way for a general collective gift and I thought that was the end of it. One day I got a call from the president. Jimmy, what about no gifts? I said, yes sir, there were no gifts. Well apparently, Haile Selassie, the lion of Judah decided that never mind gifts, I'm going to give a gift and he gave it, but here's what happened. It was sent to New York. I think it was a golden place setting thing and maybe it was very valuable. And I had known about it because the customs had called me and said, what do we do with this? And I said, just hang on to it, put it under wraps. We'll decide later, you don't reject a gift from the emperor of any country because, and I'll tell you a minute, what happened to Jefferson when he had a problem like that? So I thought I'd handle it, it won't get out and that's when the call came in, Jimmy. There was a story going around and Haile Selassie sent something and he said, I think the State Department might have leaked it. And I said, Mr. President, I can't imagine the State Department, anybody doing such a thing. You can't imagine that, boy, you got no imagination, I gotta tell you. In fact, I don't think I put it quite as succinctly as he did, holy smokes. I didn't have any idea, I mean, I was a little naive, I guess, but anyway, we got through that okay. There was a couple of other things in Thailand, for example, he said, no, I don't want a lot of these long anthems because I get hot. Well, the anthem, when we arrived in Thailand, it was about five minutes long, and he was standing with his head bowed at about 112 degrees of heat looking at me through the side of, so I was trembling throughout that. I'm jumping around and I'm trying to think of the embarrassing moments. There was one more here, I want to- Do you know what? I don't know. I gotta tell you, I think it was just swallowed up, but it may have wound up in the GSA where GIFs, I had to testify before Senator Fulbright's committee anyway on GIFs to American diplomats or officials when they go abroad because apparently the law as it was when I took office provided that you could accept anything, but you couldn't keep it until after you left office.  And so Fulbright said, look, let's say you can't take it at all if it's over $100 or something like that in value. Well, that seemed like a good approach, but then you could keep it yourself and the department doesn't have to be a storage house. So I had to go and testify on behalf of that bill because the President supported the bill and I was the kind of the steward of the results of the bill. And so Fulbright said, what do you think of this bill? And I said, Senator, if you like it, I like it. And Fulbright turned my father was on his committee and said, I haven't had an answer like that many, many years. Anyway, so we lived with it for a while and I think they've had to hike it up a little because there's hardly anything under $100, not except a plastic doll or something like that. So I haven't checked lately to see how protocol handles gifts, but I think probably they've raised the amount. And it's not uniformly enforced or anything like that. I think probably some people shave on that a bit, but it goes back. It's a constitutional requirement that you don't accept emoduments and gifts from foreign governments. Getting back to the idea of refusing a gift upfront, which is an affront, apparently there was a consul in North Africa during Jefferson's presidency who received the viceroy of a certain sultan and the viceroy said that sultan is pleased with your president and pleased with your country and wants to present a pair of match science to your country. And the consul said, well, you know, we have a law, you can't do that. So thanks a lot anyway, but you can't do that. And so the guy said, well, that's too bad for two reasons. One is there'll be no further relations between our countries and the other is not so important really, generally speaking, but quite important to me, I will be beheaded. So the consul says, give me another crack at this. So he writes a note to Jefferson. It goes a little boat across the Atlantic and Jefferson reads it, okay, take the stuff. And I think they were sent over here. But that's the moral is, the lesson is, you know, the receipt of a gift and the way you receive a gift is just as important, if not more important than the giving of a gift. That's, it goes all the way back to the days when tribes would not knowing each other too well would leave some food outside the camp or the other one say, look at this, if you take it, that means we might be friends, you know what I'm saying? Go after one another. So there are a lot of those niceties like that in the protocol. I, when people said, what is protocol? I came up with some kind of a description that it is really the practice or the art of establishing the context in which conversations and relationships between heads of state or their ministers can occur with a minimum of difficulty of embarrassment. And so because of that, many rules were set up to make it work that way. For example, before the Treaty of Vienna, if the French ambassador and the German ambassador both wanted to be admitted to see the Queen of England or something like that, they would be fighting in the doorway to do it. And people actually got wounded and maybe a couple of were knocked off to get their entourage inside before the other one did.  So they, that's when they, I think the Treaty of Vienna established the order of precedence that the country whose ambassadors, whose credentials have been accepted ranks the one who came after him all the way down. And that is why, of course, here in Washington, instead of having a French and Dean of the Corps, you had the Nicaraguan Dean of the Corps, Guillermo Civilius Acasa, because he'd been there longer than anybody else. And he was one of, I think, Samosa's brother-in-law or something like that. And a fun guy, and I really liked him and, of course, his crowd was knocked off later on in Nicaragua, but he's a good man, pompous, but funny and pleasant and I liked him a lot. The other fellows that I worked with, I think were Dobrinin, because Russia, he stayed there a long time and I had spent some time with him going up and down the elevator during the Cuban Missile Crisis when he was presenting his government's case to Bob Kennedy in the Justice Department. And eventually he got braced in the White House with those photographs and that took care of that. And that moment of when the quarantine was laid down, that was a tough moment. I'm taking back again to Kennedy presidency, but all of us were a little nervous about what the Russians would do when they came up to the line. They turned around and went home and they said, Khrushchev blinked. It's true. And what we didn't know was we'd agreed secretly, really, to take our stuff out of Turkey in response to that, which was okay. There's a lot of foreign policy is better not published anyway when it's happening so that it doesn't get interrupted. But Johnson, one time we were getting ready to come downstairs, you know, the great winding stairs in the White House to a reception, maybe a dinner for a king or something. And they played Hale to the Chief. Ba-ba-ba-ba-ba-ba-ba-ba-ba-ba-ba. Of course, I was the Chief of Protocopsy, so I turned to the President and said, Mr. President, our song, he looked at me like, are you kidding? So you didn't, light banter wasn't always the way to get along. There's something desperately I'm trying to remember to tell you again, think what it is. Let me see what I got here. How brief was he on matters when you took somebody and visited with him? Pretty well, because you see the way it would work when the department and the White House together decide that they'll accept the visit from a certain sovereign, the whole team at the department, the regional assistant secretary, the desk officer of the country, they put together briefing papers. And I had the impression the President read those quite thoroughly. And his questions were to the point to these fellows. And some of them were so overawed and his President couldn't talk at all. And of course, that sort of hampered the exchange. This one fellow, I think he was the head of Lesotho. He was tongue-tied. And the President kept leaning forward like a dentist looking down his throat just hoping something would come up, you know? And the guy couldn't talk at all. And he was a large man, about 400 pounds, something like that. And I remember saying, to sort of break it all up, I said, well, President, all I can tell you is that, Chief, what was his name? Can't remember. It was very happy to have his visit here. He's had some very good visits with the State Department representatives and the Commerce Department. And the guy looked at me and just nodded like that because the visitor, he couldn't talk. He was like, oh, oh. And so the President's, well, I'm glad to hear that.  And he never had said a word. Unlike Hailey Salasse, he never stopped talking. He, the minute he got into the President's office, he let fly. And he went on, I think something like an hour had been set aside for the visit. And Hailey Salasse took it up entirely. And so the President, after he finished, said, well, very interesting. Amber, I'm sure glad to see you. Have a good stay. Never responded because he'd never had a moment to do it. I think he was kind of glad Hailey went away. We had, but most of the visits were much more productive than either one of those two where there were real exchanges. We had something like 70 state and official visits while we were there. And the President had helped in another way, and so did Mrs. Johnson. One of the things that I thought would be useful is for the diplomatic corps to feel that they had contact with the President, even though they really don't. After they present their papers, that's maybe the last time they see the man, or maybe at a reception or something really big, or if their king comes, as like say, Indira Gandhi came. Then, of course, the Ambassador gets to see the President in her presence. I'll get back to that Gandhi visit in a second. But what I was trying to do was to bring the corps into a situation where they felt they had been with the President. So Jack Valenny and I had the Valenny-Symington lunches which were in the White House. We would invite about maybe up to eight compatible ambassadors, i.e., you don't put Israel and Saudi Arabia in the same room. And we would invite them to lunch. And then over coffee, the President would come storming in. What's going on here? And shake hands all around, you know. Let me tell you what I had to do with the Congress today. Let me, and he would sort of gripe about the Congress. And these fellows would be absolutely thrilled to have this going on, you know. They've had a good lunch. The President of the United States been with them. They can cable their Minister back home. I've just spent lunch with the President and it gives them a real feeling, you know, of contact and of a secure relationship. When these, at the time that they were presented, because there's so many of them, we had to usually have two or three be presented successively and then take them all into a waiting room for the President to talk to. And, of course, that was awkward for most of them because they didn't want to take up, you're not supposed to take up a substantive subject anyway on your presentation of credential, but so that's how you get away with doing it that way. But the President would come in and say, if all is anything going on, I should know about, you know, and they'd say, well, Mr. President, now we're very glad to be with you. I remember the President of Bulgaria, I mean the Ambassador of Bulgaria who accompanied, I think his Prime Minister made an observation that strawberries were the principle product of Bulgaria or something. I may have this a little mixed up, but then the President said, that's very interesting, strawberries. Well, about a month later, I encountered this Ambassador at a reception, he said, I have the most terrible thing to tell you. What could it be? I said, I meant to say blueberries. I said, well, not to worry. Now, I gotta tell you about Mrs. Gandhi. When she came here, the way a state visit works, mind you, a state visit is a visit by a head of state as head of state. An official visit, which contains many of the same details, is a head of government coming who may or may not be head of state. Usually, if it's the sovereign, it's called a state visit. For example, I think Mrs. Gandhi was the Prime Minister, and I think they had a President too, like France. So she may not have been considered the head of state. If I could be faulted on this, I accept responsibility. But anyway, she was treated with a state visit. And those two forms of visits have the same contours. It's a three-day visit. It's an arrival, the White House lawn, you know. We used to do other, there used to be other ways of getting the sovereign to see the people like driving through the streets, but so many, so few showed up that it was a little embarrassing, the Saudi Arabia guy, he didn't have anybody to look at, and the President was glowering at me because we were going through all these empty Washington streets. So after that, the idea was to bring in the federal workers, a couple of thousand dependable people who could cheer and laugh and bunch them up in the White House grounds there on the south lawn, and then be prepared to receive the head of state and the President who would then address them. And so that's the first thing you do. Then there's a lunch, I think, at the State Department, and that night is the state dinner. And the next day is a reception at the Embassy of the Visitor where the President comes to the reception and the Vice President accompanies him and the President returns to the White House and the Vice President stays as the guest of the Visitor. And then the third day, there's other visits to the Foreign Relations Committee and things like that in between and then they go home on the third day. So when the President went to the Embassy of India on the second day to the reception prior to the dinner they would have for the Vice President, he stayed and stayed and stayed. And finally, I looked at my watch and it's the President's time now. We should take our leave and we'll go return to the White House and Mr. Humphrey will remain. And he turned to Mrs. Gandhi and said, do I have to go? She said, of course not, Mr. President. We'd be thrilled to have you stay for dinner. So what happened at the Court of Poor Humphrey, he would stand all the way to the bottom of the table and you'd already see him. And then the next day the press really leaned on me. That was a breach of protocol. The President was supposed to go. And I said, no, I think not. I think you've got that wrong. If the visitor, the visiting sovereign, is a woman and invites the President for dinner and the President accepts, that's protocol that got around it that way. But that was quite an experience. I got ready to run for Congress in 1968, which was in the end of my second year, I guess, as protocol. 1967, Christmas was the visit of Levy-Eschkahl to the ranch. Parenthetically, having your visitor come to your home spot is not only even more of a gesture of affinity and friendship and acceptance, but it's a more comfortable thing for someone like President Johnson. And I think when he did that, he didn't do it a whole lot, because he couldn't, but it made it quite a fine impression.  And I sang some folk songs for them. I remember Lucy's baby was very small and sat on the President's lap and we had quite a time. And then I took the occasion to tell the President, later on, I thought I was gonna run for Congress the following year. This was Christmas time, 67.  He said, if now, I'll campaign four-year again, you whichever you think will help you the most. That was lovely. And he, of course, was feeling himself, December 67. I think what he was going through his mind, getting ready for what he would say, and in 1968, I forget the date that he went on television, but that was the same night. Oh, he asked me, this is important. Oh, when I told him I was gonna run for Congress, he'd find, he said, do me a favor, do not announce until the 1st of April. Next, I said, of course, Mr. President. And that was a little awkward because of that three or four candidates in there and they were all howling up and down. And I was saying, well, it's interesting, but I don't think. Well, guess what? I think it was March 31st and I'm having a little dinner with some friends in the Blair House, a farewell dinner thing, and someone calls us to the TV there as a president announcing his decision not to be a candidate for reelection. And my mind rolled back to that moment in December and I realized that in my judgment, he was turning over in his mind what he might do, but he didn't want members of his official family signaling in one way or the other what his plan in other words, leaving the ship, you know, the ship's going down, getting out of here. And so I'm quite sure that that's why he asked me to wait until April 1st and that date must have been in his mind. Why would he say it otherwise? I've never had that confirmed by anybody else, but maybe some of the other folks you talk to and find out what they think, that MacPherson and others. But anyway, that was the thing and I felt, of course, I went home the next day and announced, but needless to say, my announcement was somewhat dwarfed by this. So hardly anybody noticed that I was going to run for Congress, but that's what we did. And he wrote a wonderful letter. I should have brought it to this, but to me, thanking and thanking Sylvia for her stuff. Because what Sylvia did, Mrs. Johnson has a very quick study on people and on ideas and was so supportive of her husband. What Sylvia did with Mrs. Johnson's agreement and participation was to bring over the wives of compatible ambassadors for tea in groups of half a dozen or so. But she would brief Mrs. Johnson single-page, one-pagers on each of the wives. So these ladies would come in and be quite impressed to go up to family quarters, sit and tea is served and Mrs. Johnson would turn to the wife of the ambassador of Madam Malagasy or something. Mrs. Anabuba, what interests me most about your career is that you were the golf champion of your capital city. Mrs. Johnson, how in the world did you know that? Oh, everybody knows that. So it really made quite an impression. And those, I really think that the, I don't know of a time when the diplomatic corps felt more comfortable, even though the Vietnam War was on top of all of us with our president then in those days. And I think that was fun to kind of make those arrangements. The other thing I tried to do was to make a call on every single ambassador and just say, without an agenda, is there anything you need or anything I can help with? And boy, you've got to know a lot of these fellows that way. I only got to 70 because I couldn't work them all in. At that time, there were 117 countries. Today, forget it, there's about 100, how many, 150 or something, can't keep up with them. And of course, when the Soviet Union broke up, bingo, you had about six more, I forgot to say. So the job of protocol is getting taller, harder and harder. I was lucky because I had such wonderful staff. Felling him Chet Carter, he was an African-American. Boy, he was good, he was a deputy. He'd been a Peace Corps director in some countries. And I inherited him from Andrew Biddle-Dook and others. Of course, Lloyd was a great friend of the president and it came on trips even though he was no longer she because the president adored Lloyd and they were great, they were very close. I think Lloyd ran for lieutenant governor of California. Will you be talking to Lloyd? You must be. Yeah, sure. And I think he decided to run.