Ladies and gentlemen, the start of the program, Our Miss Brooks, brought to you, usually at this time, sponsored by the Colgate Palm Olive Peat Company, is being delayed momentarily due to technical difficulties beyond our control. In the meantime, we offer you Organ Melodies by Dick O'Rant.  We've delayed the start of our Miss Brooks program, brought to you by the Colgate Palm Olive Peat Company. But now, technical difficulties being restored, we take you to Our Miss Brooks. The importance come up that I've got to discuss with Mr. Boyden. I'm meeting him at the zoo right now. Tell you all about it when I come back. Goodbye. Oh, Connie, you left a couple of letters on your dresser. She must be in a hurry. She didn't even open her mail, except this one. Dear Miss Brooks, as principal of Sunnydale Finishing School, oh, I shouldn't be reading Connie's mail. I'll get on with my vacuuming. First, I'll get this corner here. It's pretty dirty over by that dresser. As principal of Sunnydale Finishing School, I hereby acknowledge your application for a position. Oh, I'm terrible. I'd better get over to the other side of the room. Pretty dirty around Connie's letter, uh, dresser. Application for a position as an English teacher at our school. This is to inform you.  I can hardly hear what I'm reading. This is to inform you that we have an opening at Sunnydale, and we're looking forward to a personal interview. Very truly yours, Jonathan, if Margaret Davis, don't you dare read another word of this letter. No wonder Connie was so excited. Sunnydale is one of the most exclusive schools in the country, but that would mean her leaving Madison High. Oh, this is awful. I'd better phone our good concrete. As principal of Madison High School, he ought to know about this. Hello? Hello. Martha. This is Margaret Davis. Oh, hello, ma'am. It's a fine day after Thanksgiving. Well, frankly, Martha, I'm a little upset. I saw yourself into a coma yesterday. Oh, then maybe you'd better not tell him today. What? That our Miss Brooks may be leaving Madison High to take a position at Sunnydale Finishing School. Sunnydale Finishing School? How do you know, Margaret? Did Miss Brooks tell you? Not exactly, but I got it from the principal of Sunnydale himself, Jonathan F. Jonathan F. Who? Hold the wire a minute. F buyers. Is he there now? Yes. Right on Miss Brooks dresser.  Oh, I promise, Margaret. Just by accident, I just happened to read a letter that was lying on Miss Brooks dresser. By accident, Margaret? Yes, I just happened to have my glasses on. But what are we going to do, Martha? We can't let Miss Brooks leave Madison. No, we certainly can't. Tell you what, Margaret, I'll talk this over with my daughter Harriet and call you back later. All right. And remember, you promised not to mention a word about my reading Miss Brooks' mail. Of course, my dear. Goodbye. Goodbye. Oh, I hope they'll be able to think of some way to keep her here. Oh, hello, my nerve. I'll get you some milk in a minute, dear. No, no, no. No, no, my nerve.  You know what it does to you. You seem tight. Of course, Mr. Boyden. I thought you were a little tall for a pelican. It was nice of you to meet me. How'd you get over here so quickly? It was simple. I dressed on the way over.  Well, perhaps we'd better sit down while I tell you. Here's a bench under this tree. Fine. Eh, you comfortable? Perfectly. Miss Brooks, I... Yes, Mr. Boyden? Well, you must forgive me if I seem overexcited. But frankly, I never thought this day would come. You didn't? No, I didn't. Well, before it goes, why didn't you think it would ever come? Because of... Because of the things that have happened in the past, or rather because of the things that haven't happened in the past. Those are the ones that bother me, too. You realize, of course, I'm referring to the migratory habits of the arctic grackle. Oh, I realized that a long time ago. I just didn't want to let on. This is the first time a grackle has ever been south of the Canadian border. Maybe he was waiting till after the election. I knew that something was afoot when I looked out of my window this morning and saw a black and blue bullfinch. What happened to him? Get caught in a badminton game? The movements of the grackle can be very accurately charted by closely observing the bullfinch, since they are in the truest sense of the word full cousins to the mottled thrush. Where did he come from? Hudson Bay. And almost all of these species would like to lay their eggs in a soft downy nest amid quiet surroundings. That's why I sent for you, Miss Brooks. Good.  I know this sounds terribly involved, but actually it's quite simple. You see, my uncle's farm in Booneville is a perfect nesting ground for both bullfinch and thrush. How does your aunt feel about that? Well, they both love birds. I don't worry for them, and I have an idea that if I get right up there, I may stumble into some grackle eggs. Sounds like fun if you don't mind albumin on your oxfords. That's why I asked you to meet me, to say goodbye. Goodbye? But Mr. Boyden, haven't you heard the old saying, two grackle hunters are better than one? Oh, but you couldn't come along with me. It's a ten-hour drive to Booneville and we'd have to stay overnight. But we'd certainly be well chaperoned, what with your aunt and uncle and all those bullfinches. Yes, they did have 600 of them up there last year, but I don't know, Miss Brooks, even with the chaperones, you know how people talk and unless we didn't say anything about it, but even then they might find out, although we'd only be gone a couple of days, still if the rumor got around that we were... Of course we wouldn't be, but then there are those who might... Oh, but how could they? What do you think, Miss Brooks? Why should I, by then, decide among yourselves? Well, it is kind of a long trip to make alone. Maybe you're right, Miss Brooks, why with the two of us traveling together, time will fly. We can chat together, eat together, and pay for the gas together. I still insist that no one should hear about it, though, for the sake of both our reputation's absolute secrecy is essential. Not a word of this must cross our lips. Can I depend on you not to divulge our plans to a living soul? On your honor, Miss Brooks?  Our Miss Brooks, starring Eve Arden, will continue in just a moment, but first, here is Vern Smith with an important announcement. Attention, ladies, regardless of age, skin type, or previous beauty care, doctors prove you too may win a lovelier complexion with palm olive soap. But to win this lovelier complexion, the kind men admire and women envy, you must stop improper cleansing. Instead, use palm olive soap alone the way doctors advise. Remember, 36 doctors, leading skin specialists, advised 1,285 women, many with complexion problems, to use palm olive this way. Some had dry skin, some oily, some coarse-looking. Using palm olive soap alone, 2 out of 3 won lovelier complexions, regardless of age, type of skin, or previous beauty care. Now, here's the plan doctors advise. Wash your face with palm olive soap. Massaging for 1 minute with palm olive's soft lather. This cleansing massage brings your skin palm olive's full beautifying effect. Rinse. Do this 3 times a day for 14 days. It's that simple, but leading skin specialists prove this way using palm olive alone, nothing else really works. So forget other beauty care. Use palm olive soap as these doctors advised for a lovelier complexion. Last night on the CBS Sing It Again program, you were promised an additional clue to the Fatem Voice and the $24,000 prize. Here it is. The Fatem is a famous ghost who galloped far from post to post. Yes, there is the clue for the $24,000 prize on the CBS Sing It Again program on Saturday night. The clue again, the Fatem is a famous ghost who galloped far from post to post. And now as she quietly prepares for a weekend in the country, our Miss Brooks is blissfully unaware of the furor caused by the 5-year-old letter which she left on her dresser. Let's look in now on Martha Conklin as she discusses the situation with her daughter Harriet. So you see Harriet, it looks very much like your favorite teacher will soon be leaving Madison Howe. Oh, that's terrible, Mother. Have you told Daddy about it yet? Not yet, Harriet. I'm waiting until he recovers from his Thanksgiving dinner. You know how upset he gets when we mention the shortage of teachers. Oh, yes. Golly, Mother, he's been lying over there on the couch for hours. Do you think he's all right? Of course he's all right. He's just sleeping. Aren't you, Osgood? I say, aren't you just sleeping, Osgood? No, no, no. Take it away. Take it. I guess I was having a nightmare. A nightmare, Daddy? Yes, I dreamt I was still eating. I swear I don't know how those pilgrims did it. Did what, Osgood? Ate all that turkey and fought Indians beside. I'm surprised at you, Daddy. That's not the real Thanksgiving spirit. A lot I've got to be thankful for. Well, you've got us, Daddy. Mother and me. Yes. I've got you all right. I've also got the school. A school full of overcrowded classrooms and a horde of unreasonable teachers constantly screaming for a living wage. Now, Osgood, don't work yourself up into a stew. Oh, please. Don't even mention words like that. You haven't got the right attitude, Daddy. Nowadays, if you can hang on to the good teachers, you've got anything else you get as gravy. Me! Me! Not you, too. Maybe we ought to go out and let your father sleep a while longer, Harry. Yes, that's a splendid suggestion, Mother. Oh, this couch feels good. Oh, I'll get it. Coming. Hi, Harriet. Hello, Mrs. Conklin. Hello, Walter. And how are you, Mr. Conklin? You certainly look peaceful. I was, then. Did you finish your lunch, Harriet? I just finished mine. We had plenty of turkey left over from yesterday, and I just ate the part that went over the... Oh, no! You can't take him out of here! What's the matter with your father, Harriet? Doesn't he feel good? Not too good, Walter. Let's go into the other room. Okay. I hope you feel better, Mr. Conklin. I guess it was a mistake to hash over Thanksgiving again. No! Come on out in the kitchen, Walter. There's something we want to tell you. Guess what, Walter? Miss Brooks is leaving Madison High. What? How do you know, Harriet? Mother found out from Mrs. Davis. Mrs. Davis? How does she know? We can't repeat that, Walter. Repeat what? That Mrs. Davis read a letter Miss Brooks left on her dresser. It was a letter offering her a position at Sunnydale finishing school. Sunnydale? But that's 500 miles from here. Oh, we've got to do something. Maybe we can cook up some kind of a scheme to make her stay. I know. We'll pretend I've become a delinquent. But do you think it'll work? It's got to work. Have you told Mr. Conklin about this? Not yet. We didn't want to upset him. Well, he'll find out sooner or later, and there's no sense in waiting until it's too late. Somebody ought to tell him right away. I agree. Don't you, Mother? Yes, Harriet, I do. Somebody ought to tell him. Walter? Don't look at me.  Don't be silly, Walter. Mr. Conklin won't eat anything for two more days. I'm busy right now, Mrs. Davis. She would come around just when I'm packing. Well, come on in. Why, Connie, you're going away. No, Mrs. Davis. I'm not going away. Then why are you packing that bag? I'm going away, Mrs. Davis. I'm surprised that you, Constance, leaving town without even saying goodbye. And after we've been so close. Oh, please don't feel that way, Mrs. Davis. I just didn't want to discuss it right now. You see, I have a lot of things to do. Oh, I don't want anybody to see me like this. You answer the door, Connie. I'm going into the kitchen and make some tea. All right, Mrs. Davis. Hello, Miss Brooks. I've just got to talk to you for a minute. Come in, Harriet. Let's sit here in the living room. I can't stay very long, but I'll... Then it's true.  Leaving town? How did you know, Harriet? What's the difference? I know. And I realize it might be a pretty good opportunity for you. Pretty good. With half a break from the grackle, it's perfect. Oh, look, Harriet, I appreciate your interest, but I think I have to make my own decision in a matter like this. Oh, I know it seems attractive now, but later on you'll regret it. Well, I'll have to take my chances on that. Well, if you won't consider yourself, consider the ones who care for you. Think of Walter Denton. Is that a must? He's going to pieces. He doesn't do anything with him. You've got to straighten him out, Miss Brooks. What in the world are you talking about, Harriet? It's the career he's decided upon. Walter's been reading a big book lately about the Treasury Department and counterfeiters and... Well, that's nothing to be alarmed about.  But he doesn't want to be a T-man. He wants to be a counterfeiter. A counterfeiter? Oh, I'll get it. Hello, Walter. Come in. Hiya, Miss Brooks. I'm glad I nailed you before you took it on the lamb. I'm going to the living room. Oh, hello, Harriet. How's my little Confederate to be, huh? See how funny he acts. What is all this nonsense, Walter? What do you mean nonsense? I'm making money hand over fist. The only trouble is scratching the green goods. Ain't enough. It ain't? Nah! You've got to get a good queer shover to help you pass the bootle. Queer shover? A paper hanger. You know, snide pitcher. You can have the best cognac or in the world make your slush. But if you ain't got a top build poster to push the flash, you might as well slough the screwy and scramble for the McCoy. Yes, but if I could find my way back alone, I'd leave now. You've got to do something, Miss Brooks. Walter wasn't cut out for a life of crime. Oh, now listen, you two. I don't know what's behind all this, but this is... Oh, it's simple. When you get to where you're going, I'll send you a few stacks of hot to shove. I don't want to shove no hot. I mean... I wish somebody would tell me what's coming. It's a good thing I never built a better mousetrap. Good afternoon, Miss Brooks. Why, it's Mr. Mousetrap. I mean, Mr. Conklin. Conny. It's Mr. Conklin. Oh, hello, Osgood.  A watch pot never boils. Hello, Margaret. Now then, Miss Brooks, I'll be brief. I know that you're leaving town. You, too? Somebody must be reading my mail. On second thought, I'd better go back and watch that pot. Oh, you're Osgood. Miss Brooks, I think I can say without fear of contradiction that I know how to run a school as well as any principal in the country. Well, I guess you do, Mr. Conklin. That's why I'm here, to ask you to reconsider this proposition you've had. Proposition? Yes. I can offer you anything he can. How does Mrs. Conklin say if she knew you were talking this way? She's all for it. What? And so's my daughter Harriet. Oh, now, please, Mr. Conklin. I've had a rather puzzling day so far, but this is... What happens that your daughter's in the living room now? I think we'd better go in there. I'll be glad to. Hello, Harriet. Walter. Oh, Mr. Conklin. I'm awfully glad you decided to put your pride in your pocket and come over here. Did you ask her yet? Yes, Harriet, I did. The rest is up to Miss Brooks. Don't go away, anybody. I'll punch your transfers when I get back. Mr. Boynton. Yes, Miss Brooks. It's I. May I come in? Of course. I've still got a single in the mezzanine. Can I help wondering what happened to that oath of secrecy we took? That's what I'd like to know. I thought you weren't going to tell anybody about our trip to my uncle's place. I didn't say a word about it. You didn't? No. I've never seen so many wagging tongues in all my life. You think you've seen wagging tongues? My living room looks like a delicatessen. I just don't understand... Oh, hello, folks. Hi, Mr. Boynton. Oh, I'm glad you're here, Boynton. Maybe you can help get Miss Brooks to change her mind. Well, I don't wish to appear insubordinate, sir, but frankly, I thought the trip might do us both a lot of good. Both?  Pretty soon, there'll be no faculty left at Madison High whatsoever. We might as well all go. I don't think his uncle and aunt have room for you all. Besides, Mr. Conklin, I think where I go and what I do is nobody's business but my own. And I say it is our business. Why you should want to go traipsing off to Sunnydale finishing school is beyond me. Sunnydale finishing school? Of course. Now, there's no sense in beating about the bush. Mrs. Davis read the letter you left on your dresser this morning, and we know you've been offered a job there. Oh, that letter? Well, that's over five years old, Mr. Conklin. I turned that job down long ago.  But then what were we talking about? You, Mr. Boynton, what was... How was... Why did Miss Brooks just where were you and Mr. Boynton going together? That, Mr. Conklin, is a secret between Mr. Boynton, myself, and 600 bullfinches. Martin, as our Miss Brooks, returns in just a moment, but first... Dream girl, dream girl, beautiful luster cream girl. Tonight, show him how much lovelier your hair can look after a luster cream shampoo. Only luster cream brings you K. Dumit's magic formula blend of secret ingredients plus gentle lanolin. Gives loveliness lather, even in hardest water. Glamourizes your hair as you wash it. Luster cream, not a soap, not a liquid, but a dainty cream shampoo. Leaves hair frequently clean, free of loose dandruff, glistening with sheen, soft, manageable. Gives new beauty to all hairdos or permanents. Four ounce jar, one dollar. Smaller sizes, either tubes or jars. Tonight, try luster cream shampoo and be a... Dream girl, dream girl, beautiful luster cream girl.  And now, once again, here is our Miss Brooks. Well, after everyone found out about our plans, Mr. Boy naturally called off the expedition. But things weren't a total loss, because Saturday afternoon, he took me back to the zoo, and we sat by the aviary once again. You know, Miss Brooks, even though our trip didn't work out, I was very gratified to learn of your interest in birds and wildlife in general. Yes, I'm a great bird for wildlife. Since this common interest of ours has brought us so much closer together, there's something else I want to tell you, Miss Brooks. Something I think you ought to know. What's that, Mr. Boyman? There's a pelican here who holds 18 pounds of fish in his beak. What's he waiting for, the price to go up? Our Miss Brooks show brought to you by Tamaric Soap, your beauty hope and luster cream shampoo for soft, glamorous, green girl hair. Our Miss Brooks, starring Eve Arden, is produced by Larry Burns, written and directed by Al Lewis with music by Wilbur Hatch. Mr. Boynton is played by Jeff Chandler, Mr. Conklin by Gale Gordon. Dentists know what cleans teeth best, and over 4,000 dentists say Colgate Tooth Powder, with a two-minute routine, gets teeth sparkling and super clean.  Brush inside, outside, and biting surfaces. Always brush away from the gums. See how this gets teeth naturally bright. It removes dull film that improper brushing misses. And Colgate Tooth Powder also sweetens your breath. Try it. Buy Colgate Tooth Powder today. For mystery liberally sprinkled with laughs, listen to Mr. and Mrs. North. The exciting, fun-packed adventures of an amateur detective and his beautiful wife. Tune in Tuesday evenings over most of these same stations. And be with us again next week at the same time for another comedy episode of Our Miss Brooks. Stay tuned now for Laminabner, Bob Lamont speaking. This is CBS, the Columbia Broadcasting Service.


Hi, thank you for being here today. So we are going to talk about our recent work with Cluster API. My name is Kotaro. I am a software engineer at L.Y. Corporation. And? I'm Shotaro. I'm also a assistant engineer from L.Y. Corporation. Yeah, nice to meet you. And these are today's topic. First, let us briefly introduce our company and services. Then let us introduce our platform and its scale. LINE is a communication app that connects people, services, and information through various services, especially free messages, voice, and video calls. LINE was launched in 2011, and we now have over 171 million users in a total of four major markets. Almost all of our services are running in our data centers. Verda is our private cloud platform that helps LINE service developers build and run their services on our infrastructure. Our mission is to build a platform that enables infrastructure automation for both provisioning and operation. Verda consists of infrastructure as a service, like VM or VMETAL, platform as a service, and a set of managed services, just like the various public cloud platforms. This is the service catalog of Verda. Now we have over 40 services in Verda, and users can use them easily via the Verda dashboard. Verda is built on top of the sales service principle. Users can manage their resources via a graphical interface, but also use API to operate Verda services. Verda is now becoming larger. We have three regions, over 100,000 VMs and over 30,000 VMETAL machines in total. Our team provides one of the managed services in Verda called Verda Kubernetes Service. VKS is a managed Kubernetes platform for Verda. Our service aims not only to just simplify the cluster lifecycle, but also to provide native integration of various corporate platforms to reduce engineering costs. Our team has seven members, and we manage over 1,000 clusters in total. OK, then let's move on to the next section about the story of why we chose cluster API and how to adopt it with minimizing users' effort as much as possible. This is the overview of our legacy system. We have a cluster provisioner that is responsible for bootstrapping Kubernetes clusters and machines used by those clusters. Machines are turned into Kubernetes nodes by cluster provisioner via SSH connections. In front of the provisioner, we have an API server to abstract basic operations for the cluster itself like creation and deletion. Since we launched VKS, we have faced several issues with our legacy provisioner. The first thing is the management of the cluster provisioner itself. Our cluster provisioner was D1, worked from the upstream code base, including multiple patches for custom features and bug fixes. We've been maintaining six repositories and their patches, which makes it hard to backport upstream changes into RF work. This will be bad in terms of both security and stability. And we've got to need to find another solution. The second thing is about the node management of the cluster provisioner. Our provisioner uses SSH for the initial node bootstrap and established stateful connections to all nodes for managing them. The more users use our platform, the more clusters and nodes are needed to be managed. As the scale of our platform grows, those connections become often unstable. We would like to quit establishing stateful connections for all nodes and make it more scalable. To overcome existing issues, we need a new cluster provisioner that meets our requirements. Verda is built with the power of open source technologies so the new cluster provisioner should be an open source project. Lower maintenance cost is also an important point. We don't want to maintain any fork so pluggable interface will be better to have. Scalability is also required for our DEX provisioner. Based on these requirements, we finally chose cluster API. So what is cluster API? Cluster API is a project hosted by the Kubernetes community focusing on simplifying provisioning, upgrading, and operating multiple Kubernetes clusters. It has a declarative API and a set of controllers that are useful for cluster and node management. The key point here is that cluster API has pluggable interfaces called provider. There are three types of providers in cluster API. Infrastructure provider is in charge of preparing actual resources like log balancer or VM. Boostup provider is in charge of turning the machine provided by the infrastructure provider into Kubernetes node. Control plane provider is in charge of managing Kubernetes control planes. We can use community provided providers for each and also we can implement our own providers. Okay, so which provider did we choose? Basically we want to use community provided providers as much as possible. As for the bootstrap provider and control plane provider, we decided to choose providers powered by Cube ADM, which is provided by cluster API community. So how about the infrastructure provider? Verda is built on top of OpenStack and we have cluster API provider OpenStack, which is maintained by the cluster API community. So why don't we use it? First, we tried adopting capital, however, we weren't able to adopt it directly. Most of use case should be covered by Keppel, but we have several reasons coming from company-specific limitations. Verda uses its own customized API for both infrastructure as a service and log balancers as a service. So forking Keppel is required to use it. Also, we cannot use the community provided image builder for building VM images because of our internal image management policy. We don't want to maintain an internal fork anymore, so what should we do? Our decision is to implement our own infrastructure provider. Cluster API provider Verda is the infrastructure provider for the Verda platform. The key point here is that we only need to implement three custom resources, cluster, machine template, and machine based on the provider contracts defined by cluster API. Keppel has Verda cluster, Verda machine template, and Verda machine respectively. Also, we have a load balancer in front of each cluster, so Verda load balancer resource is also included, but this isn't required from the perspective of cluster API. Building our own infrastructure provider makes it easy to introduce custom features based on our internal requirements. Let us share several examples of our custom features. One of them is user script. Users often want to customize their worker nodes. One use case is configuring kernel parameters of such nodes. Several users want to increase ARP table entries when they want to access a lot of hosts. Another use case is installing node-level software. Some users installed network emulator kernel module and KOSD for a KOS testing purpose. Our Verda machine controller internally merges the user-provided script with the cloud-init config provided by the cluster API bootstrap provider. Another example is a static IP notebook, this feature assigns predefined IPs for worker nodes. It is useful when working with IP ACO system because nodes have the same set of IPs even recreating them. Everything discussed here was on our side, but how does it look like from the user? Thanks to the VKS MPI in front of the provisioner, the user interface and user experiences won't be changed before and after introducing cluster API. Users still can use the same CLI or dashboard to operate both clusters provisioned by different providers. Thank you. From here, I will explain our failure domain design with cluster API, specifically regarding how we treat regions and other ARP zones. Let me start with our region design. Our region design is simple. We provide our Kubernetes service independently among regions. In our private cloud, we have one region in Tokyo and another in Osaka. So we built our whole system in Tokyo and built another separated system in Osaka. By the way, this design diagram implies that each cluster has nodes in only one region. In other words, we doesn't support much region cluster. Thanks to this simplicity, we now can manage our product with small team for seven members. Next, let me explain our multi-AZ cluster design. We take different strategies between control brain nodes and worker nodes. As for worker nodes, users can specify availability zone per node pool. So if users, I mean, application developers, create multiple node pools with different AZ settings, they can distribute their nodes and achieve AZ auto-exploration. In cluster API, our node pool is implemented with machine-deployment customer resources. So much AZ is achieved with multiple machine-deployment distributed across multiple AZs. On the other hand, control brain nodes belongs to only one customer resource called QADM control plane. So we cannot take the same strategy with worker nodes. So how we achieved AZ distribution for control brain nodes? Here's the answer. We utilized cluster API built in failure domain support. Here in this example, Builder cluster is our cluster API's infrastructure cluster resource. And we specify our availability zones in specs of failure domains. In this example, we specify Tokyo 1, Tokyo 2, Tokyo 3 as the AZ. These values are automatically propagated to cluster resources, and eventually, one AZ is picked per machine resources so that our control brain nodes are equally distributed across three specified AZs. The key point is that this is done by cluster API core controllers. We haven't done controller implementation for this feature itself. So we could reduce implementation and code maintenance cost. We believe that this kind of pluggable built-in features are good points of cluster API. Next, let me move on to in-place migration to cluster API. To begin, why we need in-place migration? Right now, we managed two cluster provisioner, legacy provisioner and cluster API-based provisioner. But as you can easily guess, it's highly costly to manage different component for the same purpose. So we wanted to resolve this situation, but on the other hand, from the user's perspective, they want to keep using existing clusters built with legacy provisioner. They don't want to create another cluster and they don't want to migrate the workload by their own. This is the motivation of our in-place cluster migration or conversion from legacy provisioner to cluster API. To achieve our goal, we referred our QBECOM presentation last year from MailSales Spence. They had also worked on migrating clusters from their legacy system to cluster API-based system. In that presentation, they explained how they handled cluster API-related custom resources to replace legacy resources to cluster API-based resources on top of their open-stack-based on-premise platform like VELDA in our case. So it gave us a great knowledge to us. However, we had another critical challenge or issue to overcome. Let me explain it. That's the inconsistency of Kubernetes installer between legacy and cluster API like QBADM. In cluster API context, we face issue in node bootstrapping. Right now, with our cluster API-based system, we use QBADM. On the other hand, with our legacy architecture, a Docker-based Kubernetes installer was used. At the beginning of this migration project, we tried to reproduce the way that's taken by Mercedes-Benz. I mean, we naively tried to run QBADM join command to add nodes to legacy cluster, but it failed. That's due to some Kubernetes setup inconsistency. Let me pick up some examples of the issues we faced in the process of migration. The first gap is API request load balancing. In Kubernetes cluster, each node, or more specifically, KubiRed access QBAPS servers. In HA cluster, such Kubernetes API requests from KubiRed should be load-balanced among replicas of QBAPS server. In our legacy cluster, that's done with client-side load balancing. On the other hand, our cluster API and QBADM-based cluster it uses our cloud-based load balances provided by Verde. To resolve this gap, we prepared two load balances also for legacy clusters to be migrated. Next gap came up after the first gap is resolved. That is subjective alternative name inconsistency in TLS certificates. Kubernetes API access is done with HTTPS requests. Generally, such TLS requests are verified by comparing HTTP request and TLS certificates. In legacy cluster, Kubernetes API access is done with a domain name. On the other hand, our new cluster uses domain for load balances as a target host to access Kubernetes API. So request variation was failed. In this example, HTTPS request to CAPI.example.com was denied by TLS validation mechanism because the host name CAPI.example.com isn't included in the SAM in TLS certificate. So we added the expected domain name to SAM before we proceed migration. The third gap is configuration management. This issue is more specific to QBADM. In order to manage its configuration, QBADM uses Kubernetes API. I mean it stores config map in the QB system name space like QBADM config or QBADM config or like that. On the flip side, QBADM cannot properly handle nodes without such configurations. So to avoid the issue, we prepare such resources, I mean config maps, based on the cluster setups of the legacy cluster to be migrated. The last issue I pick here is H3 member discovery. In our system, we manages H3 clusters together with control plane components with QBADM. Unlike QBAPS server, where load balancer is prepared in front, QBADM have to know every members of managed H3 in QBADM based cluster. H3 members are managed as QBADM static ports, which can be seen from Kubernetes API. Like this, QBADM lists the ports with filter component equals H3 and tier equals control plane. By this, QBADM gets the H3 member. However, our legacy H3 members are managed just as Docker containers, independently from Kubernetes API. This cannot be seen from Kubernetes API. So we needed a way to tell H3 member locations to QBADM. Our solution is to prepare dummy ports on the nodes where legacy H3 instances are running. Like these gaps, we had to overcome some issues of node bootstrapping or cluster bootstrapping for in-place migration. To sum up our project for cluster migration, we have two things to do for in-place cluster migration. The first one is to resolve cluster setup in consistency, specifically with node bootstrap like QBADM. And the second point is to handle cluster API customer resources like the way Mercedes-Benz has explained in the past QBADM session. If you are more interested in the details, please contact us and let's discuss. Yep. Okay, thank you, Shotaro. And next, we're going to share some of the obstacles that we faced and the insights we got from them. Today, let us share two topics. The first topic is controller scalability. Now, the number of clusters and nodes is increasing for both environments. For the production environment, over 2,000 nodes are managed by the single management cluster. Since we have a lot of nodes, the machine controller work queue becomes constantly long. The operation for the cluster, like cluster creation or node provisioning takes time. One of the solution would be scaling up controllers. Another solution would be scaling up. However, it wouldn't be easy since controllers have related forward architecture. Scalability of cluster API is one of our interests. Currently, community seems to be also working on scale testing. Our target for now is 1,000 nodes per single cluster, so we'll continue working on the scalability. By the way, there was a maintenance session about the recent improvements to the performance of the cluster API controller yesterday. Yeah, really good. Thanks for the... Which was introduced in 1.5.0, so check it out. Thanks for the great work and for sharing the effective ways of improving controller scalability. Yeah, if you have interested, yeah, check it out. Next topic is SAD Snapshot and Restore. Disaster recovery is one of the crucial parts for cluster operators. Taking SAD Snapshots will be easy to implement. In our case, we have a job to take Snapshot using SAD Cuttle and upload them to our internal object storage. They're deployed for each cluster and running periodically. However, it's more complicated when we try to restore the cluster from Snapshots. For those clusters managed by cluster API, we need to work together with cluster API controllers to restore clusters from Snapshots. For example, when we try to restore the cluster from Snapshots, we need to prevent cluster API controllers from reconciling the control plane and worker nodes. We currently perform the entire process manually, but plan to automate the whole process. If this use case will be common among cluster API users, it will be nice to have such functionalities on the cluster API side. I was wondering if there is any good architecture to include this as part of the cluster API ecosystem. If you're interested, I'd really appreciate it if you could join this proposal. This is all of our presentation. In conclusion, let me summarize today's contents. I would like to say that cluster API is great for a private cloud platform environment. We can reduce the cost of cloud management because of its pluggable provider concept. It allows us to have custom features which is great for such a large platform with its own needs. Also, cluster API is friendly with a multi-regional model with the fairer domain concept. Adapting cluster API as a next-generation cluster provisioner could be a long journey for a large-scale environment. However, we minimized various costs for cluster operators with platform engineering. We provide consistent UI and UX with API abstraction. It allows users to operate their clusters without knowing the details of underlying technologies. Also, we support in-place migration with cluster API and QBADM to minimize the migration cost. And thank you very much for your time and attention. We are on Kubernetes, so questions are welcome. Also, we can talk in person, so feel free to ask anything about our session today. Thank you very much. So thank you for a great presentation. And I have a question related to custom feature that's stated, IP node pool. Yes. Yeah. The first one is, when was the process resource created and is it tied to cluster or node pool lifecycle? And sorry, could you repeat that? When were the process resource created in OpenStack Neutron? So the static IP resources are created when user create node pool, you know. But also, sometimes user want to expand node pool size, so that's also trigger creation of static IPs. It's internally, you know, OpenStack port resources are used. Is this answering your question? So I think, and is this tied to cluster or node pool lifecycle? Sorry. Is this tied to the cluster or node pool lifecycle? So earlier you said that the process created after user create a node pool. When user created the cluster, static IP node pool is not created. When user calls our APIs, yeah, it's created. So, and I have another question for this. Does the number of ports need to be equal to the number of nodes? Yes, so, oh wait, sorry, it's more complicated, but each node can have one private IPs and one public IP, so user can specify the policy independently. So if user specify both public static IP and public private static IP, yeah, so two ports are created for one node. Okay, so, thank you. Yeah, thank you. First of all, great presentation. I have a question about the capability for users to customize their init scripts when creating their nodes. Is that something that's actually exposed, like if I were to be an user and I wanted me myself to, I don't know, increment the number of TCP ports, of it, it's something that I can do, right? So, how often do people shoot themselves in the foot and do like wrong things with those scripts? And if it happens very frequently, how do you guys deal with that? Yeah, some of the users have used that feature actually, but in that case, the node will fail to provision, so maybe that's not harmful and the user can notice that. So the user will fix it, so actually. Yeah, also, one another background is our Quantities nodes are built as user's ER infrastructure as a service VM, so it's user's VM. So, yeah, we treat it as a user's responsibility. All right, so I'm assuming that you guys have metrics internally trying to predict that your nodes, like nodes that are being created by Kappier are healthy. Does that not impact those metrics?  Yeah, that's a good point. So actually, we cannot distinguish the course, but hopefully, unfortunately, yeah, the user's using such features not so large, so we can manually help such users when they face issues. All right, thank you. Yeah, thank you. Thank you. Fantastic talks. Arigato. I have a few questions. You talked in the beginning that your private cloud is a bit different to what Cluster API can do, and you don't use the Octavia API. Have you considered writing your own provider so that it can be used, actually? Because we are in a similar situation, we also run OpenStack clusters with 8,000 nodes, let's say, and we implemented our own Octavia provider so that we can actually use the Cluster API provider for OpenStack. Have you considered that, like writing your own? Actually, we haven't considered that. That's more because of our organizational structure. So this Cluster API adoption is decided only by one test team. So yeah, also, the code base is not so large, so that's why we decided to write by our own. Okay, perfect. A second question, if I can. Can you share what type of like CNI do you use in your communities? Is it Calico or? Yeah, we use Calico. Yeah, it's the same. And have you considered using Crossplane? Kind of new to this, but it seems to me these two options are similar in a way. Can you say more about this if you have or have not? Or if you can maybe just say what do you think is better for us because we are now in a situation where we are trying to get as much information and decide which way we will go for it. Honestly, Crossplane is not on the list when we consider another solution. So maybe it will be better to consider when you try to decide the final solution will be. But as a result of our adoption, the Cluster API could be one of the options. So maybe it will be. I'd say that's one of the good option. Okay, thank you so much. Yeah, thank you. Hi, I'll try to make my question pretty quick, but I really liked how you had the infrastructure providers, like you kind of made your own kind of custom CRDs, which is really cool. Do you ever have any problems with like your development teams trying to take those and make them kind of a little too fine grained for maybe their particular application or maybe use case that they're using for? It kind of feels like you kind of built a really nice framework for people to do that, but I don't know if that's like the intention or people take it too far. Sorry, I'm not grasping your questions, details, but you are asking our controllers, sorry, could you repeat your question? Yeah, no worries. Yeah, so like in the case of like the Verda machine template, right? Do you ever have problems with people kind of saying, hey, like I need to deploy my application for X, Y, or Z, right, but I need this really complex config added to it. So they kind of like kind of really focus that template down onto their persistive use case and kind of break the formula that you kind of made for them. Actually, I'm not sure if I understood correctly, but user cannot create the Verda machine resource directly so we abstract that layer via the API, so user just request something like the create node, it translates by the VKS API to create the Verda machine resource. I think there is no special request to add some functionality, hey, I need Verda machine, I need this feature to Verda machine, so it's completely independent. And also I think another good information is because this cluster API cluster is just QBADM cluster, so sometimes we allow users to run QBADM command by themselves, for example, to add some PEMs to the clusters, so it's the money only. Okay, yeah, that makes sense. I appreciate your time, guys. Thank you. Maybe is that all? Okay, again, thanks for being here today. Yeah. Thank you.


Welcome back folks. That was up 88. You get the NAS like up 30 S&Ps up nine and a half Let's go over to our man. Mr. Basil Chapman. Don't forget folks basil doesn't outstanding show here every trading day 12 to 1 Eastern Standard Time also has a great newsletter the opening call Now as you get the way you get the opening call you come over to our website of TFNN And you're gonna see right under featured content the opening call I now as you hit that you're gonna see that our man. Mr. Basil Chapman is gonna be doing a Workshop for his subscribers this workshop folks is April 3rd, which is next Wednesday. Yes, you heard that right We're going into April already. Gotta love it the workshop is gonna be at 5 p.m. Eastern Standard Time and What the battle is gonna be talking about is where we are currently in the marketplace how to read the technical tools That will show you where the market is heading the outlook for 2019 and 2020 are we starting talking about 2020? That's pretty amazing The current risk and the potential rewards inside of this market specific sectors and stocks to monitor the workshop is going to be Archive for all subscribers now It's really easy to be a subscriber to Basil's newsletter folks as you come here and to featured content You're just gonna hit the opening call you hit subscribe and you'll get one month for 128 dollars Six months for 595 which is a hundred and seventy three dollars savings our year for 995 Which is a five hundred forty one dollars savings now They all come with a 30-day money-back guarantee So you can go over there you can check it out whatever one you like and then you're also gonna get a great workshop On next Wednesday, April 3rd, which is how to comprehend Basil Chapman. What's going on man? When you said 2020 I thought wow Serious and Tommy said that this morning when he was talking about your workshop. I says oh my god, man I just remember the year 2000. I'm looking at him and he was like all of 20 years old then it was like wow Time flies So this is gonna be very interesting if you're looking at the chart on the left This is the dating chart. It seems so far. We've been making a lower highs We've made one lower lower went from the 26,000 to 41 high of February the 25th Went straight down to the 25,000 to 52 level in the Dow Ran up and there's a big percentage. It did go from 25,200 to the 26,000 100s very quickly But then it gave it back and you can see there's this great trend line I love to use very simple tools whenever I can and there's a very simple tool I just took it from the low that was made back on the on 8th of February of this year. It's a 24883 There's a rising trend line to the low that was made on the level of 25252 and 25,501 was the low that was hit yesterday So that's the trend line source to the 50 day moving average However, what I am concerned about in the day you not yet the weekly as I say I've been talking about this for some time the weekly Dow S&P the QQQ that's the index 100 Have all the weekly charts are fabulous The IWM is starting to suffer some from the selling pressure but the others are pretty good and That means it's gonna be very important one of the reasons why I want you to have a webinar straight after the March Monthly chart is concluded is that I think it's gonna tell us a tremendous amount So I'm looking at the daily. I'm saying well the daily looks to me like we're making arch formations And if there is a if there is a doesn't have to be a close in the next few days if the S&P goes into the candle close of yesterday and that was 25,000 516 so in other words at any point if it goes 70 or 80 points lower I think we've set the pace now that we're going to test that trend line And then we probably going to break the trend line And then we're going to have this cup formation the cup formation in the chapter main methodology always says Watch out for that left side low. That's going to be key. That'll be 25,000 252 So I'm going step by step. Why look with all this effort The MACD the moving average convergence average and sassons technical tool right here in the middle between the price and This moving average here. It's very weak and the stochastics now only at 47 percent I love when it's over 80 percent especially 90 percent You're in the 47 percent area that is really not good the relative strength is starting to weaken So for subscribers, we started to take take off some of our positions along positions Some of them have raised the stock and we're building up a cash position We do have one position where we've got Option calls on a particular stock that got really hammered I think it has a chance to have a pretty good balance and that's all we're going to play there But in the meantime, I am concerned I think that this is a digestive phase now at this particular point because the SMH is a semiconductors And this is this is going to be a very big test of fundamentals as applied to technicals if you look at the SMH is at a hundred and six point ten The price has gone substantially higher. Well, first of all 114 area was March of last year. That's exactly a year ago makes it's all-time high plummets down to the 80 level and it's face of 30 points. That's a 27% 28% Pullback, but then it has this rally which is very strong and that was helped by the rotation through my micron and You had various what was the other one? It's Massachusetts company Was it XL and X? Yeah, Xilinx had a fabulous rally and each one of these health estimators go higher But I'm hearing from all sources that and I showed a chart the other day in fact I I think I discussed it with you when we were last on and I said to you There's this chart that I've been looking at which shows that the shipping of the semiconductors Is way down going into the early part of this year This is not absolutely current but it is up until the beginning of this year Look at that chart the yellow one and the blue line is the global leading Manufacturing index growth and that's down quite sharply and here you are you could invert this and that's what you've got for the semiconductor ETF the SMH something doesn't It just doesn't work out that way because you have to regress to the mean either the Semiconductor is gonna have fantastic orders over the coming six weeks to two three months and you just haven't seen it yet But it's there and the price is telling us that it's there But I'm hearing that they they shutting down some of the fab fabrication Centres the they are laying off people that doesn't make sense I think that the semiconductors are somewhat vulnerable and I'm gonna be watching that to me That's a bit of a benchmark so at the workshop on the week from tomorrow Wednesday at 5 p.m. April the 3rd I'm gonna be talking about the Divergences how for my subscribers. We are trying to handle this what we're doing why we We are starting short positions what I'm looking at how severe it could be We'll know a lot more when the candle closes on the monthly charts coming up this week at the end of this week on Friday That's the end of the month and I think it's gonna be a really interesting Webinar because we've already started to look at positions that we want to take on any big pullback We're looking we have some positions that we are holding that have done very nicely And we've lightened up we're taking off anything that looks somewhat vulnerable We are I just don't need the risk right now. I just want us to get out have some cash Cuz I think there's gonna be some really good buying opportunities So a very timely workshop for us because we are waiting for a pullback We're waiting for certain sectors. You know you spoke about the XLF the XLF the financials. Yes I Until we can really go Maybe I'm wrong, but I really think for the next phase of The bull phase. I think we need the financials. It's not interest rate anything. We need the financials because You I want the backing the structure of the financial sector to be with us on the upside So that's not ready yet And folks the way you get into this workshop just come over to our website at TFNN You're gonna see right under featured content Man mr. Bows what's happening the opening call you hit that subscribe button you are off the races Bows you have a great one safe one look for the program tomorrow. Thank you very much on YouTube. Thank you Stay right there folks come right


All right. Let's get started. First, thank you very much for coming. I'm also very happy to be here to attend this summit, since Vancouver is the place where I started three years for my graduate degree. So that is another excuse for me to attend this summit. Today, I'm going to talk about the challenges for open-stack-based hybrid cloud deployment in China and the solutions that we come up to meet the challenges for deploying open-stack-based hybrid cloud to enterprises and cloud service providers in China. First, I'll quickly go over what is the reality of the cloud marketing in China and what is the momentum, the growth trained for open-stack-based cloud, private cloud, hybrid cloud, or public cloud in China, and give you a brief introduction of who we are, what we do, and what is our positioning, and our business model, et cetera. And then now, go over the details of the challenges, the problems that we have to solve when trying to deploy to help enterprises to go to a cloud, open-stack-based cloud. And then I'll go over the details of what we did, what kind of solutions we provided and functionalities we add, and the pinpoints that we resolved for the customers. And lastly, I'll go over a deployment case and tell you the background of the project, the customer, and our solutions. And if you have time, I'll do a quick live demo of our public cloud service site here. All right. So this is a rough estimate of the cloud market share size estimation for China. You can see US, of course, the biggest market in the world, and more than 50% of the market share. And China, surprisingly, the market share is in the lower single digit, believe it or not. As the world's second-largest economy, its market share, the cloud, is actually really its infant state compared with the United States over the rest of the world, for that matter. And I have two slides of some interesting data, researched down by IDC this year, shows you the cloud market, the trend, the cloud management software share estimation, et cetera. And this slide shows you some interesting data and their estimation for the cloud management software in China. So the inner circle shows you the current market share of VMware-based solution, Microsoft Azure Service Pack, commercial OpenStack, and a community OpenStack release. So the dark blue is VMware. And the light blue is Microsoft. And the gray and orange is the OpenStack. As you can see, so for the private cloud sector, VMware is dominant. It's more than three quarters of the market. And Microsoft is gaining a lot of momentum and traction. And it's about 13% while OpenStack here, the commercial version and the community OpenStack is fairly small, so 6%. And IDC, based on their estimate, based on their survey and the estimate that 3% is from now, OpenStack will be almost 20% here.  And it's market share will be from three quarters to about 60%. And also the research looked at the cloud service market area, the IDC operators who wanted to adopt a cloud transformation and started from, transformed from providing the rack and space to cloud service. And the telco service providers. And there are some enterprises who also would like to provide community services, the some vertical clouds, community clouds, those kind of things. So you can see here currently, VMware is about 30% of the market. And three years later, it will shrink to a quarter. While the OpenStack market, you can see first for this market, the gray is self-built. Because the cloud service providers, they have a lot of resources. They have the technology and the expertise so they are really built. And the rest of them, they either use VMware or Microsoft or OpenStack. Again, in this sector, we see a huge increase three years later, based on IDC report, estimation, from 6% to 26%. And there's another way of slice this data, excluding the first year CSPs, like Baidu, Alibaba, and the AWS, et cetera. You can see the OpenStack market share will grow more, excluding the first year CSPs. Now, there is another interesting piece of data is the user survey. The pinning of which cloud management software do you prefer? So when asked this question, 20% of the respondents says it's a commercial OpenStack. And almost 30% said a committed OpenStack. Folks believe, oh, we have in-house IT resources and we can do it ourselves. And 30% said we cloud, VMware still has some big mind share. And about 20% of them select Azure, Microsoft Azure. So the observation or the conclusion is that you can see 50%. 50% of the folks being asked this question choose OpenStack. And OpenStack actually has become the most important cloud technology in the Chinese cloud market. And who are the big players, the super users of the OpenStack technology in China? Similar to what happened worldwide, there are financial institutions, financial service institutions. There are auto manufacturers. Of course, the e-commerce companies Suning and Jingdong and a lot of internet companies like Siner, Citrips 360, NetEase, et cetera, and other education institutions. Based on our feeling of the pulse of the market, of the enterprise market in China, we see a lot of CIOs and CTOs, they already selected their cloud platform technology. That's OpenStack. However, there are three major pain points for the enterprises, for some IDC service providers who want to transform the cloud. The three major pain points are a lack of cost-effective, stable, reliable cloud platform. Now with the event of OpenStack, this pain point is not that big. And the second and the third pain points are the lack of DevOps resources. Cloud is different from a lot of people who say, hey, OpenStack will be the future Linux for cloud market, for cloud service platform. But I don't think that is very accurate, because after all, Linux is a piece of software. While OpenStack is the integration engine, its end product is a service delivered to its users. So you'd have to have people maintaining, fixing bugs, maintaining the clouds, and delivering services to end users on a 24 by 7 by 365 basis. So most enterprises and CSPs, IDC, especially IDC operators, they don't have the DevOps resources. And we are funded to solve these pain points. So the team of Ketum Cloud come from the first public cloud builder and operator in China, the Grand Cloud. Our team has five years of experiences building cloud from scratch. And also we have four years public cloud operation experiences. So last year, we secured VC funding from two renowned VC firms in China, a China broadband capital and GoB partners. And also we received strategic investment from Cisco Systems. And our positioning on the business model is that Ketum Cloud is a hosted hybrid cloud service provider. So this summarizes all what we do in terms of our product and services and in terms of our business model. So our product is OpenStack-based hybrid cloud. Hybrid cloud, I mean interconnected public and private cloud. To support this product, to support public cloud, you have to have a billions. You have to have a reportings, online payment, charges, those kind of bells and whistles that are needed for public cloud, for enterprise or for CSP to provide public cloud services to their end users. And our service model is managed cloud services. So that is our positioning and the business model. So now, what are the challenges? What is hybrid cloud? What are the challenges of OpenStack-based hybrid cloud deployment? Why hybrid cloud? Of course. So the most common definition of hybrid cloud is interconnected public and private cloud. However, there are some other types of hybrid cloud definitions in China, quote unquote. The mixture of OpenStack resource pool, resource cloud with VMware, they call it hybrid cloud. The mix of OpenStack cloud with Power VM or Power VC, they also call it hybrid cloud. In a sense, they are hybrid cloud, different cloud technologies. And also, the bare metal plus OpenStack cloud. And oftentimes, when we work with financial institutions, we found that these customers, these enterprises, they have all of them. They have bare metals. They have a requirement for us to provide them with technology that can manage bare metal pools. They can manage existing VMware investments. And also, they want us to be able to manage their Power VM through Power VC. And they don't want to put everything, all their acts, into the public cloud, especially in China and Asian Pacific countries. Because these guys are worried about their sensitive data. There are critical business applications to be run on a public cloud managed by e-carnals provided like Alibaba. Especially, I met a lot of CIOs and CTOs of the enterprises in China. They say, hey, I will never trust the Ali's Cloud in China. That's the biggest public cloud service in China, not like AWS here. In the US, AWS has had a lot of certifications and gaining a lot of momentum in the federal governments, but not in China. All right. So what hybrid cloud deployment does enterprises really need? So number one, they need a very powerful, for public cloud, they need a very powerful user-friendly tenant self-service portal. So the current horizon provide a good foundation for that, but that is not enough. And oftentimes, some folks, they just use our engines. They build their own portal enterprises and CSPs. The second one is for providing public cloud. You need to have buildings reporting, as I said. And also, you need very competitive ICE or PAS functions that you can compete with AWS, with other public clouds like Ali Cloud or Tencent Clouds. That is, you need a powerful, a cloud admin, a boss system, a business subsystem, or operation subsystem. And the other challenges is what I have just described. They need to manage multiple clouds. And they want us to provide an open stack that can manage the VMware pools, the Power VMware pools, and also bare metals. These are the advanced functions that OpenStack Community Release has not implemented yet. The auto-skilling of VM numbers in an elastic load balancer. The live scale up, increasing the CPU and memory of a virtual machine without rebooting it.  The ruleback of the VMs OS to any prior snapshot. To ruleback, I mean ruleback, of course, OpenStack provide the snapshot functionality for VM OS for volumes. But developers, what they want is not just a snapshot. They want the capability to rule back their OS, their volume, to any prior snapshots that they have taken for their projects. On and on and on. I'll go over some of this. I have quite a few slides describing, actually, these are the functions that we enhance, we implemented on top of OpenStack. Based on the requirement, when we deploy OpenStack-based hybrid cloud to Chinese enterprises and the cloud service providers, all right. So this shows you the OpenStack as an integration engine for multiple cloud management. So now let's talk about what we did. This is core. Everybody is very familiar. There are three tiers, the UI tier, which is horizon and some command line interface command line tools. And the middle layer is the Alexis services, the novice, Cinder, Swift, and Neutron, et cetera. And the third tier is the share services, the images, the ID management, logging, orchestration, et cetera. As I said, number one, we need a very user-friendly, the cloud platform that gives you the taste of Amazon or other major public clouds. The functionality-wise user experience, the whole nine yards. In addition, we need an admin portal, an engine that allows the cloud admin from a CSP, cloud service provider, or from enterprise, to be able to manage that. That includes, of course, configuring the resources in the OpenStack, which part of the OpenStack admin functionality has already horizon. But more, they need to manage their asset. They need to know the capacity utilization, the monitoring, the billing, the pricing modules, and all the bells and whistles that you need, especially for public cloud services. And they need, as I said, advanced ICE features that allows them to compete with private solutions from WeWare, with public cloud services, functionalities from AWS all year or the rest of them. And lastly, they need billing, service package management, online charge and payment, account order management, et cetera, et cetera, to actually allow them to run their cloud services to their end users. And that is what we call Ketum Cloud. So what we do, we did a lot of addition. And some of the folks, some of our competitors, what they do, they do subtraction. They don't even support the functionalities OpenStack already provided. And we support all the functionalities that OpenStack provided to our end user, and we add more to meet the challenges and the requirements that they have. So this is another diagram shows you what we did. The red ones are the OpenStack modules, the keystones, the new projects, and all the projects that provide the great functionalities. As I said, we need a tenant portal. We need an admin portal, the BSS and the OSS. And we need all the rest of the shared services that allows you to actually manage it, provide the services, be able to give the user the visibility of what is happening in their data center on the server. And you can see there are other factors that we need to take into consideration to deploy Hybrid Cloud, OpenStack Hybrid Cloud, to enterprises in China. I guess I believe here in the States it will be the same thing. Because oftentimes enterprises, they were already investing in clouds. Most of them, probably they already have VMware implementations. And that is especially true for the financial FSIs, financial service institutions. And all of the banks, insurance companies, and security companies we talked with, they have VMware, they have PowerREM, and they have bare metals that they want you to manage. Because oftentimes some of the applications cannot run on virtual machines due to various reasons. While these virtual machines, while these bare metals has to be on the same network or virtual or physical as the other tiers of the application, the app tiers of app tiers. So that presents the challenges for you, for us, to manage the bare metals and also put them on the same virtual subnet as the rest of the virtual machines. So we have to be able to manage the PowerREM pools, the VMware pools, the bare metal pools, and utilize all the existing fiber channel equipments from various vendors. If it's a new deployment, they just buy the commodity hardware, the servers. They don't have any of these troubles. All right. So this is a quick, if I have time, when I have time, I'll show a live demo of this. This is our horizon. You can see the overview of your resources, the compute, the databases, networks. And this is the topology that we enhanced on top of. Basically, we built this tenant portal from scratch using the OpenStack APIs. And this shows you the virtual routers functionalities. Does you have a VPN gateway? And what kind of network is connected, et cetera. Now, there's some details of what we really did. So this shows you the functions that these OpenStack project modules have, you know, Vanuva, Glass, Cinder, Swift. And the comparisons of the releases from the community and the solutions from Red Hat, Morantis, IBM, and HB. And I also compared it with the public, the leading public cloud service providers in China, Qin Cloud and Ali Yun. You can clearly see what we did. First, ruleback of VM, as I said, to any prior snapshot. Auto-skilling the number of VMs in your elastic load balancer, live scale up of VM, that is actually a requirement from a financial service institution. They said, hey, I don't want to stop my application. I want to add CPUs and memories on the fly without shutting down the VM, their application. Resource recycle being, resetting password, et cetera, et cetera. These are all the features that we added on top of OpenStack. For Glass, we added bring your own image. That is, you can create an image from your application in a physical server or another cloud. And you can just upload it to our OpenStack cloud. And that cloud will become your own cloud, your kind of a system cloud. Then you can use it to create virtual machines. And that is really facilitated and speed up your cloud migration from either VMware or Cloud Stack or other public cloud to your own private cloud. In the neutron part, what we did, we added a lot of interconnected abilities. OpenStack support IP stack VPN side to side out of box. It has all the other good stuff. Galaxy IPs for ELB, for virtual router, firewalls for V routers. What we added, we added GRE tunneling, L3 GRE tunneling. We added a host client to site VPN services over PPTP and open VPN. And also we added managing bare metal. The key here is on the same virtual network as the virtual machines. That's what we did. All right. To summarize, we did a lot of work enhance the VM functions, the auto scalings, the live scale ups, et cetera. And this is auto scaling. With heat, you can actually scale the number of VMs in the ELB group like that in a single cloud. So you can come up with a policy to increase the number of VMs based on the CPU, the memories, or any variables, any parameters that you can monitor. And you can decrease this also by setting up some of the triggers, OK? And this shows you the monitoring of one ELB. It shows you the seesaw of the CPUs of this ELB group when you increase or decrease the number of VMs. And this shows you this diagram of the live scale up. You can on the fly, without shutting down the VM, or your applications, and you can increase your CPUs and memories. Ruleback of virtual machines OS snapshot. Bring your own image, BYOI, which is critical. What we did here, actually, we added two APIs, two glass, import and export. So what we did here, actually, is we automated what the Cloud Admin did with their image store. And we implemented the user portal that allows the tenant to do what a Cloud Admin can do. So they can actually create an image, an import. Basically, they can add any kind of images, system images, to our OpenStack Cloud. This shows you a case where the user imported DSL, damn small NX, is only 50 megabyte. And this shows you the virtual machines that are created using their imported DSL image. It's quite different from the flavors of the red hat or Ubuntu of the virtual machines, you can see. Now, the advanced testing functionality that we added, like all three GRE tunneling, site to host, and VPN services over OpenVPN and PPTP, the cloud bursting from one cloud to another. So this is the IP stack VPN that's supported by OpenStack Outbox, and this is what we added, L3 GRE tunneling, allows you to connect one cloud to another if that cloud support GRE VPN gateways. Cloud bursting, a lot of people talk about cloud bursting. So your private cloud has limited resources, have limited elasticity, and during peak time or promotion, you want to burst your applications to the public cloud, where you have a lot of capacity, resources, elasticity, et cetera, et cetera. We actually implemented this with AWS. So we got an AWS account, we created a VPC with a subnet, and we connected these two clouds through IPsec, IPsec VPN tunneling between these two cloud, and we set up our cloud auto scaling, the ELB. Now you have extended ELB over the two clouds, and you can scale out your VMs to the new ELB group in another cloud, in AWS. And when the load comes down, you scale it back.  Scale back your application to your own cloud. That is, we call it cloud bursting, cross-cloud auto scaling. And this figure shows you the VPN from host to site VPN services over PPTP and OpenVPN. To summarize, we developed based on OpenStack. We developed, enhanced the platform so we can be deployed as a public cloud services for the CSPs, cloud service providers, IDC, SPs. And you can also deploy it as a private cloud. And we can connect it with the site-to-site secure VPN services. And we also can connect to their customer office. Our third enhancement is we prioritize our boss system, our engine. So you have account management capabilities, IDC asset management, deployment, monitoring, online invoice, integrated support ticket, the bells and vessels that a service provider, a public service provider need to provide public services. And this shows you the boss system, the key engine, the pricing. OK, I think I'm kind of short on time, so I'm going to speed up a little bit. The third improvement what we did is we must be able to support the complex deployment environment in enterprises. As I said, they want you to connect their private cloud with public cloud.  And we can do that, because when we are recenter, open it up their API, recenter 5.5, open up all the API, restore API from OpenStack. Actually, you can add a resource pool. So when the user say, I want to go to the VM or pool, you can direct the user to that VM or pool and create VMs in that VM or pool. You don't have to do with anything. You don't have to remove the recenter, the resource management software for VM or pool. You just leave it there, and you can manage it. And that's the requirement. The same thing for Power VM. And we did that. I'll give you a couple of snapshots that shows you so here. So when the user select the image, created their own provision, their virtual machine, you give them a choice of the local resources using KVM or the VM Ware. So they show you the images that are available on the VM Ware pool. We can do the same thing for Power VM. Most importantly, for bare metal, Power VM and the VMs in the VM Ware resources, we are able to put them on the same network, to allow them to join on the same subnet, the virtual network. I think I'll give a quick demo of five minutes. So all right. This shows our key cloud. You can see you have the compute storage. It's a little different from what you see in the horizon. It shows you the network. So this is the virtual machine, the topology. It shows you the virtual router, which is where it is on. This one has a virtual VPN gateways. It has a floating IP, a public IP. This is a SUSE instance. Shows you the image, the network it is on, et cetera. So overview shows all the resources, the recent activities. Compute shows the instance images, recycle bins. Recycle bin is the place where we store the resources you delete for two hours. And after that, it's permanently deleted. NOVA has this API called soft delete. And it also has a force delete. Basically, if you delete by mistake, it will be here. The other place that you can see all your resources is here. It's a network topology. It shows you what virtual routers you have and what you have on these water routers. Do they have a gateway? Do they have a floating IP?  And also the virtual machines on these subnets. All right. I'll do a quick showing of the, this is ELB. And this ELB has two members. And I set up auto scaling on this ELB. It doesn't need to add one VM when the CPU is greater than 90 and delete one when the CPU is below 75. And what you see here is you have this seesaw CPU pattern. And that shows you the system auto skills, auto add, and shrink the resource pools based on the load. A quick look at our engine, the cloud enemy engine, an enterprise, or a CSP. They need this. They need to know what physical servers they are on, the CPU utilization on this server, and how many VMs on this server. They need to know the asset, the servers, the switches, the IDCs, what resources you have. Their IP addresses, all the information you need. And they need billings. So resource settings are what admin of the horizon already have. We added asset management, billing, support, and statistics. So this is a pricing engine. It shows, first, you can design your product. What product you want to charge. The images, the bandwidth, the volumes. And then you can decide what kind of price you want to charge. You can change it here. And also, you have special prices for your promotion. And this will all write the prices you set in the normal engine. And last, but not least, it gives you the capability to see what is going on, the visibility of the resource utilization in your data center. So this shows you the number of paid VMs, the CPUs on these VMs, your daily charges, the new users, the daily active users, and the flavors of your virtual machines. And user statistics, since we were the first public cloud service provider, so we know a CSP for a public cloud. You will need this information.  That's it. Thank you very much.


Garrett Temple has been anything but silent when it comes to the ongoing issues with racial injustice in America. He understands that there are bigger issues at hand than winning basketball games, and his leadership has been invaluable both on and off the court. Crescendo has more. Garrett Temple wants to be heard. The National Basketball Players Association member and hopeful lawyer knows the importance of using your voice. With recent events surrounding the Black Lives Matter movement, he feels that some people are finally listening. I think this situation is a little different. Unfortunately, not in the fact that another Black man has been killed, you know, subsequent to another Black man in Atlanta and then a Black woman. But the fact that how people have reacted to it, specifically how White America has reacted to it, it seems like people are finally starting to care about, you know, unarmed Black men being brutalized by the police and just Black Americans in general being marginalized. So the biggest thing I see is the difference in how it has affected or how everybody else is trying to help and realizing what the devastation has gone on. In my opinion, because of how I was raised and how I do things, and more so now, you know, I think the NBA, the average NBA player is more socially conscious now than 15, 20 years ago. Definitely more so than when I first got in the league. So I can't really speak on 15, 20 years. I wasn't in there, but more than 10 years ago, the Black NBA player is more socially conscious. And because of that, I would hope that you're not only conscious, but you're active in using your platform and your resources to help things that you feel like should be changed. Whether it is putting money into certain situations, having a foundation that helps fund certain issues that you want to see get better. And I think the resources that we have, because of the score we're able to play for a lot of money, we can utilize those resources to help the communities that we grew up in, to live in, care about. The ability to speak up, the ability to have an opinion, the ability to raise conversation with the group is huge, whether that's on the floor or off the floor, and Garrett fits in line with that. We talked about being able to communicate with your teammate in positive and negative situations and grow from it. And so he is the prime example of how we want to carry ourselves on a daily basis. At the end of the day, everybody has their own thoughts on how to effect change. I think the main underlying point is that everybody wants the same thing. Khairi, myself, most of the black men in the league that are passionate about this, or if they weren't passionate, most of them are passionate about it now. We want the same thing. And there are a lot of different ways to skin a cat. The conversations were actually those conversations, thinking about ways, what way, what is a way that we can most utilize this extra push, these extra ears and extra eyes that are on this situation.