Good morning, everyone. Good morning. Almost good afternoon. Just want to thank everyone for coming to this session on OpenStack Trove and Message Bus Security. My name is Mary M. John, and I work for IBM. And I'm a core on the Trove project. I'll be co-presenting with Amrith Khosa, PTL. And he's from Verizon. So today in this presentation, we will be talking about focusing mainly on how we've enhanced security for our Message Bus, just one of the features we worked on this release. I'll start with a general introduction of Trove, a brief introduction of the Message Bus architecture within Trove. And then Amrith will talk more in detail about how we've enhanced the security. So Trove is the database as a service component of OpenStack. Its main goal is to provide a scalable and reliable platform to provision and manage databases, SQL and no SQL databases, in a uniform and consistent way. So the way I see it, one of the main advantages of Trove is it provides a consistent platform to provision databases, whether it's SQL or no SQL. And what that means is you have a single interface, whether it's through Horizon or the OpenStack client, to, for example, create databases, provision clusters and manage replicas with just a single command. It can create a MySQL database or a Mongo database and manage them so that it frees the user from getting to know the details about each individual database. In addition to that, one of the key features are user and database management, resizing instances and volumes to help with scaling, cluster management, replication and failover, backups and recovery, managing configurations, and providing the ability to do database instance upgrades, whether it's having to upgrade the operating system version or the database server version or applying patches and things like that on the database instance itself. Now, in this chart, we look how Trove is placed with respect to all the other OpenStack services. So as you can see, Trove uses many of the core OpenStack services like KeyStore for authentication, Cinder for block storage, Swift for backups, Glance for storing the database guest images, Nova for provisioning neutron for networking. And while all the other core OpenStack services can use a shared message bus, Trove has its own separate message bus that it shares between the different Trove services. It does not share that with the rest of the OpenStack services. In this slide, we talk a little bit in detail about the Trove architecture. So we have Trove consists of mainly four components, the Trove API, the Trove Task Manager, the controller, and the guest agent. So the three services, API, Task Manager, and Conductor, they run on the control plane, whereas the guest agent, along with the database server, runs on the computer node, which Nova provisions. Now, as you can see on the left side, all these different Trove services, they share the same message bus and a database over there. And this is separate from what the rest of the OpenStack services use. So when the user makes a REST API call, say it's to provision or launch an instance, it goes through the Trove API. And then from there, it makes RPC. It communicates with all these different Trove services. They communicate with each other using RPC. Over this message bus in Trove, we use RabbitMQ to implement the messaging service. While also messaging RPC supports many different messaging services, we use RabbitMQ. So to talk a little more about the message bus communication, like I mentioned earlier, the different Trove services, the API, the Conductor, Task Manager, and the guest agent, we talk between each other over the message bus, which is implemented using RabbitMQ. So for example, now since this is an RPC-based communication between the services, what it means is that there are RPC service and clients. So the Task Manager, the guest agent, and the Conductor are configured as RPC service. And what that means is that they expose a list of endpoints or methods to which clients can call over the message bus. So for example, we launch an instance, the API sends that message to the Task Manager. The Task Manager then calls Nova to provision that VM. And once it's provisioned, it sends a message to the guest agent to prepare that instance. So what it does at that point is that it does all the database configuring, whether it has to install and set up the database. And that message goes to the message queue that's configured for the guest agent. So each RPC server has a queue defined. So Conductor has a queue, guest agent has a queue, and the Task Manager has a queue. So all these different RPC servers communicate with each other, post messages to these different queues to get things done. Similarly, the guest agent, once it's up in Rani, it sends its status to the Conductor. And that's how the communication between the Conductor and the guest agent happens. To let it know that it's pretty much called hard beat messages, you just let it know the status of those instances. And this happens every couple of seconds so that we know that whether the guest instances are alive or not. Now all these RPC clients, they use the same RabbitMQ credentials. And I'll talk in a couple of slides where these credentials are stored. So how do these RPC clients communicate over the message bus? So like I mentioned earlier, we use RabbitMQ as a standard for our messaging service. So we use the RabbitMQ username password and the message queue names, which are all stored in configuration files. So each Trove service has its own configuration files. And these credentials are stored in those files. So how do the credentials get to the guest agent? Now the guest agents, they sit on the guest instance, which are separate from the control plane. So once we create these guest instances, how do we get the configuration files over to the guest instance? It's pretty much what we are doing is when the client launches, the task manager asks the guest manager to launch an instance. What happens is it tells task manager tells Nova to create a VM. And once that happens, it prepares two configuration files. One is the guest info, which contains the guest specific information, like the guest ID, which identifies the guest instance. And then the other file is called the Trove guest agent configuration file. And that one contains all the information about the RabbitMQ credentials, like the username, password, the server name, and then the queue name. And this one gets injected into the instance using config drive during the bootstrapping process. And so that's how these configurations get into the guest agent. And Amrith will now talk about the security part of it. Thanks, Mariam. You guys able to? No? OK, now you can. OK. Thanks, Mariam. Thanks for the introduction. So I know there are several people here in the audience who have deployed Trove at reasonable scale and who probably know about some of these problems. So just to make it interesting, throw questions up at any time. We don't have to wait till the end or something like that. As Mariam said, the fundamental issue, which has long been raised as a security concern, I'm going to go back a couple of slides to the architecture here, is when Trove provisions a guest instance for you, it does that by talking to Nova. And whatever Nova provisions as a guest, that could be a bare metal, that could be a VM, whatever it happens to be, or if you're fancy and you're using LXC containers, it's a container, you have the guest agent running there, and the guest agent is connecting back to the message bus. Now, whatever credentials you need for the message bus, let's assume it's rabbit, those credentials are stored on the guest. Now, if for some reason, that guest happens to be compromised, and there's any number of ways in which it can be compromised, if you build a guest image where you happen to leave port 22 open, and you have a security group which is not tight enough, or somebody is really clever and they manage to break out of the database and land at a shell, those credentials are stored in plain text in a file. Once you have those credentials, you have the ability to connect back to RabbitMQ or whatever your AMQP server is. And at that point, you're talking to any other service which is listening on that bus. So one of the things which typically Trove has always advised is you use a different message bus for Trove. Part of the reason for that is Trove is often deployed not as a project which is part of the control plane, rather it's deployed in a Nova VM by itself. And if you want to install Trove in a Nova VM, the last thing you want is a VM running in tenant space to be talking back to your control plan to your message bus. So great, you have an independent message bus and all that.  And now you have the credentials, and you can do a fair number of bad things. So the guest instance is compromised. If you have the guest credentials, now the message bus is compromised. That is the fundamental issue which you're trying to solve. So pretty pictures, now this is the words to go with that. The important thing to remember is that when you're evaluating the security of a system, you start from the point which says if a system can be compromised, it has been compromised, what happens next? So since you have credentials in plain text, and it's possible in some cases to compromise a guest, now the guest is compromised, what happens? Anybody who is clever enough to connect to your message bus can now send any message they want. So it's really important if you're deploying Trove to protect your guest image. And there are any number of things which you can do in security like the proverbial onion is multiple layers. So prevent port 22 access, no shell access. Have things like app armor or SE Linux or whatever you want. Make sure your database is configured such you can't have a escape. All of those are things which you can do, but still, if that's not good enough, then what? And it's important to go that far because there are going to be situations where even though you do all of these things for troubleshooting, you need to get onto the instance. So you might have port 22 access even if it's from a control network. So at the end of the day, how do you further narrow the damage radius from somebody compromising your guest? And so I'm going to talk about something which we actually implemented in Okada just to do this. And the solution we came up with was relatively straightforward. We're just going to encrypt all the traffic which runs on the message bus, okay? And we're going to do that encryption with unique keys depending on what the traffic is being used for. So there's a unique key for every guest. There's a unique key for the control plane. And the key for every guest is stored in the database on the control plane. So is everybody kind of clear with this or do you want to see a picture which explains this a little bit better? Yes, no. I'm going to go back to the picture just to make it easy for me at least. There is a database down here which is the database on the control plane. That one over there. The key which is used for the API service to talk to the task manager is stored on the control plane. The key for every single one of these guests is stored in this database. And each one has its own unique key. Everybody okay so far? All right. If you were to do that, then every piece of traffic which is on the message bus is encrypted using a specific key. On the control plane, we use a key for the control plane. For every guest instance, we use a key which is unique to that guest. And we'd regenerate that key. We generate it when an instance is created and we regenerate it when the instance is upgraded. Okay. So we got some of the OpenStack security folks to take a look at this. And while I was doing that, I came up with this analogy to compare it with a security mechanism which everybody kind of knows and understands. Everybody is familiar with HTTPS, secure sockets layer and so on. And the way in which SSL works, whether it's with your browser or anything like that, is you rely on PKI, public key cryptography, for one part of it. But the bulk of your actual communication is using reversible ciphers. We can stay away from the discussion of why you do that, but the costs involved and so on. But in standard HTTPS, the three steps are first to establish a session and securely transfer a key. Once you transfer the key, then only one and one person alone can get that key. And then you do everything using that reversible cipher. That's the standard steps which you have. So you start by a server identifying itself based on a certificate. The client receives the certificate as part of the connection and says, do I trust who you are? HTTPS colon www.google.com. Are you Google? Well, I use a mechanism of trust and a hierarchy of trust to decide whether you are or you're not. And in your certificate, I have your public key. Since I have your public key, I as a client generate a private key and cipher my session key with that public key and send it back to you. As long as your private key is not compromised, only that server will be able to extract that key. So you've safely transferred a key across. After that, everything is a reversible cipher based on that session key. And on the server side, the session ID is associated with the key which is being used.  And the mechanism you rely on is the fact that the transfer of the key securely is the important thing. So that's eventually what it all comes down to. So we've done exactly the same thing and we've said, I'm gonna use some mechanism to securely transfer a key across. Once I transfer the key across, there's a shared key in place and then I use a reversible cipher. Everybody okay so far or do you want me to go over this one more time? Okay, the mic's probably not working because nobody's saying anything, all right. Okay, good. So every guest has a unique key. All the communications are now protected. So the same situation here is that you have a public transport, the internet. Everybody can snoop on what's going on on the internet. We assume that's the case.  More importantly, if I decide that I want to put my head in the middle of that banking transaction you're trying to do, I don't have the shared key to do any damage. So even though I know www.yourbankname.com, I can't do anything more, okay? So, confusing picture. On the control plane, there is a key. It's established when Trove is deployed. What this basically means is that if you have multiple task managers and multiple API services on multiple hosts, they should all have the same key. Otherwise bad things will happen. When Trove API receives a request of any kind and wants to talk to the task manager, it's going to encrypt that using the control plane key, reversible cipher, sends a message over the message bus. So if the message bus is compromised, can you impersonate the Trove API and tell the task manager to do something? Nope, you don't have that key. Okay, task manager gets a message. You tell it to create an instance. It goes, it talks to Nova, creates an instance and uses the config drive mechanism to securely inject a file. In that file, it has the key. Again, I'm relying on file injection to be the secure mechanism here. So if you compromise cloud in it, yep, bad things could happen. So now there's a key on the guest stored there, but on the control plane, the same key is stored in a database. Whenever the task manager wants to communicate with the guest, it will encrypt the message using that key. Whenever the guest wants to reply to the task manager, it will encrypt using the same key. When it wants to send a heartbeat, it will encrypt it using its key. Guest one always encrypts using that key. When the conductor receives a message and it's supposedly from guest one, because the envelope says it's guest one, it goes and looks in its database and says, give me the key to decrypt this thing. And it uses a key which it trusts from its own database and it decrypts a message. And so long as this is guest one, really the person who generated the message, it gets a valid message, otherwise it gets an invalid message and throws it away. So if you were to compromise this guest, you now have access to this key. It's stored in the exact same place where the other passwords are stored. But if you wanna send a message to the task manager and you want to tell the task manager, hi, I'm the API service, go create me a cluster, which is this ginormous thing, you don't have that key so you can't do it.  You try to send task manager a message to kill this, well, you can't do that because you're talking with guest one key, task manager says, sorry, that message doesn't make sense. So the only thing which you can damage if you compromise this key is this guest itself. But guess what, you already damaged it. So you now again have a public channel which can be compromised and you have encryption based on a securely shared key which guarantees that the damage from compromising one of these guests is localized to that guest. So I know there's at least three people in this room who've thought about this problem a long time. Is Crystal here? No, he's not. So it's all yours, Matt. Does this work for you? So great, great question. So that was the reason why I said there's specifically two places where the keys are regenerated. So think through this path where you say you have a guest which has no key and you have software which doesn't understand this stuff. You have a control plane which has software which doesn't understand encryption either, there's no key. So now you're going through the upgrade process. So the upgrade process is always tight to upgrading a guest.  And it's also gonna come down with one additional flag which says control plane still doesn't understand encryption. So while the key is down there, the guest is not gonna encrypt. Once you upgrade the control plane, oops, wrong button, once you upgrade the control plane and you have this site in place, and by the way setting this up is a control plane only thing. Once you have a control plane which understands encryption, then there's a message across to the guest which says now switch to encrypting. So you will transition the entire system over with software which understands encryption but does not do encryption. And then you move it forward. And the plan was that we would do Okada with encryption and no encryption coexisting. And in Pyke we would turn it to all encrypted. We haven't yet done that. So that is still currently the plan. Works for you? And I did mention that the keys are stored in the database. Only the guest keys are stored in the database. The control plane key is not stored in the database. Yeah, so we did, for those of you who are not familiar with Barbican, it's the secret service which is part of OpenStack. So let me try and answer the question this way. This is currently the list of services which Trove already depends on. When we tell somebody to go and install Trove, having them install these first six services is usually not a hard sell. Telling them that they need to use Horizon also not a hard sell. Telling them to use Mistral for scheduled backups is an uphill battle. Telling them they need to use Designate, yeah, it's a problem. If I want to, as a project, if I want to make my project more ubiquitously deployed, making it depend on another project which has a lower deployment doesn't seem like a winning strategy to me. At some point I would like to go to Barbican, but it's not something which I wanted to make a hard dependency at this stage. Absolutely, absolutely, there's, absolutely. So personally my, I will tell you that this, there's a technical term for this and that is a hack. It's a horrible hack. The place where this stuff should live is in Keystone. But I can't convince Keystone that they should give me the ability to store stuff on a poor project basis in a secure manner. They say that's not their bailiwick. But Service Catalog is, I don't understand why that's the case. Barbican would be the logical place to store it, but this is a starting point. At this point, all I have is a task manager creating an instance or upgrading an instance, storing something into a database. The conductor and the task manager when they receive a message, looking at an in-memory cache, and if there's a miss on the cache going to a database and fetching it. Replacing that with Barbican is not going to be a big deal. So at some point I would like to do that when it's not an impediment to the project. Any other questions about this so far? The communication between Trove API and... So the Trove API never actually talks to the database. Oh, sorry. Which database? This database? This database is on the control plane. So that is secured by whatever MySQL, suppose you're using MySQL, you can turn on MySQL in your client and say communication between, okay, so different picture. It's TLS to MySQL and you're all set. So in that particular picture, this communication here, first it's entirely on the control plane and that you can encrypt using whatever settings you want if you're one of those brave people using Postgres there, knock yourself out, you can do that, or TLS with MySQL, okay. Okay, so how does this actually address the issue? I'll be very clear that no solution which I've been able to come up with or the team has been able to come up with actually prevents you from being able to compromise a guest. So I'm still going to always assume that the guest can be compromised and what happens if it is compromised. All we can do is we can control the blast crater which comes from such a compromise. If you compromise an instance, all you should be able to do is damaging things about that instance. But guess what? If you damaged an instance, you already have damaged it, so you can't do much more. You cannot damage other tenants. That's the important part. And I don't want you to damage other instances by the same user. That's exactly all we can do at this point, okay. And with that, I'm pretty close to the end of all I had to say. If you have specific questions about this, you can ping either Miriam or me on IRC. There's an onboarding session where we can talk more about this. There's also more time where we can go into a demo if you're interested. Or if you have other questions, go ahead. A separate message box in what way? We do recommend that. So if you think about the typical way in which people end up using OpenStack, they start by deploying these six. And then they want to try out Trove. They typically don't put Trove on the same control plane. They typically go get a Nova VM and stick Trove on that. So in that kind of an environment, what is easiest for me to tell somebody is you want to evaluate Trove? Great, spin me up three VMs. One of them is going to control, have a MySQL server. One's going to have RabbitMQ, and one's going to have the three Trove control plane services. And that's entirely running in tenant space. Sorry, I think I may have misunderstood your question. Did you say one message queue or one message bus? Okay, we're using an independent message bus. But on a per instance basis, we're also using discrete queues. It has its own topic, correct? It is doable, we did not do that because with Oslo messaging, you're allowed to have one RabbitMQ endpoint. Or you can have one, whatever your underlying MQP transport is. Oslo messaging doesn't support having multiple. So that's, and we didn't go try to fix that. And yeah, so Murano uses its own RabbitMQ exactly the same we do. It keeps the two of them separate. Okay, I should go and chat with them in that case. The simple solution in that case is to put Trove on your control plane. Sure, sure. And the answer to your question is, even from tenant space, you can talk back to the control plane. It's a matter of whether you're willing to open up that security port or not. Zane. So first of all, congratulations on getting this thing. Thank you. That was really awesome. My question is, is there an issue because RabbitMQ is not inherently multi-tenant?  Yeah, absolutely. We absolutely have that issue. And we consider the option of, how do we avoid that and how do we go to a multi-tenanted messaging system like Zuckar? Well, guess what? I'll do the denial of service on the front-end port for Zuckar. It's the same thing. Okay. So if the issue is a denial of service, Zuckar's not the answer. It's exactly as susceptible. It's not, and to the earlier point about having dependencies on other projects which are less deployed. But in theory with a multi-tenant message bus, you could do rate limiting at the API. So rate limiting is another way of saying that you're going to have a fancy title for what happens when you have a denial of service attack. You can write limit per tenant. But a denial of service attack is still going to overwhelm the service which has to receive that and say, yeah, it's the same tenant, drop it on the floor. Sure, potentially, yeah. Yeah, so since it's, if you're talking about somebody who wants to do a distributed denial of service attack or a denial of service attack, the fact that they know the endpoint to go to rabbit or they know the endpoint to go to Zuckar is not a material difference. But we did consider Zuckar and the reason we didn't, so Zuckar was also presented as the answer because you don't need to store credentials on the guest. Well, you need to store something on the guest for it to be at least able to get back to Keystone and authenticate. So you're not storing the RabbitMQ credentials, you're storing some Keystone credentials because you need to go get a token. You can store signed URL as well. But once you have the signed URL, it's exactly the same thing. It's a secret which you're storing on the guest which is now compromised. Other questions? Yes, sir? Why not use, oh, why not use RabbitMQ virtual hosts? Okay, so we tried a couple of things with RabbitMQ to go down this path. So the first thing which we tried to do was on this message bus here, we tried to have per guest credentials, we tried to do virtual hosts and all of those things. When we got to the point where we had more than a dozen guests and closer to 100 or 150, the performance of RabbitMQ went in the toilet just dealing with multiple virtual hosts and dealing with multiple ACLs. The CPU utilization of Erlang was going through the roof and it pretty much became unusable at that point. The best practices seem to be that RabbitMQ is a protected message bus. We're really using it not as a protected message bus but as something to do RPC. If you wanna do RPC, we should really do this as RPC. That's the fundamental problem. Oslo messaging RPC is an abomination. So the correct solution to this would be that communications to the guest and communication from the guest to the conductor and task manager should all be restful endpoints. If the guest agent exposed a restful endpoint, that's it. The reply from the guest agent should be to a restful endpoint but we're instead using RPC. Other questions? Yes, sir. So we've done a fair amount of testing of what the actual overhead is and that's one of the other reasons why we suggest keeping the RabbitMQ bus away. A guest agent typically sends a heartbeat message every 30 seconds. A reasonably sized rabbit cluster can happily handle a couple of thousand guest instances before it even breaks a sweat. There is certainly an amount of traffic going over the wire but it's not a lot. So the entire heartbeat message literally is, as far as the payload is concerned, it's four bytes. Whatever RabbitMQ wraps around it is some amount of overhead. It's not a lot of traffic. Anything else? Thank you for coming.


Hello and welcome to another edition of the Leaders Room. My name is Michael Kosslin. It's my pleasure today to be speaking with Dr. Paul Chan, the co-founder, vice chair, and president of Help University. Dr. Chan has had a very interesting career both as an academic and economic advisor and as an entrepreneur, as you'll learn during the course of this interview. He's read economics at the University of Malaya, McMaster University, Canada, and completed his Ph.D. at the Australian National University. Dr. Chan has a very distinguished academic career, including being the chairman, division of applied economics at the University of Malaya, and in 2012, he received the Premier Branch Icon Leadership Award for recognition of his contribution to the community. Paul, it's a pleasure to have you here with us today, and thank you for taking time to join us. There's an honor for me to be here. So let's start by talking about Help University. You started Help University in 1986. You literally started it in a shop lot with only two classrooms and 30 students, and I've got to confess to you, I had to look up what a shop lot is. It's literally a corner store in a building. And from there, Help has grown to become a household name here in Malaysia, and it's attracting international attention in the field of academia. What was it like starting something from the ground level up? Actually, it's somewhat of a crazy idea to start an education institution in a shop lot because you need to have a campus to do it, but I have no resources. But my wife and I had a conviction that it is a noble thing to do to provide equal access to the disadvantaged people, especially for tertiary education. So we had and still have a conviction that everyone deserves a sound education. Not be the best in the world, but a sound education that can prepare him to be successful in life. So the conviction became a cause, a religious cause. So in our corporate motto, shall we say, educating is our cause. So I think that provides the drive and the purpose for setting up Help. And incidentally, it means at that time, higher education, learning program. But later, about 15 years later, it became higher education, learning philosophy. So it is our aspiration that in perhaps another five to 10 years, help stands for help, education, learning philosophy. So that's the background of how we started Help. That was your inspiration for the shop. And we started only with about 25,000 ringgit or about 5,000 US dollars because academics don't have money via poor people. So it was very difficult. You want to get some seed capital, who would believe that at that time, you can start a private tertiary education and to have some quality. So that was a real challenge. That was a real challenge. Well, I want to come back to help in a little bit, but I'd like to focus on you and your own career path. And sometimes starting out with humble beginnings can be a real obstacle to one's career path and their advancement into leadership. And when we look back at your life, you were one of nine children in your family. You had to come home from school and help your father in his own business. But yet you persevered. What were some of the lessons you learned simply in those humble beginnings? And how did they influence you? I spent a great part of my life living in urban squatting areas. Actually, the nine of us live in a squatter settlement. There was no water, no electricity, no toilet facilities. And we used to slap on the mud floor. And I can remember that every evening, my sister and my brothers, all of us had to carry water on Kerosene can every evening. But I think that gives us the appreciation of good things in life, because we had nothing. So it made us emotionally resilient. It made us robust. And there's no such thing as hard work. Hard work becomes a virtue. So we are not scared of failures, because there was nothing to fail. So every little success is a great celebration. So when you are so poor and so down below, everything becomes an achievement. So you continue to excel and build up on all the previous platforms. You see, my father was a petition writer in the streets of Kuala Lumpur, in Jalan Mountain, it was called. And when I was young, I accompanied him. And maybe every other month, he had to appoint his Olivetti type writer. So I know how tough life was. And unfortunately, the younger generation are now affluent, because the previous generation, their parents work hard and have wealth and opportunities. So they don't really appreciate the difficulties and challenges of life. That it is something ennobling to create things. I remember that I told my students when I was teaching in the University of Malaya, a faculty of economics, don't come and ask me for a job. You create a job. You see, we must learn to create opportunities, create an employment, not consume, not consume, create. So that becomes a sort of energy inside us, because we need also a certain predisposition. And I think that has to be cultivated by the circumstances of life. It's not something that you pick up in a leadership course, in a top university. It has to be learned in the real hard way. So there's not such thing as a failure, but there are lessons. Not failures, lessons. And it sounds like you've learned a lot of lessons. But not enough, you know. I mean, being a human being, I remember Margaret Dushman's book on the march of folly. You know, there are pitfalls, sometimes you still march into them. There's a problem about human beings. You know, there are landmines all over the place, but you still move into them. Because our DNA is such that we are not a complete person. We are not like a software that we can improve first generation, second, third generation. Every person that is born has to learn to crawl and walk and learn how to speak a language. Those were the most difficult challenges. But after that, the rest should be simple, but actually not simpler. We mess up our life all the time. So to me, every day is actually a learning lesson. So life is a work in progress. And we cannot be a complete, well sculpted piece of Michael Angelou's work. The journey never ends. It's a lifetime journey. Exactly. But I think the important perspective is to enjoy the journey, even if there are failures. So don't be afraid of hard work. No, never. Always be looking up, appreciating what you've got and what you can still obtain. But you also, the very first phrase you used, Paul, was you said, it taught you resilience. Yes. And how often have you had to draw upon that personal resilience? I mean, when you look back over your career, before you started Hope University, you attempted several other things, not once, not twice, but several times, it didn't work out the way you expected. How did resilience play a role? I think the reason why I failed in quite a number of businesses, whether it's construction, rubber gloves or whatever, is because I think I never really believed in them. I had no conviction. I think neither did my wife. Because to succeed in something, you must have the passion for it, that the values of doing something is more than just the bottom line, that as I've said earlier, it should be ennobling in terms of making other people successful. I think part of the reason was maybe I spent too long a time in the academic world. And I did not realize that that should be my core competency. So although I failed in the other businesses, I must say that I did the right thing by starting the other businesses first. Because that taught me a lot of lessons about the things I should not do, rather than the things I ought to do. Can you give us some example? You see, we should not do things that we do not really believe in, that we do not really have the capability. And that in exercising your leadership, you cannot mobilize the hearts and minds of people. Because when you talk to your colleagues, they must genuinely believe that you have this type of leadership to create the direction and the destiny that they want to willingly follow you. Because if not, it is just a transaction. I want this job, you pay me so much. So there will not be a shared vision, values, or a very strong sense of identity and ownership. So yes, they will contribute, but they will not commit.  But when the hand dies for you, it's a commitment. So nobody dies for a company. Perhaps maybe you die for your religious beliefs, but for some multi-level organization, the ownership is stronger, and so on. But otherwise, in most organizations, it is pleasant and nice to talk about vision, mission. But real down, when you lead a group of people into battles, are they willing to die for you? That's most important. So you've told us two things, two critical things, lessons of experience, if you will. One is how important it is to be resilient, to not to give up, even when faced with failure. Another thing that you've learned is passion. It's one thing to have a vision, but you've got to be passionate about that vision. Without passion, you can't engage other people. You can't expect that. Yeah, you must have the energy. You've also been quoted in the media talking about how important things like self-awareness are to you, and the importance of self-awareness and knowing one's limitations. Can you say a little bit about that? Because earlier you were saying, my humble beginnings taught me to always look up and to believe that more was available if I worked hard. But now you're also saying recognize your limitations and be self-aware. Tell us about that. I think the first thing to know is to know yourself, and to know what you actually do not know, so that you can cultivate on your strengths and develop good habits in thinking and behavior. So if we are applying to ourselves that we have a lot of self-limitations, then we develop a sense of arrogance and hoardiness, especially when you are reasonably successful in the beginning or in the short run. Look at all the big major companies, big names in town, the reason being Blackberry and so on. If you do not cultivate a high sense of self-awareness, reflective, even meditative or contemplative, then every day you continue to do what you have been doing. But without a great sense that the world has changed and you have not changed, that you may be strong, but you're actually not adaptive. So sooner or later you will perish. But being self-aware is more than just being successful in business. It is about character formation. You see, every night I would spend a couple of minutes to find out what foolish things I've done in the day, because you may have an outburst or too endosciastic about some ideas. And sometimes you really regret doing certain things and some of the things that we do are not reversible. And when I get up in the morning, I also notice it's like, oh, it's like a miracle. Where were you during the last eight hours? But suddenly you open your eyes, you are alive. So whether you believe in God or nature, you are given another day to live. It's a miracle.  So that's why I say that self-reflection, as many times as possible, is a good habit. So what I do, if I can remind myself, I'll take a minute per hour just to say sing through a little bit before you go through a meeting or you're going to meet somebody like coming here today. Just one minute is enough rather than one hour at the end of the day. Paul, you and I have both been through enough developmental experiences. And we've heard other people say exactly what you just said, how important self-awareness is, reflection. There's a lot of research that says that one of the problems why executives fail is they have these opportunities to learn, they go through them, but they don't slow down enough to really reflect on what the learning was. Did something happen to you that forced you to slow down and say, wait a minute, I need to really think about this experience? I mean, where did you learn the importance of self-awareness and self-reflection? That something like that doesn't come from the book. As I was sitting here listening to you, you spoke with conviction about that importance of it. How did you learn that? I think some of us have cultivated this from very young or was a very independent person. So every little decision that I had to make, I had to make myself because I had no mentors. My family was poor and there to make do with life. So you don't have the luxury of someone teaching you about great universities, the importance of education. You really have to muddle through. But I think as you do that, you cultivate a certain frame of mind and certain good habits that you need to think through a little bit before you do something. But of course, when we started, it was not a perfect model, but you built this up as a frame of reference. So after a certain age in our life, we should have this frame of perspective and having this cultivated reflective mind. So over the years, I have developed my own frame of reference, first to begin with, I have a contemplative mindset and of course an analytic mindset. But above all, a consequential mindset, meaning decisions that are trivial, forget them. Only consequential decision that impact your life, other people's life to begin with, then bottom line and so on. Otherwise the rest is not worthy of our talents and the little time that we have in this world. So I think each of us has our own way to remind ourselves. Some are blessed with great spouses. My wife is a constant source of reminder, so we say, and she will remind me. I have some great colleagues, although they are not holding high positions, they will also remind me. I think we must have this humility to listen, then only can we develop this self-awareness. Because on our own, it is very difficult to teach your mind to improve the mind. Because the mind is reconditioned. So how do we teach a conditioned mind to uncondition itself? That itself is a challenge. So that's why a lot of us as CEOs find it very tough to uncondition ourselves. Yes.  And let me just kind of quote from what you were telling me. You said that you believe in the three C's, character, competencies and leadership and competencies and profession. Can you tell me a little bit more about them and especially how you see those three C's relating to leadership? I think the first C is the easiest to acquire. Competency in a certain profession like accounting, engineering and so on. It's just acquired knowledge and passing some exams and so on. And then you are considered competent to do something. Then the next C is about leadership and that's a very difficult part. To manage yourself as we grow, including the family. But the more important part is just managing resources, making certain decisions. But how to be capable in leading towards the future? Why should people believe in you? So it's not just being a great engineer, you must have the leadership at some point to lead. So competency in leadership must be acquired. You therefore can be a great accountant, but you can be a complete failure in people-to-people relationship. That's why this second C is important. But the point is this, having the first C being competent as a professional and even having leadership competencies is still not complete because we often have moral failures. So this character is most important, the last C. Because it's all about moral choice between right and wrong. And it's just not about right and wrong, sometimes it's also between two rights. So I think the character formation part is the ultimate part. You look at some great leaders, they are fantastic mobilizer of human beings, whether it's a Mao Zedong or Hitler or some chief executives of failed companies. But they may lack the moral character in them. These certain finer qualities of life. As we all say, we must walk our talker. So what we say must be consistent in what we do and what we believe in. So I think the three Cs are therefore a must. So the professional competency, the leadership competency, but the most critical one to you is the character. Of a person. Okay, so let me ask you a slightly provocative, chicken or egg type of question. You talked about competencies of leadership. Do you believe that leaders can be developed or are they born? I think they can be developed, but most of the time you have to acquire in your own way. It's not like having the 10 commandments and you know the 10 commandments from a Harvard course and Presto overnight.  Leadership is an influencing process. Ultimately is why do people willingly want to follow you? It's as simple as that. So yes, anyone and everyone has leadership predisposition. We are born with various qualities of leadership. It is how we develop them, because some are more extrovert, some less, but it does not mean that they are not leaders. I'm not having the leadership attributes. I think the way I look at the world, there are only two types of people, the learners and the non-learners. You cannot be a leader if you refuse to learn, that's about all. And learning is not just being taught or driven by some leadership mantras, but actually having the humility to learn from everyone and all the experiences and exposures. I think that's the way to learn. Look at President Carter, he was not very successful as a president, but in other engagements, it became a great success. Yes. So sometimes the context may not be there and we have to search for it. A few moments ago, Paul, you said that leadership is an influencing process. You and I have probably both met individuals that said, I'm the leader, I'm the one in charge, you do what I say. That certainly doesn't sound like an influencing strategy. It sounds more like a command strategy. Can you expand upon that statement that leadership is an influencing process? I think ultimately our ultimate task, if you want to achieve anything, is to influence people to willingly follow us and to act happily. But at times you need to act top down because the situation and the circumstances may be different. So it's like when to be a conductor, as they say, if you're a conductor for the Beethoven Fifth Symphony, you have to follow. Right. So maybe during crisis time, you must follow me because I know what I'm doing, but if I cannot, somebody else must take over. Other times, since I'm playing jazz, you allow your colleagues to play the drum a bit longer, to saxophone better. You can dance in there with the audience. So I think leadership requires a lot of judgment, when not to act top down and command people in an authoritarian way. But sometimes you need to be quite authoritarian, otherwise things also may not be done because there's a crisis happening. So it's a great judgmental skill and it is very difficult to teach someone and say, this is the time and this is not the time. It sounds like perhaps one of the leadership competencies that somebody needs to be able to develop is to colloquially, I might say, be able to read the situation, to be able to assess the situation. It's very contextual. Very contextual. And knowing whether the situation requires me either to use an influencing style that is more engaging, more enrolling versus one that is more authoritative and commanding. To use your example, it's a crisis situation. I'm not gonna ask your opinion. I'm gonna take the decision. Yes, definitely. You cannot decontextualize leadership when you make a decision. I mean, look, every country has a different type of government and governance. For example, President Obama must appeal to his people, to Congress, to support him whether to act or not to act. In another country, maybe it doesn't need to do it, but the situation is different. But at his level, the leadership required is sort of an alliance leadership. It's not leading a team or even an organization. It's, as some other leaders have said, it's a collision type of leadership. I was reading recently that one way of defining leader versus leadership is leader is a singular entity. Leadership is multiple people. And in an organizational setting, leadership is a collection of individuals that come together with a focus, purpose, or to use your language from earlier, a passion. But let's not forget that in the process of acting out our leadership, we have what is called responsibility areas. But you cannot run away from accountability. So in collective leadership, you still must hold one person accountable for failures or shortcomings. Otherwise, you cannot hold the whole group collectively. So there must still be, shall we say, someone who stepped forward and said, I am accountable for what happened. But yet you read Fortune magazine, you read Forbes magazine, you read articles on Harvard Business Review. You don't often see or hear leaders of corporations stepping up and saying, I'm the one that is responsible. I think therefore, there's a lot of hype and hubris and you read lots of management books, leadership books, teaching us what to do and so on, it's all of idealism. But the real world is actually very tough. How to be ethical, how to be moral is really a serious challenge. It goes back to the character of the one, one of the three C's that you mentioned, okay? Well, let's come back to help university if we could. And you've been quoted in the media saying, and let me paraphrase here, help started as a social business enterprise so that the less fortunate would have an opportunity for education. You mentioned that at the beginning of our conversation. What's your rationale behind that vision? And more importantly, Paul, what have you done to ensure that helps stays true to that vision? You see education is somewhat different from other businesses. In fact, it may sound like dirty word to use industry or business to characterize education. Although now it's, shall we say, legitimate and accepted. When we say help is a social enterprise, maybe without realizing it, we were somewhat ahead of time. Because nowadays everyone says we must have a corporate social responsibility. When I write whatever I have to write about help, I do not have to say I have corporate social responsibility. Well, there's a very reason why we exist. There's a very rational why we have help to begin with, that it is a social engagement, that it is Arthur-centered. Not self-centered. Other centered businesses. And using education as a social enterprise requires us to be responsible, steward of whatever resources that different stakeholders have given us. But in particular, the trust that parents, students have given to us. And therefore, this is a very heavy responsibility. So whilst we have to make ends meet, but profit is like oxygen, water and blood, but we have our idealism. So for an education institution, we must live by our idealism. So that's why it is a very interesting challenge to combine what we do as servants for people, to serve the people, using education. Because education has noble ends and the act of education is noble. Therefore, it's ennobling. It's not that other businesses are not ennobling, but without offending my other colleagues in the world, I must say to educate someone is very different. It's not the formal process of having a campus to educate. The whole life process is about teaching and learning. That's why in Asian societies in particular, a teacher is revered and highly respected. It's different. And while education and academia can be ennobling, there is still a business element to it. And earlier when we were talking, you were saying that you are guided, your business choices are guided by an acronym, IEDI, which stands for Innovation, Expansion, Differentiation and International. Can you tell us a little bit about that acronym and why you see that as being different when applied to an academic institution as opposed to palm oil business? Okay. You see, over the years, I have developed my own, shall we say, frame of reference, my own template. In looking at business and education. And I have four acronyms to begin with, the four A's and the four S's. So the first A is, are things available? Availability. Are they accessible? Accessibility. Are they appropriate? Appropriateness. Are they affordable? Affortability. So this gives me my first reference of whether I should do something or not to. Then the four S's will come in. The first S is standards. Do we have standards? At least industry standards. But we aspire to have global standards. But the best is we set the standards. The second S is scalability. To grow a business, there must be scalability. So the models of handling resources, human beings, and giving a good value proposition to, in my case, the students is important now to scale it up. And have a global reach, R-E-S-H, without destroying the rich R-I-S-H, as in richness. The third S is sustainability. Because if we cannot persevere and sustain, then we die. See, we must remember that, as Peter Drucker has said, probably our first duty as a business organization is to create a customer. But let's not forget the Darwinian principle and what is built in our DNA. We also have a duty to perpetuate the life of an organization, or our species as human beings. So between creating a customer and perpetuating the life of an organization, we therefore have to do things efficiently and effectively and rightly. And the last S is, coming from the academic world, I realize we are very slow in decision-making because there are so many layers of decision-making committees and so on. And the academics have a very different world view. It's a timeless world, you know. So I therefore added in the last S, speed. Speed. Yeah, speed after long reflections, speed to the market. So the four S and the four S would govern us. But I also noticed over the years, we have three things that are not well integrated. One has to do with entrepreneurship, like starting help in 1986, or the startup business. So starting up a business, the type of entrepreneurship is different. Now we have matured into a sort of a bureaucracy. There are roles, procedures and so on. These are good, but organizations are not meant for entrepreneurship. They are only meant for efficiency at best. So the more procedures and rules you have, the greater will be the chance of leadership dying. So how do we develop intra-preneurship, corporate entrepreneurship? I admire great companies like the 3M, where the mission statement states very clearly that it must renew itself, renovate its products, innovate new models and so on. But to do that, we must have a growth mindset. A growth mindset. Coupled with certain types of competencies. So I developed an integration of e-entrepreneurship, L-leadership, M-management. So I have my own M model. Okay? M is the name of a tree, but it stands for entrepreneurship, leadership and management. And recently, our graduate school at Help University is called the M Graduate School. Because when you look at the literature, it is now starting to recognize that we need to integrate the entrepreneurship, the leadership and the management. So when we started in 1986, our leadership competency was very low. But now as we grow to manage an alliance of partners, the leadership is very different. So the leadership has to grow. Not just personally, but also as you say collectively. So that is the great challenge. But in the meantime, what about the M? The management. We only managed a couple of thousands of dollars in 1986. Although proportionately, there was high risk. But now we have to make decisions about a few hundred million relating to campus development or whether to have a campus overseas and so on and so on. So the management competency must also grow. So you have this triangle, the E, the L and the M. So all these together, the 4S, 4S, the growth mindset and the ELM competencies will act on the I and the EDI. So I use innovation as a verb. Not as a now. So I have to innovate, expansion, differentiation and internationalization. Now if I don't innovate the EDI, the EDI becomes DIE.  We will just perish. And in our case, especially in my early life, I've learned this lesson. There's no option for failure. So I constantly remind myself about, if I remember, you said Apollo 13 or something like that, that you are in outer space. You're going to have two days, so much oxygen, and there's the oxide coming. You have to take care of two, three lives. You cannot blame anyone. The command center on Earth cannot help you very much. You have to make do with what we have. So in running help, we need to develop a robust business model, a set of actionable strategies, and all the time deliver on an hourly basis in our case. Because if you don't teach well, you'll be challenged in the Facebook. It's as tough as that. It's as tough as that. We cannot replace broken promises. For some other businesses, if you deliver a car and it doesn't work, I can replace it.  So we have to act like doctors. The way I look at running an institution like a university, I have to treat every student like the doctor treats his patient. That means everything is almost life-threatening. The reason why customer care is so poor and bad in every organization is that we do not have the mindset and attitude of a doctor, the type of caring attitude. Doctors make the wrong every morning. How are you? What's the temperature? Teachers don't do that. CO don't do that. Don't look at the end results, the quarterly results or the semester results. Then only we react. But doctors have to make the wrongs, especially if it's a life-threatening problem. So I think we must cultivate that type of attitude. So I seldom look at what other universities do, that they're not great, they're great in other ways, research and so on. But to care for a customer, I have to look at maybe the hospitality industry, but in particular, healthcare. That's how I look at some of the things to innovate what I do. Well, and people at your age, given all the achievements that you've had and are continuing to have, would be pretty happy at just moving into retirement. But yet you've stayed very active with help. What is your source of energy? What keeps you going? What gets you up in the morning that you haven't pursued other things that you're sticking with help? When I say I'm going to stay on some of my colleagues may not be so happy. I better leave us alone. The only reason if I have one to continue with what I do, but not necessary to hold positions, is that I genuinely have a love for people and a love for, shall we say, wisdom. And what I have been doing through education offers me those moments. Every moment is a moment to learn how to be wise and wiser. There's no other opportunity like education to self-educate until we lose our breath and to help to educate other people. So if I give up what I love to do, how can I find an alternative? There's no second best alternative about self-education and to educate people. So it's more than a business, more than anything else in this world. Whether you're with a formal institution called a university, you are educating yourself, your family members, like in my case, I have five grandchildren, and I suddenly discovered that I have forgotten so many things. And as I've mentioned earlier, I noticed that for a human being, it was a remarkable thing that a baby learns how to stand up and walk. And he also does not fear or feel ashamed when he mispronounces words to learn a language. But when we grow up, we fear to learn things because we fear criticisms, we fear failures. So even watching my grandchildren learning how to walk and to learn how to speak and pick up words and phrases, it's already teaching me in an educating way. So I cannot give up what I do because everyone I feel should be involved in education in some ways, not necessarily the formal way. If so happens, I have a structure to do it. It's the only difference. It also sounds like if I were to link back to something you said much earlier in our conversation, it's that personal passion that you have as being one of the sources of your leadership energy. It's your passion, it's your values, the different acronyms of four A's, four S's and L's, that all are some of the sources of your energy. Yes, definitely. Some people ask me to have hobbies. It's not that I don't have, but to me they are secondary hobbies. My first hobby is what I'm doing because to me sometimes I tell myself, eh, do you give up what you do for the last 12 hours and certainly switch to carpentry as a hobby? No, but in my case I'm so blessed. I love what I do. So how can I get tired of loving what I do? Consider John Kees. How can I get tired of composing poems with his friends in the Lake District? Or consider Patu and Mossad and all these people. Should we tell them to stop composing and rest a little while and do some carpentry or gardening or something like that? No, because you love what you do. So you cannot get tired of it. You cannot say your energy is drained. Your energy is drained only when you feel very tired of people and tired of what you do and think. That's why you need to have recreation. But I'm actually doing my recreation every moment. There are some people that might say you are a workaholic, Paul, with that type of definition. But another way of looking at it, and I think this is the way you're looking at it, is how can you call me a workaholic when I'm doing something that I love, that I derive great pleasure from? I think even if you call me an alcoholic, it's a good one. Because, as I said earlier, it's other-centered. Because I do it to serve people. I'm not consuming to be drunk and forget the environment and lose my sense of the context and the future. Whatever I do, in fact, enhances my sensing of the environment. And that gives me a very centered way of living it. Lose my sanity, inspire or despise all the pressures. Because they are not precious. Because there are things to be sculpted and improved. You see. Final question for you, Paul. What's the legacy you want to leave behind? I did not dream of living big things. I seldom think about these because I know this will lead to some hoardiness and a sense of self-importance. I think what really matters is, have we, over the years, together with my other teaching colleagues and partners, created a set of conceptual, creative, critical thinking skills for the students. Such that they are enabled to continue to succeed. That's why our mission is very important. To help people succeed in life and to live a life of significance through education. So the first part of the statement says that it is our duty to create some enabling competencies. But the second part is more important in a way that we must not forget the purpose of life. That's why significance, which is beyond the maintained success of life, having lots of money, power and status. So to support our mission, which incidentally took us 20 or years before I put it down, because nowadays I notice many companies write down a mission the moment they start their business. We actually had no mission. We just practiced and lived what we were doing and suddenly one day I said, this is in fact our mission to help people succeed in life and to live a life of significance through education. So later on I supported these with just five statements about our values. And the first one is pride of achievement. It's all self-motivating. There's nothing to do with money and power. Pride of achievement. Number two, sharing success, which means that it is other-centered. We must have the willingness to share our wealth but in this case our knowledge and so on. Then the third value is the courage to be as in being, to have this courage to choose who we want to become. So from be as in being to becoming, a beautiful English, be and becoming. The fourth value is to be compassionate. I mean the world is full of violence and turbulence. Everyone is talking about how to subjugate other people, how to make you submit. It's through wars and weapons but not through education. So recently in fact I created a platform entitled Education for Peace and Prosperity. So I'm going to develop that. And the last value is to be significant, meaning to not waste our life. Even if you live up to 33 like John Keats, make sure you have written some great poems, like Oath to Autumn or whatever. So I think these five values will support our mission very well. But at the same time I also created another acronym. I have a WOW, a V-O-W. A WOW is a commitment. V stands for values, the values that I just articulated. And all opportunities, jobs, there's a chance to study. And W is very interesting. W means wealth. But in Chinese there's a Chinese character. You can spell it as C-A-I. And it's pronounced as chai. Chai is written differently when you write the characters. But it's pronounced as chai. Now it has two meanings. One is intellectual wealth or knowledge. The other one is material wealth. So you see this beautiful word chai has two meanings. It's not just creating material wealth or assets, but also the intellectual development. So I have a WOW. So when you look through all this process of the M model and the IEDI and our mission, our values, ultimately should we or shouldn't we, therefore depends on WOW. So I have a WOW in whatever I do. And I have conceptualized all these. And I think my next stage of what I would love to do is to promote worldwide. And I hope many people will join in education for peace and prosperity. Paul, I'd like to thank you for your time, for your generosity, for sharing your thoughts about leadership and the insights about Hope University. We'll take a vow with you. So I'd like to thank everybody for joining us with this edition of the Leaders Room. And we look forward to seeing you the next time.