Professor Michael, Professor Mike Calvill, Professor James Bartley, a friend of Saltin Islands, distinguished guests, friends of Saltin Islands, all get a one talks, ladies and gentlemen. Yes, I have a cold, but you can blame it on the kiwis. I bring greetings to you from the people and government of Saltin Islands, and I'm greatly honoured to be back here at A&E. I thank my friend, long time friend, James Bartley, for convening this August gathering. I have been invited to share some thoughts on Saltin Islands relations with Australia, in particular the question as to where to from here. And I thank you all for gracing this event with your presence. Before I continue on, let me first pay tribute to and acknowledge the Ngunnawal tribe past and present, the custodians of the land on which this gathering is held, and I thank you. Ladies and gentlemen, some of you may wonder how Saltin Islands got its name. It was named by the first European explorer that came to the islands in 1568, a Spanish by the name of Alvarado de Mendana. I guess this is why we have all the islands named in Spanish names. During his first expedition, he discovered some gold nuggets in one of the ridges. I guess he must have thought this is the second King Solomon's mines, and named the country Saltin Islands. Unfortunately for Saltin Islands, he may have seen the last gold nugget during his trip. All of you would know the country's capital city is Honiara located on the island of Guar Canal, famously known as the island where the tide of the Second World War was turned. Saltin Islands has the second largest land area behind PNG and the third largest population after PNG and Fiji. We speak more than 100 languages and dialects, which also pose the challenges to effective communication and development. You would already know that Saltin Islands is one of Australia's closest neighbours. We are connected geographically through a shared maritime border. Saltin Islands and Australia also share cultural, social, educational, economic and political links and relationships. Saltin Islands enjoy cultural and ethnic links with fellow Australian indigenous Melanesian of tourist trades. We are also connected by blood and Christian faith to the Australian South Sea islanders who came and worked in the sugar cane plantations in Queensland back in the 1800s. In fact, I had the privilege to have fellowship with some Australian South Sea islanders up in Bundaberg just a day before yesterday, and I paid a visit to the cemetery of some of the early Saltin Island descendants, which Australia is now their home. In this regard, and upon reflection, it makes me proud that Saltin Islands has had, in fact, contributed in a small way to Australia's development. This contribution continues today through their descendants. Australia is one of Saltin Islands' major partners for many decades, starting prior to the country's political independence through to today. The Australian concept, a major, giving people a fair go is reflected in my country's concept of the one-talk spirit, characterized by a common identity of caring, sharing and working with each other. Australia's investments in my country over the past decades has been sizable. In fact, Australia's single largest development is the single largest development partner and donor in Saltin Islands. The total quantum of Australia's investment to Saltin Islands, like many Pacific countries, goes beyond the bilateral relations. It includes Australia's investment channeled through multilateral arrangements, such as through international financial institutions, and through other global mechanisms and arrangements. Australia also supports specific regional mechanisms such as a core funding and project funding channeled through regional intergovernmental organizations such as the Pacific Island Forum Secretariat. I guess what I'm trying to say is that Australia's investment in Saltin Islands over the years had been substantial and I acknowledge that with all sincerity. On its part, my country has made some small but serious investments to grow our relationships with Australia. Here in Canberra we have constructed a high commission located at various points throughout the high commissions, two terms representing the cultural identities from all nine of our provinces. We accept the hand of friendship extended by our hosts through the offer of the land upon which Saltin Islands High Commission is built. Ladies and gentlemen, my country and people, perhaps more than anyone in the Pacific, understood and appreciates the benefits of regional cooperation. For Saltin Islanders, regional cooperation goes beyond the abstract. For us, regional cooperation is real. We are where we are today because of regional cooperation spearheaded by Australia that saved us from ourselves. I believe Saltin Islands and Australia are much closer today in part as a result of the Australia-led regional assistance mission to Saltin Islands that was launched in your beautiful country as Operation Health and Friend and more commonly known during its period in Saltin Islands as Ramsey, the regional assistance mission to Saltin Islands. I acknowledge that many of you in this gathering would have been involved in one way or the other with Ramsey, some in more limited roles than others. Like my good friend, Professor James Badley, being involved in playing a major role in Ramsey. Ramsey was a regional response to my country's call for help, led by Australia and strongly supported by New Zealand and 12 other Pacific Island countries. Ramsey's mission objective was to re-establish the rule of law, stabilize the economy and strengthen our state institutions. I am both humbled and proud to say that when Ramsey completed its mission in June last year, all three mission objectives have been achieved. Over the 14 years of Ramsey's lifespan, the cost came to some Australian dollars 3 billion. Over this period, some 8,000 Australians served in the mission. These Australians and their families are now one-tocks and friends of Saltin Islands in Australia. This has resulted in another success story the relationship established through a regional cooperation assistance has now resulted in a strengthened profile of Saltin Islands here in Australia and a stronger bilateral relations between our peoples. I take this opportunity to acknowledge the fact that a number of Ramsey officials made the ultimate sacrifice. They will not be forgotten. As a result of the training provided under Ramsey, Saltin Islands police officers are among the best trained in the Pacific. Saltin Islands is now sharing these skills through our participation in the United Nations mission. As well as that, our police force personnel are also providing training assistance to some of our regional police officers. Acknowledging the assistance provided by the region to Saltin Islands, we awarded the Cross of Saltin Islands to eight Australians who were charged with the responsibility of being Ramsey coordinators over its 14 years of service, including the convener of this session, Professor James Bartley, who was one of our Ramsey coordinators. To the people and government of Australia who made Ramsey possible and helped my country back on its feet in what was one of our darkest hours, we owe you a debt of gratitude, not only providing much of the resources that supported Ramsey, but for also doing some of the heavy lifting, working in partnership with New Zealand and the rest of the Pacific Island brothers and sister states. I thank you all sincerely on behalf of the people and the government of Saltin Islands. Australia is Saltin Islands' security partner of choice. Saltin Islands and Australia signed the Australia-Saltin Islands Security Agreement last year, highlighting a new dawn, the two countries' security cooperation. Australia's support to maritime boundary surveillance through the construction and supply of modern patrol boats addresses huge gaps in capacity that my country faces. We anticipate receiving two new patrol boats early next year and are in continuing discussions in relation to other gaps in our security parameters. Saltin Islands is also watching with interest the proposed Australia-Pacific Security Colleges with the view to seeking more clarity on what it is, its purpose and mode of country engagement. Ladies and gentlemen, Australia had and continues to make huge investments in Saltin Islands. It provided substantial financing support to many key government sectors and institutions including huge investments in health and education. Australia also targets its assistance to infrastructure areas that can unlock economic opportunities, for instance. During this visit, I will be witnessing the signing of a landing agreement for a new submarine cable link between Saltin Islands and Sydney that is jointly funded by the governments of Australia and Saltin Islands. In a few weeks, I hope to be back in Australia to sign the final agreement on the submarine cable project with Australia and Papua New Guinea, who is also benefiting from this cost-sharing arrangement involving the three countries. Australia has also invested in the new Tina Hydro project. When completed, Tina Hydro will provide up to 80% of the power requirements for the national capital, Honiara. However, much still needs to be done as Honiara represents less than 20% of the total population. My government is perceiving further opportunities to shift power closer to the rural and remote populations using renewable energy to increase the country's power penetration to many provincial and rural households and reduce reliance on fossil fuel. Other areas of interest including tourism, renewable energy, minerals resources, fisheries, the development of small and medium enterprise sector including cultural industries. Saltin Islands participates in other broader regional cooperation platforms within the wider region aimed at enhancing win-win opportunities for countries involved in these cooperation and collaborations. Examples of these include the Melanesian spearhead group which comprises of more than 90% of the total population of the region. The parties to the narrow agreement, Saltin Islands is one of eight countries in this group which focuses on matters relating to the management of the tuna resources based on a vessel day scheme that recognizes the value of tuna harvested in the country's EEZs. Joint Continental Shelf claims with the United Nations Continental Commission between federated states of a state of Micronesia, Papua New Guinea and Saltin Islands. And Pesa Plus with Saltin Islands has played a leading role in concluding the agreement and is now focusing on its stratification. Regional cooperation benefits countries where joint efforts result in better value and or greater efficiency of scale than doing things individually. Saltin Islands participates in regional cooperation and collaboration that lead to benefits for the country and people. Pacific Island countries are classified by the United Nations as small islands developing states. Forum leaders advocate we are large ocean states that occupy the world's largest ocean. To put this in context, land area of Saltin Islands is about 28,000 square kilometers. Our EEZ is about 1.4 million kilometers, a square kilometer. Kiribati, on the other hand, has a much smaller land mass, but its EEZ is one of the biggest in the world, stretching as far as Hawaii, the United States, covering more than 3.5 million square kilometers. Policing our waters from illegal fishing and poaching is difficult given the expanse of the ocean. The ongoing Australia Pacific Patrol Boat Program is another example of a successful regional cooperation program going as far back as 1984. Saltin Islands Patrol Boat to lucky gifted by Australia made global headlines by apprehending a U.S. trawler, the Jeanette Diane. Our friend and partner, the United States, imposed sanctions on Saltin Islands' tuna exports as a result. The Pacific came together upholding the sovereignty and respect for Pacific Islands 200 mile EEZ. The matter was resolved. Pacific countries later signed a Pacific Multilateral Fisheries Treaty with the United States. Saltin Islands will also be developing our own national oceans policy. This is important for us as a maritime nation whose economy is ocean based. The policy will cover the economic, social and environmental dimension of the ocean. Work on this policy will be undertaken by the Ministry of Foreign Affairs and External Trade. Australia's support on developing this policy will be most welcomed. Ladies and gentlemen, like all other countries, Saltin Islands exists in a rapidly changing world. It relies on the actions of other nations for its long term survival. Climate change, natural disasters, epidemics have the potential to impact the country's development negatively. Lack of marketable commodity in a number of sizes affects the competitiveness of small countries such as Saltin Islands. Multilateralism is the cornerstone of Saltin Islands' international engagement. Having a rules-based international system that is open for all, recognizing the special circumstances of seeds is something we will always support. International laws, international treaties are our first and last line of defence. These are the only instruments open to us to influence bigger powers to do the right thing. Australia's continued support on several crucial issues to Saltin Islands at the global level is welcomed. Ladies and gentlemen, I have recounted the historical foundations of our relationships with Australia, as well as sharing some thoughts on some of our current relationships. I now wish to focus the last few minutes looking ahead where to from here. I do not have a crystal ball foretelling me of my country's future. Like many of you, I am both anxious and perhaps excited about some of the prospects for Saltin Islands' development ahead of us. However, experience has taught all of us that caution is an excellent friend, and mental particularly in dealing with matters in the context of a rapidly changing world. Saltin Islands will anchor our future relations with Australia on five key pillars, namely, one increased focus on trade and investment. We wish to pursue trade opportunities to supplement development assistance. In the long term, developing better trading arrangements hold the key to our sustainable development opportunities. Secondly, enhanced security arrangements. This is a mutual priority of Saltin Islands and Australia. The current arrangements and proposals are currently being discussed, will provide the platform upon which we will jointly agree future security arrangements between our two countries. Thirdly, infrastructure investment in health and education. Infrastructure investment into areas that will unlock economic development and continue focus on the development and sustenance of our human capacity through ongoing support to education and health would be priority. This will be supplemented by support to other sectors of priority to be jointly agreed between our two countries. Eradication of malaria by 2030 and reversing the trend of non-communicable diseases will be two key priorities. These two diseases currently account for eight out of every ten deaths in my country. Fourthly, enhancing labour mobility through a more strategic partnership agreement. Labour mobility is a key priority for Saltin Islands. We have already received excellent feedback from farms and companies that recruit Saltin Island workers. We wish to place a strategic partnership framework between our two countries that addresses unskilled, semi-skilled and skilled workers arrangements. Fifthly, strengthening the tourism sector. Tourism is a key part of our solution to development. We will pursue opportunities with the Australian government in this sector. I see Saltin Islands relationship with Australia in the coming decades as one that will go from strength to strength. A relationship that is based on mutual respect and trust and one in which both Australia and New Zealand will benefit. The Australian people through their government had supported my country at its time of greatest need and has continued to invest in strategic areas that will help the country unlock some of its own development potentials so that it can start to focus on a journey of sustained development. With regards to national security, Saltin Islands defines security broadly from a sustainable development context to cover non-traditional security issues such as food, water, energy, health security and biosecurity, security relating to natural disasters and climate events and the more overt security of national jurisdictions. Saltin Islands welcome Australia's commitment to expanding the Pacific labour mobility. Saltin Islands would like to take a more strategic and long-term approach on the labour mobility agenda. I have been discussing the concept of developing a strategic partnership framework for enhancing labour mobility opportunities between Saltin Islands and Australia within an overarching framework of the Australia Pacific labour mobility initiative. Such a framework would involve establishing a database of worker employment opportunities in farms and other employers particularly in rural Australia. On the Australian side, this will comprise, among other things, a list of all workers needed by older farms with the prerequisite skills levels. Approximately 18,000 young Saltin Islanders join the labour market every year. Between 7,000 to 8,000 of these are graduates from our rural training centres, nursing schools, the Saltin Islands National University and they could participate in labour mobility schemes through the seasonal workers program and similar programs for the semi-skilled and skilled workers from Saltin Islands. Tourism has tremendous potential in Saltin Islands. We will explore an expansion strategy for cruise tourism with Carnival Australia. It is my intention that within five years, Saltin Islands will be where Vanuatu is today with up to six cruise tourist ships to various parts of Vanuatu per week while pushing the labour mobility agenda and cruise tourism we will simultaneously undertake a comprehensive approach to increasing the numbers of visitors to the country under the normal, longer-stay tourism. We will step up our capacity to strengthen surveillance of our territorial waters. We share common maritime borders with Australia, New Caledonia, PNG, Fiji and Vanuatu. Having a stronger fully equipped enforcement presence along our borders is a direction we are taking. The submarine cable and the Tina Hydro projects which I have already spoken about are set to unlock many social and economic development opportunities in Saltin Islands. Both projects which Australia has invested in are anticipated to be operational within the next couple of years. Saltin Islands will discuss closer cooperation on visa arrangements with Australia. Movement of people between our two countries is increasing. Saltin Islands has visa-free arrangements with some other countries including with the European Union for visa tourist visas. Having a similar arrangement with Australia and New Zealand can be a game changer for Saltin Islands. Ladies and gentlemen, I would like to conclude by ending where I began by reaffirming that Saltin Islands is proud of its relations with Australia. Australia is partner of choice and a very important friend to Saltin Islands. We expect the good relations between the two countries to go from strength to strength in the coming years. I thank you all.


Yeah, so it seems like we're about in time and I believe we can begin. So thank you all for joining me today, for our early first session for today. Thank you for joining me after last night's parties. Actually I was quite surprised that people actually came after the parties and usually people take their time with their handovers this morning. So thank you all for joining. So I'm Arthur Berezin, Senior Technical Product Manager for OpenStack based out of Israel. And today I'm going to talk about a high availability for OpenStack and basically discuss the architecture of how to build in a highly available OpenStack environment basically what should we concentrate on and what should we resolve while we're building this environment. So I'm going to focus in on the enabling services that we use or that we can use to build a highly available OpenStack environment and to cover all the services and make sure that they survive failures. I'm going to discuss the various shared services like MariaDB and the message queuing that obviously we have to use and we have to make sure it's not, it's fault tolerant. I'm going to cover the various OpenStack services or at least the basic ones as we don't have too much time to date. And I'm going to end up by covering some of the topologies and what we should be taking, what should we think about when we basically build such an environment.  Please? All right. And Android users? All right. Much more. So I'm hoping you're all familiar with the QR function. You can use it now instead of taking photos during the slide deck. So feel free to take a shot of the QR code and just download the deck on the slide, the on slide share. All right. And I'm going to share that by the end of the session as well. So if you don't have a chance to actually do it now, you'll be able to do it by the end of the session as well. So this is the environment we're trying to build, right? We're building our top-notch environment with high-performing, high-performing compute nodes, high-performing controller nodes. And everything should be always running without tolerating any failures on the environment. If something slips by just a millisecond, so we're lost the race. But as we all know, reality is not always that simple. So obviously, yeah. So obviously, failures do happen and we have to encounter for them and something fails. With OpenStack, obviously, if one of the controlling services fails, it doesn't mean like the actual virtual machines stop functioning, right? The environment still probably keeps on running, depending on the failure, obviously. But it is extremely important to make sure our controller's controlling services are false tolerant and if something happens while the environment runs and things always happen, so we need to make sure we have another spare wheel that actually controls the car after the failure occurs. So we are trying to build active-active services. We don't want to wait for failovers to happen because failovers take time. It takes time to start another service on another node, for example, or bring the first node to keep on running, so we would like to make everything as active-active as possible. Obviously, there are some limitations in OpenStack today that we're still trying to solve and it will take us another cycle or two to solve them and there are lots of very good progress on other services as well. So and again, if nothing else, if the service itself does not support active-active configuration, so obviously we can still fall back to active-passive, but we would like to do that as least as possible. And we would like to create a scale-out environment, right? We want to make sure our environment can serve as many requests as possible and serve as many users as possible. And this is the reality we've all seen, this diagram obviously opens like a fairly complex system constructed out of many services and we need to make sure in each and every one of them is highly available. So first of all, I'm going to cover some of the enabling technologies that we use to build a highly available environment. So the first is SpaceMaker. So SpaceMaker is a cluster resource manager, basically what it does, it monitors a bunch of resources that it supports and once it detects something happening with one of its resources, it makes sure the resource comes back to life, either by restarting the service or by fencing the node and we're going to cover that just in a bit. So SpaceMaker supports various types of resources. One of them is the virtual IP, which is basically a floating IP that can move from one host to another and basically the virtual IP resists on one host and having a single IP address for both nodes that run the service. SpaceMaker also supports system D monitoring of system D services. So we don't run the services themselves by system D, rather, SpaceMaker monitors those services and controls them. So we don't have to enable the system D services themselves. SpaceMaker takes care of that. And SpaceMaker also supports clone services, which basically mean active, active configuration. So when a SpaceMaker cluster is configured in a clone configuration, the service runs actively on all the hosts that the cluster is configured on. And another very important function of SpaceMaker is the shoot the other node in the head function. So when something weird is going on and SpaceMaker is not able to actually reach the service or not able to restart the service, for example, there's a failure occurring when you restart a service or such scenarios. And we have to make sure that we have full control of the service, right? We don't want to create any split brain scenarios or any scenarios where service is out of control. So we basically should be able to control the host itself by using its power management, most commonly the IPMI device for that host. So for OpenStack, we use SpaceMaker basically by creating a virtual IP address for each and every API endpoint. So basically the users address the API, the virtual IP address. So once failure occurs, obviously, you don't have to reconfigure anything. And you don't have to let other services know of your new IP address. You just use a single IP address that bounces between the services themselves. We also use the system declone the resource functionality, which means, again, SpaceMaker monitors the services themselves and controls the system resources. And we configured to shoot the other node in the head functionality, in case something funky is happening in the environment. So, SpaceMaker is constructed of a single service, the transmitter host called the PCSD, and it uses the IPMI or the remote management of that host. Another technology that we use is HEProxy, which does load balance. So HEProxy is obviously a very popular web load balancer and proxy. It's capable of handling HTTP and TCP requests and it does it very well, and this is why it's extremely popular for web applications. And it also does health check monitoring. So basically, it makes sure that the IP addresses that it balances to are alive, and if something is happening, HEProxy stop distributing, stop sending requests to nodes that are not responding. And obviously it has load distribution. So we can use various load distributions with OpenStack, and it's really dependent on the service. I'm gonna cover some of the services and their policies. But by the end of the session, I'm gonna point you to the complete reference architecture that you can go over and see for each of the services. What are the recommended configurations? So some of them are round robbing. So when we have complete stateless services, we don't really care for the session to go to the same node every time a user makes a request. So HEProxy just does load distribution and just does round robbing across all the nodes it's configured to use. Or we can use StickTable to make sure that the session that started using a specific node, it's next request they're gonna use that node. So it also does the API isolation, so we don't address the API directly. And it also does failure detection. Which basically if one of the nodes fail, HEProxy stops distributing the load to that node. So let's go over the life cycle of such an environment using those components. So we'll take an example Horizon. So user try to connect to access Horizon Web UI. It could be Horizon or it could be any other stateless status service. So obviously this is a single point of failure which we would like to address first. So we can put an HEProxy in front of it. And HEProxy distributes the load between the three nodes for Horizon. Now, again, this still creates us a single point of failure. Because HEProxy by itself is a single point of failure now. And each of the Horizon components could also fail. And we should make sure that these services are rebooted or restarted once something funny is going on. So we build an HAP and a pacemaker cluster, a cloned cluster around the Horizon services. Which takes care of any failures for the HAProxy, for the Horizon services. So if something going on, pacemaker basically takes care of that and reboots the service. And if it fails to reboot the service itself, it restarts the node. And for the HAProxy, we can also create a pacemaker cluster for HAProxy itself. So if something happens to HAProxy and it fails, pacemaker can restart the service. And also, we can use a virtual IP for this specific service. So basically the flow for this environment would go from specific node. And that specific node that runs that virtual IP is gonna load balance the request coming in and basically send it over to one of the Horizon instances. So for the shared services, so first of all, the database that we like to care of, we can use a GaleraDB, we can use Galera with MariaDB, sorry. And what Galera does is basically it is a multi master node replication for the database, it does a row based replication for the database. And it basically allows us having multiple active running nodes, running the database services themselves, and basically access each of the nodes having a consistent data for across the nodes. Also supports node auto joining, recovery, conflict resolution, etc. Lots of really cool functionalities. Basically, literally what we need to make database highly available and specifically for open stack, this is really a good use case. Also supports the native client APIs for MariaDB, which means we don't have to introduce any changes to the environment itself. We can just point it to specific IP address of that node and basically replication takes care of, happens underneath this service. So the other functionality we use, we use RabbitMQ mirrored queues. Another functionality, clustering enabled functionality for RabbitMQ. Which basically replicates the message queues for each of the services across multiple nodes. This is functionality of RabbitMQ that we can leverage here to make sure that the message bus itself is highly available. So for the services themselves, Keystone is rather simple service. It runs under Apache under HTTPD, obviously multi-threaded. And this service basically accesses all the identity resources and accesses database for assignments or could also store the identities in the database itself. So again, we use pacemaker and Keystone to make an HAProxy to make the Keystone itself highly available. Rather simple service, so we have the virtual IP running on one of the nodes. And that node that has the virtual IP basically makes all the load balancing for the API calls for Keystone. And the Keystone service basically does access the environment. So the important thing to keep note here is that each of the hosts should have the same identical SSL certifications. Otherwise, you're gonna have some weird behaviors. So when we're configuring this environment, we have to make sure we copy the SSL for all of the nodes. And we also then need to take notice that the caching for each of the services is local. So obviously it would be a bit more efficient to have the calls coming to the same node every time. So Glance is also a rather simple service. It has two Linux services. One is Glance API, which takes care of all the API calls. And it also accesses the storage and we also have Glance Registry, which communicates with Glance API over HTTP calls. And Glance Registry takes care of registering all the interactions with Glance to the database. So again here, as we used with Keystone, but this time we're doing it for both services. We have two virtual IP addresses. One is for Glance API, the other is for Glance Registry. And both of them obviously are load balanced across the nodes. So again, as the flow, we have one node running the load balancing for the API calls and the other running the load balancing for the registry calls. So there's a bit more complex service, but also not that complex. We have one issue with it specifically. So we have four services. The first is API, which takes care of all the API calls. We have the scheduler, which is responsible for placing volume. So once I send a call to create a new volume, and sender scheduler decides on which sender volume to create that specific volume, obviously, based on filters and weights. And the sender volume itself makes all the storage calls. It uses basically the data path to the storage device we're using. Rather, it could be a distributed storage or traditional storage. It doesn't really matter. It uses the driver to access the storage via the management path to that storage. And then the API passes to Nova the data path. So basically the VMX accesses the storage directly. So the only thing to note here is basically that sender volume is still not capable of running active-active due to some potential race conditions. So for example, if you make the same volume creation call with multiple volumes, there could be some conflicts while the environment runs. So it is currently recommended to have this as an active-passive configuration. But lucky us, we have the design session as well during the OpenStack Summit. And basically this is one of the topics that the sender developers are currently trying to address with various locking mechanisms for sender volumes to make sure that there aren't any race conditions within the service itself. And sender volume is still active-passive as well. And I'm assuming that once the sender volume is taken care of, we can use the same principles to apply on the sender backup as well. So Nova is constructed of Nova API service, which obviously does all the API calls. We have the Nova schedulers, which does the VM placement, chooses the VMs based on weights and filters. We have the Nova conductor, which does all the database access. And basically every command that the Nova compute service does, the Nova conductor makes that register over to the database. And Nova compute runs the actual virtual machine instances using various drivers, most commonly a KVM with Libvert driver. So usually OpenStack is deployed with configuration of controller services and compute services and compute nodes, sorry. So for the controller services, all of them are capable. All of them are basically stateless services and all of them are capable of running in an active-active environment without any issues. But for the compute services themselves are still their own single point of failures, we do have system de-monitoring the Nova compute service. So if something happens to the Nova compute system, it still tries to recover that. But we don't have any virtual machine VM availability. And that's not yet fully supported. We do have some reference architectures for that, trying to solve this. And basically there are a few blueprints for Nova to enable this and basically to help Nova recognize that node failed much more quickly. So there are some blueprints to cover that as well. But this is not something that is fully supported at least by Red Hat. So the way it's gonna work once there are, as I mentioned, there are some reference architectures trying to solve this. And the way it should work basically is, we use something called pacemaker remote. And what this means is basically that the compute service is not part of a cluster for pacemakers since pacemaker supports only up to 16 nodes. Rather, it uses this node only to monitor its state with coursing. But it doesn't do any actions on the node itself. But it still can use its controlling mechanism to make sure that the service is rebooted once something weird is going on. And once we can actually make sure that the service, that the node itself was rebooted, we can start the instances on another node. Now, one little disclaimer, this still relies on shared storage for the VMs. So we cannot use any thermal storage on local disk, obviously. This will not work on local storage. So Neutron, one of the, I think, most complex services today with OpenStack. Constructed out of several services, the first is the Neutron server, which does all the API calls and all the management for Neutron. Runs all the commands, etc. Uses the message bus to communicate with all its services. And this is the only service that actually makes database calls and saves all the states of the networks over the database. It also has a Layer 2 agent, most commonly used with OpenVswitch.  The Layer 3 agent is a more interesting service. It does all the routing and all the SNAT configuration for Neutron. Basically allowing the northbound connections or routing between virtual networks for Neutron. So we also have the DHCP agent, which gives us DHCP services for the agents, for the instances. So once we boot an instance, it gets it IP from the DHCP agent, which uses DNS mask in the default configuration. And we also have the load balancing as a service agent, which uses HAProxy for load balancing. So if you take a look at the flow of the environment, we have, again, this is a common deployment mode. So we take a look at this environment. We have on all the nodes that actually do any networking activity. And this obviously the compute nodes and the Neutron network nodes. Those hosts run the Layer 2 agents, which are basically responsible for configuring the Layer 2 configuration. On those nodes and basically constructing all that, creating all the tunneling needed for communication, etc. And then we have the Neutron network node running the Layer 3 agent, which does all the routing and creates all the routers to basically enable communication between various tenants. And also enable communication to the outside world, to the external network, or to the internet, obviously. So the Neutron service itself is rather easy. It's the Neutron server, not Neutron API. This is a rather easy service. It's a stateless service as much as any other API service. So there's no problem running it in a cloned configuration and make it run on all the nodes simultaneously. The DHCP, so there were a few changes in the latest cycle, in Kilo cycle basically that enabled running the DHCP agent in active, active configuration. This is a Neutron configuration that you would have to enable. And you would also have to make sure that the Neutron DHCP service, the Linux service that runs on the host, it's that it is highly available as well. So you would have to create the pacemaker cluster on top of that. In addition to the Neutron configuration enabling the DHCP services being highly available. So the second part is also the Layer 3 agent, which is responsible for creating the virtual routers on the network nodes. So also this also relies on various changes happening in the Kilo cycle, basically enabling a VRRP between the Layer 3 agents. So what it does is basically it creates another, the service itself, the Linux service runs in a cloned configuration. So the Linux service is active, active across the nodes. But each of the nodes, since there's communication between the Layer 3 agents over VRRP, we create another passive router on another node. So in this example, if this Layer 3 agent fails, we already have a passive configuration for a router for this environment. So if this node fails, the other router will become active and will start serving as the router for this environment. So as I mentioned, another few changes happened in Kilo cycle, basically enabling HA with VRRP for the Layer 3 agent, which was one of the major failure points for Neutron. Also enabled the HA for the DHCP agents. And there are some plans for Liberty as well to enable DVR, which allow us to create distributed virtual routers on each of the compute nodes. So what this would mean is basically we would have the routers distributed across all the nodes, so we wouldn't have to create the single Neutron controller nodes. Another very interesting configuration is basically running the DVR distributed virtual routing alongside with VRRP. So for example, we would have the routing themselves done within the compute nodes, but running the northbound traffic accessing external networks using VRRP. So this would actually give us a really reliable environment that would be able to access it, would be able to segregate the external networks, but also make the east to west traffic be routed more without going to the Neutron node. And there's another initiative also making the DHCP services fully distributed, but this is still not fully targeted and this is still under research. So going back to Verizon, one of our simplest services. Again, Verizon uses all the APIs to display its data and to make any changes in the environment. It does all the direct API calls for all the services and it's as simple as Keystone actually, from the architectural perspective. Simple Django application making the API calls under Apache HTTPV. So again, we have here a simple, rather simple, active-active pacemaker configuration with HAProxy configured. So I'm gonna address this a bit cautiously since each and every one of us has his own environment, is interested in implementing his own best practice or his own use case. There are multiple use cases for OpenStack, obviously. We all love OpenStack because it's such a robust system that allows us basically to configure those configurations. So the most common configuration is multiple controller nodes. As a three controller node, the environment running all the services, including the database, including the message bus, and all the controlling services, and having multiple compute nodes accessing the environment, serving as the VM hosts. But we can also obviously segregate some of the services. So we can, for example, pull the neutral services to run on their own separate dedicated nodes. For example, to have all the networking traffic covered by those nodes, but make sure our APIs are segregated from the network nodes, for example. Another example, we could segregate our storage nodes, basically covering Cinder and Glance on their own dedicated hosts accessing the storage without enabling access to our data store, to our storage devices by the controllers, for example, running the API services. Obviously, another example would be to segregate the message bus and database through their own dedicated hosts. And this basically allows us to make the environment more scalable since we were able to serve more calls by each of the services. So some resources, you got the QR code to download the deck. There are a few really interesting resources and really great blog posts on various components, highly recommend going over them.  So I highly recommend going over the list during the schedule.  Thank you, and you can download the code if you'd like. Download the deck if you'd like. And if you have any questions, there's a mic over there. So feel free to go over the mic and ask questions. Anyone have a question? Yeah, what is the status of, I noticed you didn't mention like the two's library and any of the zookeeper work to overcome some of the problems that people were having with the MariaDB glare problems with the locking? Yeah, so the question was, how do we address the locking issues with Maria? So yeah, there are some locking issues resulting some conflicts that you would have to manually resolve. At the moment, the best practice is to use a single node for the right access. So it wouldn't create any conflicts and multiple nodes for reading for readings and this basically avoids the conflicts. But again, this is still under investigation from the Galera team itself. So it should be resolved, sometimes hopefully. This is from scaling perspective. So we can start with the three controllers initially, but then how do we go about identifying the scaling bottlenecks? Yeah, so the question was, how do we identify the bottlenecks? So this is a good question and this is actually a really kind of tough thing that you would have to go service by service and identify the bottlenecks. There are various tools for performance measuring, making sure which of the components, which of the services is your bottleneck and then scaling that specific service out, no by no, making sure your environment survives, survives the scale that you're serving. But basically, the best practice today is basically going service by service and identifying the bottlenecks themselves and then addressing them specifically. Do you have any use cases for leveraging this HA mechanism for the upgrade, for example, because you have three controllers. So you can do one of the controllers while another one is active. I don't know if you have experienced plans or can you comment on that? Yeah, so upgrades, so yeah. So at the moment, do you mean like any automatic upgrades or just upgrades in general? No, just upgrades in general, because if you have three controllers, yes, so you can basically just shut down one of the controllers to the upgrade, right? And while the other two are still active and running. Yeah, so there are various strategies with upgrading. You can do this by running controller by controller upgrade. This is one strategy, another strategy you can take, you can run service by service upgrades, this is also possible in some cases. You guys don't have like a clear reference how to do this. So yeah, so actually we do address all of them in our documentation, running both multiple service, running simultaneous controllers or service by service. Again, it's up to the use case. There aren't any like, there isn't any like best of them, right? You would have to choose your own strategy. So my next question is about some of the services on your presentation were covered by the system D. So system D can make sure that service is running, right? And in the same time, they were covered by a pacemaker. Are you aware about any problems or conflict with this? Because I see that there's a potential place where you can have like in edge case scenarios, you can have problems and kind of tricky to troubleshoot and resolve them. Exactly, so the question was, what happens when I have a service managed by, so our service is managed by system D or by pacemaker. So exactly, so you don't run the services managed by both. You would have to enable them only in a single place. Either by system D or pacemaker. Exactly, either system D or pacemaker. So for example, for VMHA, you would have to make that run by pacemaker not system D, but most commonly it's done via system D. Microphone, come on, we want to hear you. Pacemaker actually coordinates with system D. So pacemaker can speak to system D and say, system D, I want you to start this, but I want to have control over it. So pacemaker is actually controlling services through system D in a way. So system D in itself can automatically restart and monitor things. But pacemaker can start a service and say, I don't want to use system D to actually monitor this. I want to monitor it and I want to have complete control over the service. That we're utilizing the fact that system D has these nice unit files and things like that for pacemaker to manage. So yeah, thanks. Any more questions? All right guys, thank you so much and thanks for making this early session.


So this will be the history option orientation for the Trivandrum and the Bangalore batches. So let's start the orientation. So history optional. The one thing is that what I would like to say is that it is one of the safest options. One of the safest. I'm not saying that this is the most safe. There is nothing called a most safe option. There are few options which are called the safest. History, sociology, anthropology, geography to an extent. Then there's literature subjects. These all we can say that the safest option. History is also one of the safest option. Reason is that you will get a minimum assured marks. See, you know our optional paper is for higher than 500 marks. And the toppers used to get around 280 plus. This is the optional winners marks 280 plus. And anything between 260 to 300 is a good mark. And very few will get around about 300 marks for any optional. You may have mentioned some optional winners with a 340 out of 500 for some subject. But that is for only one or two person. It's not for all. History, the topper goes up to 339. History topper used to get 339. But that never means all the history students will get that mark. But 280 plus or up to 300 is a good mark. And the toppers usually one or two or a handful may cross 320, 330 marks. But unlike many other subjects where our shockers are there. See, advantage of history is that history, whatever was you will be get something around 250. And you can go up to 300. That is a good mark of the history, 250 to 300. On the other hand, many other subjects, sometimes it may go up to 330. Sometimes that may go down up to 150 also. As I am taking one optional subject and it won't be good to comment which options are like this. I don't name that options, but you can Google it. You can search for options and you will get the points. So history in any condition, it's an all weather road we can say. That is history will give you minimum assured marks. Even if the questions are twisted or difficult or whatever it is. Minimum assured guaranteed marks will be there for the history option. On the other hand, other options, not all options provide this safe net. So that is one advantage of the history. Then the second thing is that many will have the doubt that sir, whether the history option is scoring. And the recent times the problem of history is that we won't get the toppers. The first two, three, four, five ranks history optional. Nowadays, we are not getting big windows in that section. Once again, let me adjust that meeting settings. I know trade option is one second. So that's okay. So toppers are not there for the last five, six, seven years. We didn't get any first rank, second rank, third rank, fourth rank, fifth rank. But still every year, almost around 100 students clear with history optional. So getting into civil service with the first rank or 50th rank, you will be in IS or 200 rank, you will be in IPS. The idea is that you have to get into the service. So every year, almost 10% of the students clear with the history option. So every year, nearly 100 students, they clear with this subject. Okay, 100 students, they clear with this subject. So among that out of 1000 or 800 students, 100 is not a bad number. It's a good number. So this much number of students clear with this. Then the third thing is that the scoring that I will show with the marks of my students just a moment. See, these are my winners in the year 2018. See, it shows that 2017 because that year in which you write the problems that is considered as the year. This is the results that came out in 2018, then 2019, 2020, I will show that. This is the results, six winners of mine, six students who got into the services that year. See the marks, 276, 286, 286, and that's my name, Shobhen George in the top. 280, 286 and 1255. See the marks of my students, they are getting almost the same marks. These are the winners. That year, many students went for interview, they didn't clear, but their history marks was good. I will show that marks also. Even though they got good marks in history, but they couldn't clear the interview, otherwise the list would have been more. So average 10, 12 students of mine write the mains every year because you have to clear the problems. That's not in my hands. Optional comes into the picture during the mains examination only. So 12 students used to write and average six used to get into the services. And those who are not clearing, they are not missing because history marks is bad. So the point here I am saying is that my students almost get the same marks. That shows that they are following the same strategy. Three students who got into the service got the same marks, 286. And that's very good marks, 286. Then 1280 and 276 and 1255. These all are my optional students. And this is why I have the confidence that it's proven results. My students who are writing the mains are getting good marks. So I know the strategy to teach the history option. So I am not going to boast about that in any other way. So this is just for giving you the confidence. I can prove and these are the winners who are into the service. So this is the mark of Parth Gupta. He is also my student history optional student Parth Gupta. In 2018 his paper, see the history optional mark of Parth Gupta. He got good marks, see 275. But he didn't clear, see not recommended. Because his GS marks were not up to that height. So he was not able to clear that in 2018. But the same year, Mohammad Junaid with the 276 became an IAS officer. Mohammad Shabir with the 255 became an IFS officer. But Parth Gupta, even though he scored 275, he didn't clear. Because I want to show that all my students who are writing the mains examination are getting good marks. But for some reason they were not able to clear the exam. So this was in 2018. Now let me show the result of 2019. This is Parth Gupta 2019. Now this year he cracked the examination. And he scored still 269 for history optional. And in the year 2019 there was an average 20 marks less for all the option subjects. That's why Parth came down to do 69. Or else it would have been a 290. If we take with the scale of 2018, that was around 290. But still he cleared in the next year. So 2019 Parth Gupta cleared. Now in IRS IT service. So history optional students are there. They are getting good marks. See, no need to worry about that. These students are like you people. They are in a position to get into the service with history optional. Plenty more results to show. But this is enough. Okay. So history optional is scoring. Is there any advantage for taking history optional? That I will say. There are certain advantages for history optionals. First thing is that our syllabus is ancient India, medieval India, modern India and world history. These are the four topics of our syllabus. Which is divided into two. That is paper one and paper two. This is our optional history paper. And ancient India, medieval India, modern and world history. This is for optionals. Even if you take history optional or not. You have to study this modern India and world history for GS means. So that means even if you take history optional or not. You have to study this GS means modern India world history. That is 50% of the syllabus you have to study. Whether you take history optional or not. Then art and culture is mainly from ancient and medieval. So 20% of that syllabus also you have to study. So first point is that even if you don't study history optional. You take geography, sociology or whatever. Still you have to study 70% of history optional for your general studies preparation. So those students who take history optional. They need not study anything separate for GS. That time you can utilize for other subjects. So you will get more time if you take history as your option. Then the second thing every year 16 to 22 questions will be from history. For the Prillin's examination. That means almost 20% of the exam. Although 100 questions every year 16 to 22 questions are from history. No subject grand guarantees this much questions in the Prillin's. For example, quality questions may range from 4 to 24. Sometimes quality questions came down to 4. That is the constitution questions came down to 4. Or sometimes it can go up to 24. Geography can be come down to 11 to 16 questions. Recent trends last 2-3 years trends I am saying. This is how the other subjects are there. So this 16 to 22 is very crucial in clearing the Prillin's examination. And those who are taking history as your optional. You can have a cake walk in these questions. You will have a definite advantage than other students. In this 16 to 22. So this is the two advantages of history. One thing you will get more time to prepare for other subjects. Plus you will get more marks in this GS for Prillin's as well as the main's. Because main's paper 1 you have history and world history and art and culture. Definitely you will score marks plus your time will be saved. That is another advantage of history option. So these are the topics which I would like to say. Then one more thing. Facts and figures. See history main's and Prillin's is entirely different. See today morning we studied many things. That was for Prillin's. On the other hand the main's examination that is optional examination is entirely different. Prillin's is facts based. Here you have to study the names, terms, years. So many things. But that no one can save you. Because preliminary is for all the students. And general studies Prillin's history is factual oriented for Prillin's. So whether you take history optional or not you have to study facts there. But on the other hand main's is analytical. It is analysis. Why this happened. What were the consequences. How this could have been avoided. Was the decision of Mahatma Gandhi to link Laughath did good for the national movement. There you are giving only analysis. Believe me you are writing the main's optional answers in history without mentioning any name or year. Very rarely you need to mention any year or any name in the optional history. Because optional is not the remembrance of the crude facts. Optional is clear cut writing and it is the beautiful analysis. You might have seen this news reports and TV shows where they give the analysis of the events. Here you are analytical. You are just like an adhothit argumenting in the court why this happened. How this could have been avoided. What if this was not included there. That way it is analytical things. Okay that's an analytical thing. I would like to give this orientation. Just I am giving you the guarantee of history optional. Where I am teaching I can give you that guarantee that you will not be thrown out of UPSC rank because of history. That's I can guarantee. See if anyone takes history optional under me your history optional marks will be very good. That's my guarantee. Only thing is you just follow what I say that's it. Other than that nothing else. I give the notes. Okay I give you the notes. So your work is almost reduced by 90 percentage. You are not going and searching big books and making the notes. I will give you the notes. And just study that notes and try to improve that writing you will get marks. This is the guarantee I can give because I have the results with me from 2013 onwards. First year my result was in 2013. And my first optional student who cleared is Hari Krishna Pai from Kerala. Is now the Kalimpong district police chief in West Bengal. Since 2013 every year I have the results. Okay and last year it was Mithin Kunai Parambal and Rishu Priya and one angel. Three results last year. Last year only three results. And this year we are waiting for that. This year only six students are going for intervention because we want students to clear the prelims. That's it. Only few students see every year there are so many students attending the class but very few are clearing the prelims. That's why the results are not that big. Three optional students if they could manage the prelims they could have scored better in the means. But very few are writing the means. This is for all the subjects okay. So no need to worry that you are going to study fact, figure, year and all nothing. Main's answer is analytical. Clearly analysis. It is pure analysis. Nothing like what we study today in the morning for that Indus Valley thing. Indus Valley we were saying about the factual things only. But main's entire thing is different. What's your opinion regarding the origin of Indus Valley civilization. There we will give the theories like that. So it is theories. Then one more thing. Many used to say that history optionally is vast and other options are not that vast. See one thing is that ancient India history syllabus is well explained. For example I will give an example. Indus Valley civilization. Indus Valley civilization. In the UPSC syllabus it is mentioned as date, explained as date, origin, urban features, trade, art and culture, religion, decline, survival and continued. This is a problem created by UPSC. Had UPSC just mentioned Indus Valley civilization, the syllabus would have been very small. Instead of that every topic of history is detailed explanation is there in the syllabus. This makes history the well explained syllabus of UPSC. It may take five pages also if you download the syllabus. But if you take some other subjects A or B, just two pages will be the syllabus. So my advice is that don't measure the syllabus with the scale. History if they just mentioned Indus Valley civilization that would have been over with the three words. Instead by explaining that they added some 10-15 words more. So every topic like that automatically the syllabus will be lost. Advantage is that they had given us the clear idea and what to study and what not to study. If it was simply Indus Valley we would have studied more. But now by making it explained we know what to study and what not to study. That's the difference of this. On the other hand some other subjects may give the syllabus like this. Political thinkers and thoughts just three words only. And you see Indus Valley civilization that is bigger. But here the political thinkers starts from Plato, Aristotle, Socrates, Machiavelli, Chanakya, Karl Marx and modern thinkers. Some 10-20 thinkers their theories thought by the time you finish this module in that option we will finish one fourth of the syllabus. So don't think that history optionally is lost. This is not that best. On the other hand literature subject. Assume that literature subject gives the mention of some books. Say for example Mahabharata. They mention Mahabharata. Only one word is the syllabus. But can you finish Mahabharata just like a one word? There are some 23 books you will study in options. Short stories, novels, poems, new poems, all the poems essays like that. Entire book you have to read and you have to comprehend everything related to that. So literature subjects is more worse than history. By the time you finish one book. Khazagine Dihazim in Malayalam or this Grish Karnat's work in Kannada or some other writers work in Tamil. By that time history option half will be over. So don't think that history optionally is lost. It's not that lost. History is lost. But UPSC history optionally is not lost. We have a well-defined syllabus. And with good faculties it's not that difficult. It's good faculties. I don't know about the faculties everywhere. But some can complicate the syllabus. Some can make it more easier. That depends on the faculties. So this is what I would like to say about history optionally. The rest is that if you are interested you can attend one or two sessions when we start history option. And this is what I personally prefer. I don't know whether I think the demo is allowed. You can ask for demo session before joining. You may attend one or two session. You attend the session and if you feel comfortable then you join. I don't want to force anyone to join the optional classes under me. If you have confidence you can join. No problem. If you are still having the lack of confidence please attend few sessions and then decide.  And if you are not able to follow that okay you leave. No problem. No guilt feeling. Nothing is there. So this is what I would like to say about the history optionally. And if you have any specific doubts you can ask. We have two timings that has actually 530 to 8 weekdays or else 9 to 1 in the weekends. These are the two times. So what is comfortable for all we will stick to that. 9 to 1 weekend means almost 8 hours per week and this is almost 12 hours or 12.5 hours per week. No much difference is there. One month may be English in the weekend class that's all. Sir I am from Trichy as I am eligible to take a course under you.  They will give you the answer. I am not sure about that. You can just ask with your sender head. You can ask for that. Maybe the regional sender is I don't know. Maybe you may have the option to join. You can just ask. Sir since the paper is static all answers will be the same. No. That's the wrong impression. This option is not at all static. It depends on the answer because once I take the class and tomorrow I make a giveaway test even the 100 students write the answer the 100 answers will be different. It depends on the individual analysis and writing. Only the points will be the same. The way of presentation will be different. That is individually and this advantage is their staticness is having one advantage. Assume that you got a question called which India movement. It's an indirect question. Assume that you are having little doubt in writing that question. But still you studied something on quit India. You won't be blind. You will be in a position to write something. Instead of getting a very good answer very good mark is 6 out of 10. Okay. You basically use 60% for the best answer. Instead of getting 6 out of 10 you can manage 5 out of 10 even in the difficult questions also under the history. But some other dynamic subjects if the question is twisted you cannot write that answer with your knowledge. So that's the advantage of history offer. Whatever asked even if you don't know that directly still you will be in a position to write something. So that is one of the advantage of the history subject. Ah that's okay. 530 class if you are option GS is getting over by that time we can shift that to 545 also. Even if you are attending see those students are attending this GS classes you will have two GS classes per day. So attending optionally evening may get make you exhausted. So it's better to prefer weekend batches because so that you can fully focus on the option on that day otherwise you will be a bit tired also. Ah earlier when it was offline I used to take weekend only. So that the students were very happy also. Last year online I took this 530 and this issue was there teacher is sometimes exhausted. Today I am a bit exhausted 11 o'clock class, 230 class, 530 class.  Sometimes you will have 11 o'clock, 230 class then attending optional you will feel tired. So it depends. Let's say once this admission formalities are finished let's have another talk when to have the class evening or the weekends we can discuss about that later. I can guarantee that history optionally safe no doubt in that. You may attend other demo sessions and you can attend other classes also. You ask for one session for attending sociology one session geography or one session history you attend that it's your freedom but personally I will allow my students because this is very crucial. Optional subject is very crucial. Don't forcefully join one option or don't think that okay my friend is joining this option let me join nothing like that. If you have confidence in your teacher you can and we have one week one entire week of class I think for this Trivandrum Bangalore's one week I am having this GS this week. So attend all the sessions you will get a feel of history. That's what I would like to see. So that's regarding my options. Any other doubts? Now those who are listening to this you may attend the GS classes and demo sessions and then take a decision. So we will see in the optional class. Okay that's all. Bye.