Actually, it's a Samsung, Samsung already. Welcome everybody. So this is a session on on OpenStack Trove In terms of what OpenStack Trove is and how do you use the product? The audience it has been set for a beginner level. So we're trying to make sure everybody can understand the basic concepts of the Trove project itself on how it's being used So it's so both Shiva and I will be presenting. So My name is Sriram. I Work at Tesora. I'm the director of implementations there. So I work with customers both in pre and post sales My name is Shiv. I'm a tech leader at Cisco. So I'm leading the team for setting up a database service at Cisco So Cisco is using Trove project for providing database as a service and we'll talk about What Trove is in general and in particular how an end user can use Trove in an enterprise scale So it gives you a sense of how it can be used So we are going to talk about Trove at a high level. We are going to talk about the the concepts And then there's going to be a tutorial session and we'll go over my sequel and mango and you know, what are the things you can do with it? One of the one of the challenges here is that this This course I have delivered it a few times that is usually around four hours long So where I provide the environment to everybody in the class and I walk them through hands-on But given the fact it's only an hour and a half. What I'm going to do is I'm going to Provide a demonstration of how the product works a tutorial But what I will do is I have with me this information in a thumb drive It has all the virtual machines a lab exercises and everything like that and people can Combine pick it up after the fact and it has everything you need step by step with full screenshots on how you do everything that we do here already and I also have a few Open-stack Trove books. It's been written by the gentleman over there told our cans. I'm standing over there So on the guy that's sitting next to him choose. So it's open-stack Trove book It's based on the open source trove and we have a few copies of it if you would like to have it Let me know and I can provide that to you And the sir go ahead on the questions. Yeah, exactly the last thing is the questions So given the fact the audience has varied it would be good to have a baseline of understanding before we delve into questions So I would request if you have questions during the session to hold them off write it down And once we go to the end of the session I will allocate ten minutes fifteen minutes for questions and we can go into topics and any questions you may have Everybody that's gone through the session would be able to appreciate that better. So what is true? So we're going to talk about the open-stack trove project. Yeah, so Trove is analogous to the RDS within Amazon web services So for the for the folks in here most if not all of you Want to set up a open-stack based cloud in your environment or somebody has set up one And you want to be able to provision and manage databases, right? So trove is a project that has been created very similar to Nova for compute Neutron for networking Glance for image management throws a project for database as a service The trove project has been incubated with an open stack in Havana And then it has been integrated formally as a project in ISO So it's been like what two years now. So for the last two years trove has been fully Incorporated and people have been using it There are many companies that have used The product of my company the tesora and then there are a handful of companies that have set up the open stack the community trove too To give a background it was started originally by rack space and HP and Subsequently, there's been a lot of contributors. So we joined the the project tesora That is we joined the project around three years ago and right now. We are the leading contributors to the trove project Other than to sort of we have HP IBM Rantas eBay Red Hat that have all been contributing to the trove project Like I said earlier trove provides databases a service. So using trove You can provision databases and you can manage the lifecycle of the databases that you have been provisioned One of the common misconception is oh databases a service doesn't mean I can provision databases Why would I need the trove project? I have chef. I have puppet can I not use those and Before trove came into being people have been using those scripting tools to create databases because there was no formal functionality to do that Now the trove is there. So using trove. You can create a database. That's great But not only that you can manage the complete lifecycle of the database So you can once the database is provisioned all the carrying and feeding of that in terms of creating backups in terms of scaling up scaling down in Terms of setting up multiple databases like for example master slave replication failover or database clustering Multi-data center support. There are a lot of things that you want to do that are pretty complex If you are going to try to do it yourself and time-consuming using a project like trove It's push button if you use the horizon dashboard if you like to use the CLI You run a command on the on the screen and you can do that and the tutorial that I'll be showing you We're going through some of some of those Trove project has been used by organizations both in a self-service mode where you provide Your users the ability to go in and create their own databases and manage a complete lifecycle and in some cases operators have been using trove in their existing mechanisms with the idea that when a user makes a request They can serve their request very fast. So either approach is available and then the last but not the least Compared to Amazon RDS and DynamoDB using trove you can manage Relational or non-relational in the same way. You can go to the dashboard and you have access to all the data source that you have made it Available and you can create them in the same way so if you talk about SQL and no-SQL databases, this gives a list of all the databases that are available today and The ones that are coming soon So the open-stack trove project has got pretty much the community Databases like the MySQL Postgres Cassandra, Mongo community, couch-based community and things like that and when we work with customers They want supported databases. So we have taken the effort to support for example MySQL enterprise or the data slacks enterprise, which is the Cassandra version as well as Oracle and things like that So today if you look at trove both the community trove as well as the kessler adib as platform If you take the two together, these are the databases on column one that are being supported and Pretty soon you have Vertica couch-based enterprise, Mongo enterprise and those are available too. So Okay fantastic so Obviously the community has been contributing to the databases some of the original databases like Mongo, MySQL have been done by individual organizations that wanted support for them And then they made it available upstream But now what is happening is that both from a community perspective as well as enterprises like Tessora The larger user community have been asking for new databases or New functionality on a database and this that is driving it. So as you look at adopting trove for your organizational needs You would find most if not all the functionality that you want in a day-to-day basis available in trove So like I said earlier Trove doesn't just do provisioning it manages the complete life cycle management. Should you want to just talk about that? Sure. Yeah as Seema was explaining Trove not only lets you provision a database, but it helps you manage the entire lifecycle of a database So and also it gives a API which is works similar across all databases whether it's a SQL database and no SQL database a create database API is exactly the same So that's the best part of a trove. So if you look at provisioning it lets you create a database a single instance or even you can set up a cluster and For managing the database it lets you set up a replica set, which is a master's level relationship Or it lets you take backups regular backups and restore a database and also it helps you scale up or scale down the storage or the memory of CPU Depending on your needs. So as your needs close, you should be able to scale up the Create on the database as you need it and also By default the database what's installed is tuned to the setup what you have But if you need to fine-tune it for you for your own needs Trove gives an option to set up configuration groups, which will Help you tune a database for your own needs if you need to and finally Securing a database like if there's a OS patch or a database patch, which comes in through It all gives a mechanism for you to upgrade and patch your databases using the images, which that sort of provides For any critical patches as you go along So just wanted to quickly go over the trove terminologies, which we'll be using to the rest of the session Guest image is the image what you get from tesoro, which contains both the worst and the database Installed so it it will show in glance and when you probably did is this image is used to Create your VM guest agent is the API which acts helps us Supply the create a barrier between the individual data types and The API is the APS as I said earlier It's exactly same if you try to set up a mysql a postgres But the guest agent is one which will translate into your specific data datastore type or data type what you're using database type Trove instance is a nothing but a database instance which contains the guest image and the guest agent cluster is a set up for cluster databases, which you can set up for example MongoDB cluster or mysql coconut cluster Datastore is a data type basically. It's mysql postgres That's what you call it as a datastore data show version is a version of the database like mysql 5 6 or postgres 9 4 configuration group as it Expand earlier helps you fine tune your Configuration for your database. So this is where you set up. We'll show it later We'll talk a bit more later and flavor is basically where you pick your What you need for your compute memory or storage. So this is where you set up the pick up the flavor That's what we call it as a flavor. So it translates into the NOAA and the Storage in the back end Thanks. So these are all terminology that will be using throughout the session to talk about a guest image turned into a database instance spinning up databases based on datastore datastore versions Fine tuning your instances based on configuration groups that we can create and then scaling up vertically based on flavors that you want to use So how do you how do you create a database using true? So basically there are certain things that are needed to spin up a database instance using true So we just talked about flavor the kind of machine that you need so the flavor support is something that you definitely need and Then when you create a database the database itself the data in a database is stored in a in a block storage format So you would need center volume support for that On in addition to that What true offers is that not only you can you create a database instance? But for those databases that support the paradigm as part of the database creation You can actually create databases and users So let's say that you as a user or someone in your organization wants to manage access to databases So when you say hey, I want a new database they can create a new instance You can say M1 large as an example They can spin up a database in an M1 large at the same time They can create a database or the set of databases and create users that will manage those databases So when you get back a database instance, you know for a fact, it's got the databases on the user So you can turn around immediately login as those users and start accessing those database instances And then configurations we just we just talked about it configurations are a way to find you a new database set of parameters that you want and apply to one or more databases at the same time and manage that and The and the last thing is that not only can you create databases? We talked about the life cycle management So when you take a backup you can use true to restore the backup into additional instances So you have number of instances sharing the same Schema and users and the like So these are some other things that you can do as part of Proceaning a database and using the open stack functionality Now that we talked about at a very very high level about what true was the background and the kind of things that you can do With it and the terminology what I'm going to do is I'm going to actually go in into a tutorial Like I said, normally I would provide you these these thumb drives And you'll set up set them up in your laptop and you do them yourself But right now because of the time constraints I'm going to walk you through that like I said earlier for those folks that joined in late. I have Many more thumb drives if you want to try it out yourself come talk to us. We'll be happy to share them with you Very can go try them in your But by yourself So for the tutorial for today, what you're going to do is we are going to start with my sequel and go into Mongo, so we'll create a my sequel database We'll show you how to create backups how to restore from the backups how to scale up scale down How to fine-tune the my sequel databases that you created using the tuning parameters that you want to use and Then we'll then go ahead and create a Mongo database So you have a sense of using the same dashboard you have a combination of sequel and no sequel databases all living together happily Yeah So so for those folks that are interested so as part of the thumb drive you get a a true virtual machine a data Dev stack virtual machine you get some sample scripts for Mongo on my sequel in addition you also get The coursework on the lab document and a copy of the PowerPoint deck that I'm just that I'm just giving you right now. Yeah So moving on to the demo So for this demo, even though we are focusing on the open stack trove for this demo I'm using the tesoro DBS platform because we provide Database guest images and we have a bit more functionality through horizon dashboard and it's easier to easier to use there So so this is the same horizon dashboard that you get with open stack except it's got more functionality available for the database tabs So I'm logging in as the admin user and this is a standard tab You would see the only difference is you will see the tesoro screen skin there So so the main thing that everybody would be using is the is the database tab under project So if you if you look at it, you see databases clusters backups data stores and configuration groups so under under clusters you'll see the clusters under Backups you will see any backups that you have taken and then the data stores will show the The data store types she was saying earlier the database type. So for example, if you look at it, you see Mongo my sequel and Oracle in there And it basically goes to show that in the current setup that's there if you want to create databases and manage them You can spin up my sequel databases We are going to do that. You can spin up MongoDB databases MongoDB clusters and you can also spin up Oracle 12 one Oracle 11 databases and the configuration groups is the one like I said earlier to tune your database in addition to that tro uses The open stack functionality to orchestrate and to attain its goals So trove doesn't recreate all the functionality from scratch So if you look at it tro uses Nova instances tro uses cinder volumes tro uses images from glance So so when you are spinning up instances if you want to know the status of The Nova instances are the cinder volumes. You can click at the the compute section and you can look at The overview which shows the current status of your your quota and the like Instances show your Nova instances and the cinder volumes show you're the the cinder volumes for the databases that have been created So if you want to create a database using the command line So you would you would go to you'd go to the instances tab and you will go click on launch instance and you can create it but a number of the folks in the audience here are Probably comfortable Doing things using command line interface and you want to get your hands dirty so to speak So what I'm trying to do is I'm trying to show certain things that you can do in the coming obviously everything that The tro project is a full a first-class open stack project So everything that you can do in tro you can do using horizon Using CLI and if you really want to get your hands dirty you can access the rest API, right? So I'm trying to do certain things using CLI and certain things doing the horizon So you get a sense of the kind of things that you can do so starting with let's go use the the CLI So before you do that You can just see that for the folks that are not familiar with it the two tabs at the top are the user and then the And then the tenant tabs Moving on to the CLI So what we're going to do is we are going to create database instances and create backups using that So before we do that if at any time you want to say, okay, I want to run a command in tro What are my options always you can always do tro help, right? It gives you a list of all the commands that are available along with a simple description If you want additional details, you can actually specify the name of the command So here are all the commands that are available today and Alexa earlier We are adding more both tesora as well as the the trove community. We're adding more functionality to throw now if you say that you want to Create a database Then you can always say trove help create and it gives you all the options Along with the values for those options. So for example, if you look at If you look at trove help create You can specify the size of the database The volume type so you can you can specify in cinder Different volume types. So for example, you may say that depending on your needs You may have for dev test a regular spinning disk base volume type and for For staging or QA you may want regular SSDs and for production for certain use cases You may want high IOPS SSDs. So you can set up multiple volume types and you can specify through tro When you create a database what what what volume type you need now? Obviously, we are seeing them as arguments in a command line and when you create a database using the dashboard They will be available to you as drop-downs and you can create using those options, right? in addition to that we can also Create a new instance using a backup as an option. You can say I already have a backup It's got my seed data It's a bunch of users. I want to start with that rather than create from scratch You can pass in a backup. That's available and if you want to create a database instance in a particular availability zone You can specify the availability zone and then obviously you can choose the data store and data store version because that let's you choose Oracle 11 versus Oracle 12 as an example or my sequel 56 versus my sequel 57 or my sequel 55 and The last I want to talk about is the fact that replica off. So if you want to create a master slave Replication and you want to say I have a master and I want to create slaves based on the master You can pass in the parameter and they can say I want to create a new instance based on this master And you can use a flag for that. So so let's go see it in action. Shall we? So if we do a data store list you get a list of data stores Which is in this case mango my sequel an Oracle, which is what we saw there Then if you can say okay, what versions are available for my sequel you can say 5686 and it shows it is my sequel 5.6 86 is the true version. It's not related to the my sequel version. So it's true related version of the My sequel what are you using? Thank you. Sure. So This is one of the new advanced features that trove supports So a number of you that have databases in operation would say hey, you know what I have databases there and it is It my sequel or a mongo or Oracle or whatever the database that is it's is is running on a on a Ubuntu or a rel environment and It has so you are saying okay Every so often the right you guys are after you saying you need to apply the latest You know Ubuntu or rel patches security patches database patches as well as Bugs or features or enhancements that are happening with the the trove software itself. How do you keep track of all of that? So what we have done is is is used that the dash 86, right? So there is a library of images that are created and each image contains the corresponding Buck fixes security patches from the from the vendors Along with the database vendors and from trove itself. So you can take the latest Image, let's say you were using 5685 and you want the fixes available in 5686 You can grab 586 and they can replace 85 with 86 in one single operation. So you have your data You don't touch the data the only the image is changed so automatically in one single shot You know you got the the Linux fixes security patches bug fixes the database related ones and the trove related ones You don't have to worry about any more of applying individual patches and making sure all your database if you have 500 database instances running in a big server form. You don't have to worry about Did I apply that fix to every single one of them, right?   So so that is that Thanks, you were for bringing it up So this is the command For for for trove. I cheated. I'm sorry. I just wrote it down earlier so that I don't type in on the screen by mistake So basically quickly let's go through that. So basically the command is trove create and You provide a name for the instance. I am name challenged. So I chose my sequel one and then And then and then the flavor is 101 and then if you do a nova for the folks that are familiar with open stack You can do a nova flavor list or trove flavor list and you get the flavor along with the flavor ID You pass in the flavor ID in this case. I chose 101 and then you want to see the size of the database itself So I could choose to create if my sequel database with a size of two gigs Then obviously I'm saying that I want my sequel data store And by the way, I want the version 5686 and then in addition to that like I said earlier I said earlier when you create a database instance. Hey, you can create databases and users So in this case, I create a database called database one and a user called user one of the password password So I ran the command. It shows that the command ran So then if you want to look at the status, you can just do a trove list and the trove list shows that it's in It's in a build status Now given the fact it's video now keep in mind when you spin up databases It takes a few minutes for the databases to come up, but I don't want you to wait for that long That's why as a recording as a video I can Fast forward to the completion, right? So now we can see that the database is active if you do a trove show on the instance it shows the fact that You know the instance is there. It's got an IP address It's active. It shows when it was updated on the volume and things like that Given the fact it's a my sequel database, you know the IP address, you know the user, you know the password you can always connect to it using that IP address and User name and password you can connect to it. If you remember correctly, we saw the database earlier We created it called database one So you see that that particular database is there the database one And if you then use the database one and you want to see if it's got any data in there You can see that it's basically an empty database. So what trove did is trove created an instance It created a database a mysql database in the instance Created a user to manage that so from your perspective Either for yourself or somebody in your team. You can basically tell trove to create a database Instance with the instance create one or more databases users and then you can provide the endpoint your users And your users can start using that it's very straightforward You don't have to you don't have to spin up Nova instances SSH into them and manually download the software and make sure you have all the patches You can just in one one shot as long as your organization has got trove installed and configured in one shot You can create instance and start using that so One other take it later Yeah, the question. Oh, okay. Yeah, so if you could just keep your questions later Then once we cover it we can go through all the questions. Yeah So like I said earlier the purpose of trove is not spinning up instances. It is also the life cycle management, right? so just I want to quickly go into the UI and show the fact that When you create a trove database instance you also create additional things So three things happen at high-level as you're learning about trove When you create a database you need an environment for the database to run in And you need a place for the data to get stored Like I said earlier when you create a database the data for the database gets stored in cinder volumes So they would be available under volumes and then when you create a database instance You need a machine so to speak a virtual machine or physical machine if you're using Ironic For the database to run in and then those machines are in in your Nova instances, right? so what I wanted to quickly show is that when you when you look at it you see The the flavor trove medium which was one-on-one there and also the cinder volume we specified two gigs And you see that two gigs have been provisioned So if at any time you want to know the status of your Nova instance and the volume you can always check it there Now that we have created a database instance and you're starting to use that Let's see what other things you can do with that if you want to create a backup is pretty straightforward All you would need to do is run the backup create command specify the instance and name of the backup. So in this case we said My sequel one and the name of the backup is backup one So then the backup is created you can just do a trove backup list and it would show the backup has been created And and the status is completed So now obviously you can't you can have full backups like we did right now. You can also have incremental backups And and you can use them that way so now if you look at the screen the same thing that we did in the command line you can you can see it in action and If you have backups and if for example your team is creating some database and you're creating a backup And then you want someone else within your organization to use that backup to do other things Then you can very easily do that. So to restore from a backup. That's what you would do, right? You'll go to restore backup you get the same screen as You would now keep in mind we created a Trove instance initially using the command line. I wanted you to get a sense of how would you use command use command line To create and manage instances, right? That's what we said now most of the indie users and large organizations that want to use Trove databases the service will not be probably using command line some developers. Yes a lot of the indie users once it goes live We'll be using the dashboard. I want to show to you what the experience will look like So from now on for the most part will be using the dashboard and Obviously like I said earlier every single thing that we do in a dashboard can be done in a command line And if you don't know what the command is just do a trove help I'll show you all the commands are the options So now we are going to restore restore the backup and we are going to but keep in mind if a backup was created with two gigs You don't have to restore it to two gigs, right? You can say I want a bigger database You can create 10 gig database and restore the backup, right? So those are that's why you're given the options there So you can say I'm going to create a instance from the backup and I'm going to call it I Believe guest one Once I start typing in my in my recording there you go Yes one and then for right now I did everything on my laptop. So slightly resource constraints. I still stick with two gigs and then I choose my my database type the data store choose my flavor and If you had different volume types like a spinning disk SSDs high-up SSDs Then you can pick up from the drop-down and then you can choose Which availability zone you want to go into so now one of the things Locality was exactly locality is basically if you want anti affinity for your databases where you don't want the slaves on this And the master to be on the same host You could say anti affinity and then it makes it tries to provision the it provisions the slave on a different post that way you're sure that If it was goes down, there are anti cluster doesn't go down Exactly, so you can have multiple databases across different hypervisors and if one of them goes down the other ones are still available and Then here we are going this is interesting, right? So we are going to we are going to say by the way in the advanced tab We are going to restore from a backup and you have a number of backups You can pick and choose the backup that you want and in this case. I've choose the backup I just created and then when you hit launch what is going to happen is You are going to with the values you set on the first screen using the backup you're going to create a new instance So now this would probably take around three minutes three and a half minutes or four minutes on the laptop But for the purpose of this we are going to fast forward So build is happening now the build is complete. It's pretty fast So you can see that it's a my sequel five six with it's got an IP address and then you can see that it is True medium with the giga RAM and if you and if you click on the instance name You can see that it's got IP address and if you look on the users on the databases It is the same user and the same database that we created on the first instance because we took a backup from there And we're restoring using that backups obviously. It's the same thing and the other thing that is important Is that every time you create a database instance whether it is my sequel Cassandra mango couch base? Postgres Oracle, whatever it is Each database has got a connection string to the endpoint So what we provide as part of that what rather what true provides is when you click on the details screen You see the connection in points all a user has to do is grab that and then copy paste into their client And then provide that we don't show the password right probably the password in and you can start connecting to the endpoint And then there are a bunch of other things you can do through the screen We talked about scaling up and scaling down so if you want to scale up you can say Resize instance resets volume attach configuration group if you want to manage root access you can Right to basically to increase the size of the volume initially you have a database of two gigs You say hey, it's pretty pretty successful. There's more data there. I want to grow that I'm gonna go from two to three gigs So you just change the value to the one you want Now it's three gigs from from two if you say hey, you know what once again I need a lot of activity going on. I want more CPU more memory. You can go from troll medium to Something else in this because I'm going to go with troll medium resize just to show that It's very easy to use the dashboard to basically resize the flavor once again All of these things are done in command line to but you can just do that using dashboard by clicking and dropping down right, so It's very easy to create an instance to create a backup to restore the backup resize instance resize volume to do all of that and Then we and then we talked about Configuration groups, right? So what are configuration groups? So initially? The database images that are available are tuned for your needs, but let's say that You want Database you say, you know what based on the work that your DBS have done You want to tune them to a particular values for the different parameters because that's what works for your organization Lot of companies have that need so how do you do that?  Have the same template and by the way After six months if your DBS want to tune it further you go to one central point you change it once all the databases get updated automatically How do you do that using configuration groups? So what you can do is we are doing configuration group for my sequel right now, obviously But the same thing applies to all database types or as we call it and throw all data stores So you can get a configuration group we create one for my sequel So once you have a configuration group, it's it's an empty shell It doesn't have anything in there what you can do is you can basically add parameters to the configuration group and If you want to know hey, I don't know all the parameters that are available for a given data store all you would need to do is click on add parameter and Throw will give you a listing of everything that is available for that particular data store But keep in mind these are the ones that are available for my sequel if you're using Mongo or Postgres or Cassandra It would be different parameters obviously you can pick and choose the parameter you want you can add the value a Configuration group by its name Indicate it's a grouping of a number of parameters right instead of just one you can have a group that's got I don't know How many ever you want and when you apply it all the values get applied? so if you want to know One of the things is if you are going to change the value of a given parameter you want to know What was there to begin with right? So we provide you that too So if you go to this database the one that we got from a backup you click on the defaults it shows all the parameters available on what the default value for those parameters are So you can say hey, okay in this case. I'm gonna in this example I'm going to use max connections the default value is 250 and let's say that for your purposes You wanted to go to 260 As an example, so okay. You say that that's what I want to do So what you do is you go to the configuration group you add parameter and Then you choose the one that you want in this case is max connections and You and you enter the value and then add parameter So what you have is you have a configuration group with a particular parameter If you want to add additional parameters, you can click on add parameter till you have done all the ones you want once you're done, you can just click on apply changes and the configuration group is created with that parameter and Then if one apply to instances you can basically go to the instance that you want you you can say add attach configuration group and Then and then what you do is you attach a configuration group and then there you go so the interesting thing is if this parameter Would necessitate a database server restart We would automatically flag that and say data server restart and you'll have to restart before you can access the instance using The trove dashboard will we take care of that too So if you want so right now what we did is as I was talking we attach the configuration group to the instance So if you want to know that hey has it changed From 250 to 261 easy way to do that is go back to you in this case my sequel, right? It means a my sequel client and you can go back in and see if it's changed So basically what I'm doing is I'm getting the IP address by using trove show guest one. It shows it's running on 172 16 213 So I connected that instance using I believe the user is user one with the password password So I connected that instance So then obviously in the case of my sequel you have a number of variables and the variables are those different parameters That we saw on the screen and they have those values So I'm going to look at the values for the variables because the parameter that it shows was max connections, right? So we're going to look at the variables max connections Okay, so now it was by default. It was 250 We change it to 260 if you want to change it further You can go to the configuration group You can change it in one place and you can apply it to any number of instances and they will automatically change at the same time So that's what configuration group does if you want to detach it you can just detach that So that's how you can very easily create and manage and this is self-explanatory If you want to delete an instance all you would need to do is click on the checkboxes to the instance click on termited instances and It gets deleted. I think one thing to keep in mind is that For every true instance you have a corresponding NOVA instance and a center volume So when you create delete instance the first thing that happens is the instance goes into a shutdown state So obviously you can't do anything to that While it is in the shutdown state It goes behind the scenes and it deletes the NOVA instance clears it out gets that is gets the resources back and clears to the center volume So when you delete an instance the the flavor instance is gone as well as As well as the volume, but the one thing is any backups that you take They're still around so you can create an instance take a backup And you can delete the instance because you're because you no longer want to pay for it But the backups are still there if you want to create a new instance in the future The same backup, you know what you can do that on the one additional thing We are going to show is how now that we talked about Relational databases my sequel the kind of things that you can do do it now There are additional things you can do you can create a master slave replication We can do fail over all through the command line or the screen or you can create Percon extra to be clusters or Maria to be clusters or things like that What do you want to do is I want to show you how to create a Mongo instance and Show that using the same dashboard for your users. You can have any number of different kinds of databases available More importantly what I want to show the fact is that True it abstracts the way in which you deal with it So the way you in which you create the my sequel database to a Mongo database is exactly the same So once you train your users how to use it all they would need to do is pick and choose the database They want and then off they go and they want to take a backup exactly the same way They want to create a replication same way cluster same way So in this case, I entered the values for the Mongo instance launch And I have the Mongo instance so now I can see that in my dashboard Obviously for my tenant the admin tenant. I have a my sequel database and a Mongo database I wanted to do this so that you have a sense. Normally. This is something we'll all be doing it together and In the thumb drive you have instructions along with the virtual machines that you can start that have the My sequel and the Mongo database loaded so you can go and do everything that we did and more in that, right? But given that given the fact you only have an hour and a half No, I can if you want I can I can just provide that to you not a problem This is this is this pretty big. It'll take you long time just to copy it. So I'll provide that to you No, I can I can collect the information I can give it to you not a problem not a problem at all So for those folks that want this let me know I can I have a bunch right now with me And if if you run out I can send it to you no issue But my goal is for you to try it out and touch and feel and see how it works So One of the goals I have is to kind of walk you through introduce you to throw how it works before going into the architecture and Some sample ways in which you can deploy. I wanted you to have a sense of how it works Then when you talk about architecture would make more sense to you, right? So in this section, we are going to talk about how trove itself is put together under the covers And we are going to talk about Cisco intercloud and see how they have deployed trove in a scalable Highly available manner. So you have a sense of how it will actually be deployed in enterprise manner, right? So in a thumb drive running a single instance is one thing, but can you really deploy it in an enterprise way? Sure So this is the trove platform architecture So for those folks that are familiar with open stack, you can see a lot of similarities, right? The open stack projects in a lot of cases are designed similarly not the same way, but fairly similarly if you I guess the next one Hola, I think there is a lag here. I need help from the back. So, okay, you can see that Maybe this will resolve it Fantastic We are back in business So this is the the the trove platform architecture It says the Tesoro D best platform architectures because it's one of our marketing slides But this applies to standard trove too. So if you look at it, there are two boxes, right? This is the open stack box here It's got all the open stack services like I said earlier trove does not recreate anything from scratch It does not read, you know re recreate the wheel if you will it uses all the open stack structure It is a core in the sense core as in it is one of the major open stack projects So it interacts with every open stack project. So for example, if you look at Nova If you look at Cinder, Swift, Glance, Neutron, Keystone, they're all there and if you look at the trove box You see trove there. So there is a concept of when you look at trove think of three things a trove controller a metadata database and a message queue is very similar to what open stack would do So the trove controller is the brains behind the operations So the trove controller consists of three parts the API service The task manager and the conductor the API service is the one that you would always interact with to begin with right? So you would say trove create trove backup create trove resize instance trove resize volumes That would go whether you do it through the UI or through command line You would go talk to the trove API and say go do this and trove would do that and you'd get the request from trove Trove would update the message bus and say go do this There's a task manager service that's running and what that would do with that get the message from message queue It would say can now let me go talk to Nova and See first of all, let me go talk to you Keystone and validate that the user can do this Okay, user can do that What tenant is the user in admin tenant does the tenant have enough resources to do this operation if you do all of those things? And if it fails you'll come back and give a error message and then you could go to Nova spin up a Nova instance Go to Cinder check to see if you have enough volume available create a Cinder volume and Go to glance grab the image put it into the instance go to go to neutron Get an IP address make it available to that, right? And if your command was trove backup create it would go to the Cinder volume read the data Take a backup and then apply the backup and store it in object storage, which is swift We also support safe whether it's swift or safe. It would store it in there And then when all those open stack services are running and you have the trove guest agent there Remember we talked earlier about the guest image and guest agent so in the Nova in the Nova instance is the Is the guest image based instance running and it has a guest agent on any call you make goes to the guest agent? And depending on the data store type the call gets translated right because Oracle would do backup differently than Mongo then my sequel so that's what that would do and Then it would take you take that create a backup from Cinder volume store that in swift as an object storage the last thing is Any messages that are coming from different open stack services are from the guest agent goes through the trove conductor But the net net is for the purpose of today. You have a trove conductor, which is the combination of API task manager and conductor and we are going to talk about in a couple of slides about how to deploy it in an HA fashion and You can think of those when you talk about controller and the other thing are the message bus and on the metadata database You have two options You can use the main open stack metadata database or you can use Your own metadata database on the same thing with Rabbit MQR Cupid if you are using if you want to use the main one for your open stack controller You can use it or you can create your own one what we see From Customers and audience in general is that they don't want to modify or touch the core open stack message QR database So what we ended up? Supporting is you can have your own copy just for throw a mysql database a clustered format because you want HA For throw and correspondingly a rabbit MQ just for that and you can use it just for throw so that way everybody's happy So now she was going to talk about how Cisco Intercloud is using throw and how they have set it up and how they've done an HA fashion So this will give you a sense of it's all good to create single instances and talk about how throw components are But how do people use it in a real in the in the real world example? Just wanted to take a few minutes just to explain what Cisco intercloud is If you look at the current scenario you have public clouds providers like Amazon Rackspace and your private cloud providers like Microsoft But there's not in there's no very little interoperability between these two between the between the clouds This is where Cisco wants to come in Cisco wants to build intercloud where Cisco is hosting the cloud He knows and it is used internally as well as by our external customers And also we host and manage clouds for our partners who in turn can sell it to the customers And also we also help manage and set up clouds at the customer insider and manage it as well so and We give a mechanism for you to move your cloud from either the private to hosted or the public in a seamless manner You can move your best on the application needs you can Provision a VM anywhere. That's what Cisco is trying to do and Coming to how we have deployed probably Cisco Basically, this is in a single data center. We have a service cloud, which is where all the management VMs are there and and then we have a tenant cloud where the tenant VMs gets provisioned That's our service cloud is managed by the cloud operators within Cisco who control and manage the service cloud But if you look at the tenant cloud is totally for the tenants So the service the cloud operator Can look at it, but he doesn't have access to the Tent VMs that way. We are securing the tenants data so coming Taking a look at it for the trial we have it Troll controller and the DB for the metadata. We have a partner killer cluster and we have Created the rabbit I'm kids own telling DMC just for security purposes because that's the only piece which communicates between the tenant cloud And the service cloud. That's why we have secured in it. It's on DMZ and all the communications that happen So three SSL just to be clear, right? So we talked about the architecture piece earlier in the previous slide that Architecture slide would comprise a service cloud, right? It has all the open stack the trove services and that will service cloud I'm from a tenant cloud is actual Instances that an end user would spin up all the databases So that would be the tenant cloud, right? So they will be dealing with the actual databases and the instances on the operator would be dealing with all the Services and making sure the services are up and providing all the necessary functionality If you look at the tenant a that's his own network the tenant a is a the VM security needs in his own tenant network So no only the tenant has access to the database itself and the service cloud provider doesn't have access to the database Coming to like if you're doing in production, you need a day So we have built a cha across all the trouble areas So we have created the the trough controllers behind a load balancer. So that way you can assist through HAProxy The trough APIs and the trough CLA comments are even the husband. You are communicates to the load balancer and for the Back in my metadata database. We are using Parkano gallery cluster Master master in a master master mode again, it's behind a load balancer and for rabbit MQ We're using the rabbit MQ cluster and we are configuring The trough.conf to do to connect to the rabbit MQ on a round robin basis So even if any of the VMs goes down, the service is not affected here so if for example if a trough control goes down, you can spin off another new VM and Restore restore it and then how to put it into the pool and yeah, it's seamless sort of things. So there's I will be T All across the layers So do you have time for the yeah, absolutely? Absolutely. I think we are slightly ahead of schedule. So what I want to do is there is a Slide we had kept our regarding monitoring So we want to just kind of go through that too. So just one more thing means Modeling is also a very important piece in productions. Like how do you manage your? Yeah, you set up your production environment But you need to monitor it and make sure that the service is 24 by 7. It's running 24 by 7 So as I said in the service cloud, we have the hook stack which takes care of like the cabana elastic search where we Which my which does the basic we are monitoring for the service called VMs That's the draw VMs the row controller rabbit MQ and that metadata database and also we correct all the logs from the message queue Monitoring blocks and push it into cabana as well. That way if there are any anomalies It we can get us created and get paged so that a cloud admin can go and take a look at it Why the services in the grid at state and take appropriate action Similarly on that as I said since we are not managing the database itself on the For the customers if customer wants to look at a source for example if wants to get alerted if there's a slow query running on the database So what we have done is basically we have set up the basic we are monitoring it as well as the Mysql mysql Sorry, that's okay. Let's go back. Yeah This one and then you could push it to we are pushing it to Cisco Zeus all the logs and the metadata But it's a monitoring as service But you could use your own monitoring as service to capture the logs and alert goes to the tenant admin who is managing the Database VM on the tenant side. So this is how we have done the monitoring Yeah, I think I think this is very important. Thanks Shiva. I think if you take a step back and look at it, right? So if you are responsible for making sure your organization You have a scalable stable Databases of service you want to do two things you want to make sure it is highly available So there's no single point of failure whether it is a metadata database goes down Rabbit MQ goes down or if one of the trove services go down because they're all behind HAProxy and load balancer if one of them goes down you can obviously spin up another one put it in the place Load balancer sees it available and continues to use it. So there is no single point of failure That's number one, and I believe you know you could start with three or four and you can obviously Depending on the core of we can add more and more to have it really scalable The other thing is it's good to have scalable solution highly available But you also want to make sure it actually works well and it performs well for an individual database We talked about configuration groups and how we can tune it with parameters and things like that But for your services, you want to make sure you know it is not being degraded memory ways or whatever it is and it's performing well So that way both for that service cloud where the operators are there They want to make sure all the different row services the metadata database and the rabbit MQ They're all performing to the peak efficiency and you can monitor through that on from an end user tenant They want to make sure like she was giving an example of a slow query, right? If you have a mic my sequel slow query, and if it's taking a long time You want to know that so you are your DBA for your organization can go back and look at it and see what is going on, right? So is it because the parameters used to set up the my sequel itself using the configuration groups are incorrect They need to be tuned or is there something else going on? Is there a lot of IO? At the instances do you want to go back and scale up? Maybe whatever it is, right? You want to do that so from the end user perspective You have those monitoring so between the two monitoring both from the from the trove controller perspective as well as the end user perspective You have monitoring for both. So this is pretty important. So when you are thinking about setting up something I mean all of you come from different companies and you all have different Maturity level for your open stack and trove. So this should hopefully give you some ideas on what it is that you want to do To set up something that you can actually use and people will be able to get value You know one of the things I said is let's keep the questions till the end because we want to make sure we have There are people with different background levels. We want to get a standard level before we talk in the question So we have we have plenty of time. So, you know questions are welcome right now Yeah, good question. So you can you can do networking either using neutron or no one networking So given the fact that's running my laptop. I chose no one networking So when you use no one networking that gets assigned automatically, but if I were to use neutron I would go to the networking tab and assign a nick and do it that way So the flavor contains good questions the flavor contains all three, right? It contains CPU memory and the and the disks but keep in mind It is a disk for the instance, right? It is not that disk for the database So you have a decouple system you have the nova instance that contains the software on software alone But the data is stored in a persistent state in a center volume So even if the nova instance goes down your center volume is still available. I Changed the flavor on the instance I'm sorry. I yeah, so when you change the flavor it is for database instance, right? Yeah, exactly But not that but this will be clear if the database has got even though the instance might have let's say 20 gigs of space The database size may not change because that is being maintained in the center volume. Hope that makes sense any other questions Yeah Sure, so I think I think you can do one of two things, right? I think like I said earlier other than some of the enterprise Databases other than that most of the databases are contributed by the community We enhanced the database features from Tessera, but most of us in the community So if you want to participate in there Doug Shelley who's a VP of engineering who's one of the co-authors of this book he can give you Tips on he's smiling widely. So hey Doug one more member to your team You know, you can contribute start the process and the community will work with you or Now I'm plugging in from a company if you want you can come talk to us We'll we'll be happy to help you out either way other questions, please Yeah, absolutely, absolutely. It's a very good question. So as I tried to communicate Trove uses the no abstraction. Oh Oh, thank you. Thank you. So the question is we would I was demonstrating trove through virtual machines The question was can trove be used with bad metal or containers, right? I got the question, right? Yeah So trove uses Nova for abstraction So right now most of the open stack instances For the most part use KVM and you get spin up VMs, but you know You have you have heard of the ironic project, right? Which is the bad metal project? So if your open stack deployment Has got the ironic available and when you go to Nova and say Have the ironic option. So when you create a new database, you can do it on bad metal now Containers is a slightly different thing. So the support for containers is evolving So we expect the containers support is not as much as bad metal or VM support It's evolving. I think initially We had the Nova Docker project a while ago had that been around and if Nova had Virtual machine bad metal Docker support would have been easier seamlessly to do all of them But now Magnum project is there. So it's it's it's work in progress. Hey, Doug You have any latest update on containers or whatever I told is still true So just to relay what Doug was saying. There is a it is still work in progress There's a session this week to kind of talk about it. So I'm assuming it's going to be a good three to four or five months before We have something that the audience at large can start using More questions Yes, sir Yes So it's a good question. You're talking V3 API, right? Hey, Doug. So the open stack API will it support true or other the other way around This is the open stack API and not the trove API Open-stack common API valid will be be other ring to like the V3 Yeah, yeah It doesn't order. Yeah, it's working progress but I think My my guess is since all the other projects are going through the open stack based single API I think it's a matter of time before we'll support it, but I think designed to be discussions to be had Yes matter programming It's easy for me to wear my hands from here Doug absolutely Matter of programming is Yes, sir It's not single tenant if you look at the way we have implemented in Cisco The tenant a tenant B was multiple tenants only the service cloud where we had one single tenant But the tenant and beans multiple tenants. So you could support multi tenants in the existing job Yeah, that is that is one example on keep in mind when you talk about multi tenant It is a loaded term. So are there is that a specific use case you're looking at when you say single tenant multi tenant Yeah Okay, if you're if you're if you're if you're doing that on the on it Okay, so couple of different ways in which you can do that, right? So in in one of the screenshots I showed earlier Once you create a database instance. Oh, this is on the on the backup screen once you create a database You scroll down you see the endpoint, right? So once a database is created the database endpoint as long as In your case, let's take there are two tenants If those two tenants have the network ability to access that particular IP address on which the database is running So like we there's a question regarding where do you choose the network, right? So if you chose a network for the database to run on that both turns could access somebody can spin up a database and The two tenants can access it The two minutes of two tenants may not be able to control the database like take backup and failover and things like that But they can access the instance. That's one option The other option is we provide multi-tenant support through Oracle 12c and things like that and That that is available to you So if you want to take advantage of physical resources and things like that you can do that what you said holds true What I said about accessing the endpoint holds true There are many ways in which you can do that if you have if you want to have a more in-depth conversation We can talk about your use case Sure other questions, please It's a very good question. So today as of today, I didn't have a chance. I thought I'll be running over I usually run over. I want to make sure there's enough time for questions. I didn't show that today You can do a failover that's push button failover But we are soon coming up with auto failover because people have been asking for that. Absolutely other questions, please So are there things obviously I did not show clusters. I did not show Um Replication because I was thinking that I may not have enough time Are there things that that you want to see that that I didn't have a chance to so that you are interested in either clusters or cluster scaling or failover something like that that you would like to have seen today the session That you came in thinking you're going to get at it that you may not have or Whatever whatever given the background that you had Whatever we you saw today was a good start for you even even that's a fine answer Head shaking would be good or hand-raising would be good too Thank you. Thank you. Thank you. Thank you Any other questions? Yes, sir Yeah So can you repeat the second part of the questions you said I mean I got the pond about commercial databases multiple tenants multiple VMs Yep, Oracle is is a different beast all by itself, right? So if I believe not now don't hold me to this. I believe if you have the The Oracle Linux I believe you can have VM based licensing Exactly if you use they support rel so if you want to spin up Oracle on rel I think you they say license for the whole machine even though you may choose to have only one VM running on it Right now the other thing you can do we are talking to a number of customers that want solutions for that So it's an ongoing right now though if you want Oracle support Using true with an open stack the only one that supports is our product is already bus platform nobody else supports that So one option for you is you could you can spin up instances? Using ironic right so you can go to a physical machine and say spin up an instance in there And that way you can maintain the Oracle license. That's one option Oracle virtualization is another option, but it seems is more difficult than bare metal Other thing is for those customers that are talking to us today They seem to have enterprise-wide license, so they don't have to worry too much about it But it is still an outstanding issue for us as an enterprise to figure out a way for customers to do that We do end up reselling licenses for Data stacks enterprise for example if you want to do that you can do that through us, but Oracle is work in progress Sure And then one other thing which we did not talk about today is we also support Oracle 12c multi tenant database. It's expensive by itself That's but forgetting that for a second if you had that What we provision is we when you say create a new database you can create a new Plugable database on a container running on the container server. So any database that gets created Always gets created on the physical device. So you won't run into that issue You have you have to pay Oracle to use that but once you do that you are safe from a licensing perspective other questions size limitations in what sense So I'm yeah, so I mean in your case You could specify the either the physical machine or the flavor Depending on what it is given the fact my laptop is smaller I chose smaller ones, but you could have pretty big flavors and you can do that, right? And we are going to support rack very soon So if you're talking about rack clusters and you want to distribute data that way you can do that But right now if you just take a look at an Oracle 11g single instance I can spin up a 5 gig Oracle database Two or three on my laptop, but not more than that But for your needs you could spread up a m1 large or m1 extra double large or something like that Which has multiple CPUs a lot of memory and stuff like that multiple cores And you can do that. I don't know if I answered your question But you have the flexibility to specify either the physical machine or the virtual machine depending on your needs And you can do Oracle there and once rack support is available You could distribute that load across based on the rack cluster itself other questions, please Anything you want to ask show because they have spent a ton of time and effort setting up Troll for real-world users anything you would like to know Either for from a trope perspective or even from a open-stack perspective the challenges they had and things like that Platform as service you are trying to build out. We are trying to get cloud foundry, but it's still work in progress Yeah, we're still not designed that yet, but you're planning It's it's in the planning phase We have roughly 12 more minutes anything else I can help answer Yeah, um make it begin tips of what? Sure. So right now Hey, Doug There's a question on orchestration right now. We don't support heat orchestration We don't throw doesn't support it orchestration. That's something that's coming in there Are talking about heat are talking about how things work under the covers in true So what we have done at Cisco is we are using Ansible Diploments, so we have our own Ansible Diploments scripts which goes and provisions Enter Indian provisioning of the service cloud Using Ansible for that. No, it's something in build within Cisco So one of the things we are going to provide is our contact information So it given the fact that it's an open community if there are questions that we can help answer After the session is over, you know, feel free to contact us We'll be more than happy to help out Other questions anything else we can help clarify provide answers for Going once Going twice Okay, so yeah, so so before we do that two things If there's interest in in deploying true, I mean that We are contributing a lot to documentation for the community itself. It's still work in progress But if you want to know more about How customers are deploying any general questions answers you can come to tesoro.com We can help answer we can tell you how other customers have done what you can do and things like that That's number one number two is that You have my email address my phone number and also Shoes and I am happy to provide that in the thumb drive but more importantly if you go to Open stack summit page This video should be available there and you will have the same information. You can always get this and hunt us down So not a problem, right? Thank you for attending it like I said I have I have a thumb drive for you But if you want more let me know I have we have a booth and I have a I have a few there I'm happy to share that to you for those ones that don't have it. I can get the permission. I can I can mail one to you


Ranger Bill, warrior of the woodland, struggling against extreme odds, traveling dangerous trails, showing rare courage in the face of disaster, in the air, on horseback, or in a screaming squad car. Ranger Bill, his mind alert, a ready smile, unswerving, loyal to his mission. And all this in exchange for the satisfaction and pride of a job well done. To the spreading chestnut tree, the village smithy stands. The smith, the mighty man, is he. Yes, the great poet Henry Wildworth Longfellow was so inspired by the physical strength of the village blacksmith that he immortalized him by writing this famous poem. Well, we have a giant smith in Naughty Pine. His shop is at the main intersection of two highways just at the north edge of town. Jacques Macintosh is his name. He has arms like the piston rods on a steam locomotive.  Jacques uses a five pound steel maul like most of us would use a tack hammer. He's a spiritual giant in the community too, as well as a physical tower of strength. A lot of truck drivers and townsfolk purposely pass this place of business just to hear his singing. But things weren't always this way with the beloved Scotsman, who carries some of his father's accent and his speech, and who radiates the love of God in the joy of salvation like the heat from the forge in his shop. But let's go back a couple of years and begin the story in Jacques's own home with his wife Anne. I like to call this one The Wrong Value. Oh Fred, how are you? Oh, I'm fine. How about yourself? Oh, I'm fine, thanks. But right now I'm up to my elbows and canning. I can't talk very long, Fred, because I've got a batch of watermelon pickles in the jars, and they're in the oven and due to come out soon. I can't talk long either. How's that big brother-law of mine? Am I nephew? Oh, they're both fine. Ernie's in school and he goes over to the shop after school to give Jacques a hand and also to learn the trade. How are Phyllis and the children? Oh, they were fine when I left. Oh, you're not on the road again. Any girl? Well, the boss says Freddie Boy hit the road. Freddie Boy hits the road. Oh, that's a shame. You haven't been back more than three weeks from the last one. I know. Maybe next year I'll get promoted. Then I can stay at home like I want. Oh, Fred, I must hang up. It's time to open the oven. Oh, right. It's time for me to get to work. Bye. Bye. It was so nice to hear from you. Hello, Bill. Afternoon, Jacques. I wonder where the fire is. I don't know, man, but it's been some time since I've heard the sirens crying out their bitter wail. I can tell you that. Well, what's on your mind? When can I bring the horses in for new shoes? When can you go out on the trail and take care of the rest of them?  I've already taken to the order of the shoes. And next week they should be here. And I can do the whole job in three days. One day here in the shop and two in the trail. How does that strike you, Fessy? That strikes it fine. Jacques, we just took your wife to the hospital. She's badly hurt. Her stove blew up.  Come on, man. Get in. Go on, Jacques. I'll close up. All right. Ernie will be... Jacques, stop walking in circles. I'll take care of everything. Oh, go on. Get in the squad. That's where the fire engines were going. Hi, you're Bill. Oh, hello, Annie. How's school, huh? Okay. Say where's Dad? Out on a job? No, son. He's over at the hospital. Is he hurt? No, but your mother is. She is? How? I don't know. Take it easy. She had an accident. I'll take you over in my car. I'll make it quicker on my bike. Hey, hey, Ernie. Come here.  What's wrong? Finish up locking up the shop and get in. I'll take you over to the hospital. What? Did Anne die? No. Well, spill it, man. What's wrong? Jacques, by Ernie, has been hit by a car. Jacques, how is she? I don't know, man. The doctor's still with her. Jacques, I've got to add to your load of grief. I'm sorry, but you must be told. I already know about Ernie. Well, how did you find out? I was in the hall when they brought the laddie in. Oh, Jacques, that was my fault. I should have made him right in the car. The laddie's impulsive, man. He told me you asked him to ride in the car. Well, there's nothing I can say. The English language doesn't have words to express my sorrow and sympathy. But I'll pray for all three of you, Jacques, that the Lord will heal your loved ones and give you strength to pair the load. Hello, Jacques. You want me to leave? No, no. Stay by all means. Jacques can need all the help he can get. Uh, Jacques, the doctor wants to talk to you. Aye, I'm listening. Give me the bad news. Jacques, your wife is blind.  Is there no hope? Not right now, Jacques. Perhaps later. How's the laddie? His right leg's filled with multiple fractures. It'll take months to heal, and he may never have full use of it. Jacques, I'm brokenhearted for you and your family. I'm sorry. I'm only a doctor, not the Lord himself. How can you do this to me? I've been a good man. I've been a good Christian.  I've accepted your son as my savior. I've lived by your laws. I've given you my money, and I've testified of salvation to men. Is this what I get for my reward?  How do you feel? Very good, Jacques. Dear Burns, pain you much? Not too badly. How's Ernie? He's in good spirits. But the doctor has no encouraging news. Here. Here, I'll get the water for you. No, no, dear. I've got to begin to find things for myself. You should've let me get the water for you. Don't you realize you can't see? Jacques, how can you say that? Forgive me, lass. I'm beside myself with despair. For Ernie and me, or for yourself? Explain yourself. Give me your hand, Jacques. What's come over, Ernie? I'm worried about you, dear. I worry constantly how you're going to take this.  No, no, no, please. Please, you mustn't be bitter. Oh, the Lord has a reason for all this. If it's His will that I shall be blind, then we must abide by His will. How can it be God's will that a wonderful woman like you should lose her sight? And the young lady of ours, it cannot walk and may never be able to walk very well again.  No, Jacques, no, promise me you'll never say that again. What good will I do because I'll be thinking it, even if I'm not speaking it? Horace of yours takes a king-size shoe, man. That storm's a big horse, Jacques. Thanks to your expert craftsmanship, you got a fine pair of feet. Yeah, that should fit like a glove.  So, Andy and the lad had told me, who now storm, who?  Ah, that's fine now. Have beer while you hand me that trimmer knife. I've got to shave his hoof just a wee bit. There you are. Thank you, man. Who now stormed? Oh, woe storm, easy big fella.  Why don't you get a good workout this afternoon? Ah, that's a fine fit. I'll rasp his hoof smooth to the shoe and he can run to his heart's content. Jacques, you're really a master craftsman. That's a beautiful job. Thank you, man. I was hoping that Ladiomine would follow in his father's footsteps and learn the trade. But God has fixed that so it can't be done now. Jacques, why do you use the forge? I thought you'd know that, man. To soften the metal so I can mold it and shape it to size and to temper the metal. How many times have you read your Bible through? Three, and I'll begin in the fourth.  I wouldn't care if God made me suffer, man. But to blind and burn my wife and cripple my Ladi? That I cannot say what God would do. I've been faithful to God and I've served him all of my life. God did not do this to my father, not to my grandfather. Why should he do it to me? Answer me. Can you tell me why he should do it to me? Can you tell me that, man? Can you? No, I can, Jacques. But it could be that God is testing your faith just like it is Job's. God and Satan had an agreement about Job. I cannot see how they could have won about Jacques McIntyre. I cannot see it at all.  Even though he has done this to me. But I'll go him one better. What do you mean? I will not turn my back upon God. How are you today, Ladi? I'm fine, Dad. Except for this leg. Sometimes it's pretty painful. There now, Ladi. I'll ask the doctor if he won't be giving you some more sedative. That'll help. Dad, is it true I'll never be able to walk again? That is something nobody can say, Ladi. It'll take time to find it out. Yeah, I guess so.  But it's done now. Don't, don't blame God, Dad. It was my fault. First of all, I should have let Bill drive me here. And secondly, I should have been more careful riding my bike. Hey, Al Campbell, how are you? I'm fine, but my poor old flat feet are killing me. Well, set a spell and rest, man. Thinks I'm pretty restful here at the firehouse tonight. Maybe this room in and maybe this room in itch. And I'll get back to pounding my beat.  I like babes, they are, Al. Everything's quiet. Hey, that's a Box Alarm movie. That's right. Let's take a look. That's Box Seven. A horrible thought just struck me, Mervie. I think I can read your mind, man. And Box Alarm came from there. Jock splashed myself. You're in my mind, all right. No, but it's not true, though. The poor man's had enough grief. Man, is that place burning? Ah, much good stuff and tools run. Oh, see, everything will be ruined. Oh, Jock has insurance. Hey, where's Bill? Over there with Jock. You stay away. It's no place for us now. You said it, sonny. Man feels foolish saying I'm sorry in a mess like this.  Best thing to do is say nothing. You're right, old timer. We're almost out now. They not need us. All right. Let's go home.  We've got a big day coming up. Yeah, I saw Jock. He probably wishes it'll never come. I've read a lot of your tools and equipment are ruined from the fire, Jock. Yeah, it wouldn't surprise me none. Let's put them all in the boxes I brought and I'll cut them across the street to that old grouch. I'm gonna carry on my business there temporarily. Mrs. Schneider can send it back to work then, huh? Aye. I thought she was an old grouch. But after talking to her, I find she's a fine old lady. She said she was going to offer the grouch to me. How about that? You made a new friend, huh? I hope I don't have to have a fire every time I make a new friend. What's that silly grin on your face, man? I don't see anything funny. You are, Jock. Me? How so? You're trying so hard to be better and it just doesn't come natural to you. It will. It will. Just give me time. Ann, I came as soon as I could. Glad you did, Bill. I had the nurse help me to earn his room here because I wanted to discuss our problem. Our problem? Yeah, Bill. Mom and I want to talk to you about... about Dad. All right. Go ahead, talk.  Jock is so bitter against the Lord. He blames God for all this. How can we change his mind? Accidents happen all the time to people. I don't see how anybody can blame God for them. Well, you have to understand Jock's feeling on these unfortunate happenings.  I see. I can't get him to talk about it. He closes up like a clam, and I felt tears on his face. That's the whole point. He can't stand to see his loved one suffer. How did he take the fire? I really don't know. He hasn't said much about it. Bill, will you try to help him as much as possible? Spiritual help, I mean? Yes, you're saying I will, but there's one thing you must understand. Yes. Jock has to come around to God's point of view on this, and he has to do it himself. I'll try to help him in this, if you like me. Jock, what's the matter? You keep your blood pressure down, man. Ah, that's easy for you to say. You don't have to work in this makeshift blacksmith's shop. Nothing fits. I don't have the room I need, and I'm going to lose business if I can't get the right place to work. Did the insurance company settle yet? Well, yeah. Why'd you ask, man? You ought to be able to get started building your new shop. All the builders are tied up. It would take a miracle to get it done in less than two months. Did you deposit the check? What check? I don't have any check. The insurance settlement? Oh, though that check. It'll be here in two days, and I'll deposit it. Jock, I'll make you a proposition. What have you got in your mind?  Bill, you've been a good friend, but now I think you've gone daft. How can you get the new shop built in a week's time? Let me worry about that. You gain? What good will it do? God's turned his back on Jock McIntosh, and that's for sure. Answer my question and stop feeling sorry for yourself. All right. You have my word. It's all I want to know. Now I got work to do. I can't do it, Bill. I've got three architects besides myself, and we're all snowed with work. Look, Mal, remember what you promised the time I carried your son two miles through heavy snow to get him to the hospital for an appendectomy? Remember you promised me something, and I never asked you to live up to it until now. I should have said yes right at the start and saved all this discussion, Bill. I don't know, and I wouldn't win anyhow. I'll have the plans ready in about 24 hours. That is the foundation plans, and the rest will follow. Thanks, Mal.  I've got four foundations to pour this week, and you asked me to pour one in 24 hours. Who do you think I am?  My arrangers and I stayed on the job 40 hours before we found her? 40 hours without a minute's rest. Okay, okay. I'll do it. When the blueprints are in my hand, I'll get my boys and we'll do the job. Come on, thieves.  Nah, you don't have to do that, Bill. I owe you too much for that. I'm just trying to plan my work so we can do it. I'll take half my crew of masons and carpenters. I'll talk to Johnny Olson. I know he'll help, too. You can be sure we'll get the job done like you want. Thanks, Knives. Thanks a lot. New forge and bellows and all this new equipment and tools. How many days? One week. Okay, I'll have it. No fight? Listen, the words around town is to what you have in mind. We think it's pretty wonderful. I'll get the stuff if I have to run a truck I cannot believe my own eyes, man. It's a miracle. No, it isn't, Jack. You have a lot of friends. Friends who believe in you, love you. And it must be. Tony poured the concrete footings two days ago. Now the concrete is hard and they start the walls. Look at those masons laying brick. Yeah, and the carpenters are ready to work right behind them in a few hours. It's a miracle. And you know who gets the credit, don't you? Sherman, you do. Oh, not me. God. But you asked these men to help build a new shop in a week. The Lord put the answer in their hearts or gave me leverage to get them to help. If the Lord didn't want your new shop to go up, it wouldn't. Look at this shop, man. Everything is brand spanking new. I'll say it is. You've got more room now than you ever had because it's laid out better. I cannot believe it yet. Well, you will. And now you've got a promise to live up to, Jack. All right. I'll listen. Where's your Bible? It's right over here. Here you are. Thanks. Let's sit here on the bench. All right. You know the trials and tribulations that Job went through, don't you? I? What did Job say after everything he loved and owned had been taken, his body covered with sores? He said, though his slay may, yet will it trust him? That's the first sign of Job's faith in God. Now, Job wanted to talk his problem out with God, too. He wanted a reason with God face to face. And that's what I would like to do. I want to ask God why I have to suffer all of this when I've been faithful and have honored him. Well, tonight I want you to read God's answer to Job. It's in these chapters. I want you to pray and ask the Lord to take the bitterness out of your heart. The whole answer is faith, Chuck. When Job came to the place in his mind and heart he was ready to accept all that had happened by pure faith and God blessed you because of his faith. You want me to have faith in God that all of his trouble is to test my faith and nothing more? Is that what you mean? When you can say to the Lord, even though you kill me, I'll trust you, then you'll have the answer that you want. No man can reason with God. That's what God tells Job in the last chapters. Chuck, how can you reason with God? Why not? Can you stop the ocean waters at high tide? Can you make the rain and the wind? Can you feed all the birds of the earth? Do you know where the universe begins and ends? I see what you mean. After you finish reading Job, then read the 23rd Psalm. And you'll find out you've been walking in the wrong valley, Chuck. Get out of the valley of bitterness and despair and walk in God's valley. Chuck, I was just driving by your house and Mrs. Morgan stopped me. Your wife is making supper. She can see you through the window. She'd kill herself. Get my home quick. Chuck, you're home early. What's wrong? Honey girl, have you gone daft? Or do you want to put me in an early grave? Chuck, you're shaking.  Wait a minute, Chuck. What's wrong, Cal? How does Annie know you're shaking? She's 10 feet from you. How does she know? Annie. Annie, you can see. Annie, you can see. Yes, Chuck, I can see. A few hours ago, my sight began to come back. It's almost completely returned now. That's the time after Bill left and I got down in a parade. I ask God to forgive me for crying out against him. Hiya, Dad. Well, it sure is good to hear you sing and be happy. I've learned to trust in God and to take everything He does by faith, laddie. Who am I to question God? Watch your crutches on the wet floor. What did the doctor have to say about your leg? He's beginning to think it might men pretty good. Maybe good is new. He says I'm strong and young and healthy. And that's your laddie. What's more, your father's learned to walk in God's valley now. What do you mean, Dad? Someday I'll explain it to your laddie. Someday soon, I'll tell you how to keep walking in God's valley. Why are we stopping here, Uncle Jim? Listen. That bike Smith sure is a big man. A much bigger man than you know, Dennis. Every chance I get I come by this way. He gives me a real lift. There's a real man of God. How do you like delivering mail in the rain, Mr. Phelps?  Just fine. You've passed the blacksmith's shop already. Yes, that's right. There's a really wonderful man. He helps me every day with his singing. Listen to him. So it is right now as you come into the north end of town with the two highways meet. Early in the morning and far into the night you can see the glow of the forge and hear the methodical pounding of the five-pound steel hammer. And above this sound of hard work you can hear Jacques's booming voice singing praises to his Lord. Hundreds of people stop in the course of a year to listen to him. They lean on this tower of strength, not his physical strength, but his spiritual strength, his deep and pure faith in God. Never again will Jacques leave God's valley. Well, see you next week. For adventure with...


The police command in Emo state says two of its officers died while three bandits were killed during an attack on the Zombie Police Division on Friday. The command spokesman CSP Mike Abbatam said the police successfully repelled the attack and recovered rifles from the bandits while others fled with bullet wounds. According to a press release by the Emo command, armed bandits attacked the Zombie Police Station at about 2.45 am on August 13th but could not gain access due to the swift response from the police. The command appeals to Emo residents to assist them with credible information that will lead to the arrest of the fleeing bandits.