Okay, let's get started. Thank you so much for coming to the JuJuCharm Workshop. We've been running these at every OpenStack Summit for at least the last two years, so I'm really happy to see such a great turnout. So the way this is gonna run, we've got pizza and beer, which is great. Feel free to have some. And we do know that there's also other sessions you might be interested in, so I really want this to be a discussion about your problems and things like that, if you need to nip out, that's cool. If you want to go check out something and then come back, that's cool too. We wanna kind of have a nice little groove here, and then we'll talk about all the problems we're trying to solve.  How many of you have no idea what JuJu is about, right? You're like, what's that funny name thing I keep hearing? Oh, not bad. How many of you are using JuJu? At all. And how many of you are just interested in kind of seeing what George can do for you today, see if I can help you solve a problem? Awesome, great. So I'm gonna give you the quick intro, it's like a little spiel in case you don't know what JuJu is and stuff, and kind of do, it's like a 10 minute intro or so. And then I'm gonna kick it over to Mark Mims, and he is going to, JuJu is best learned when we just show it to you. So we will go ahead and kick it over to live demos. We are doing everything you will see here today in the command line is live and running on HP Cloud. HP Cloud's an open stack cloud. So if you go home today, or when you get back, you can do everything that you see here today on a production cloud when you get home. So I think Drew from HP is here. If you wanna go talk to them, stop by their booth, it's a great combination. So everything you see here will be live. So stuff will break, we'll have a good time, you guys can laugh at us and stuff, and it'll be awesome. So welcome to the JuJu Talk, that's that slide. Okay, one down. All right, so who are you? Why should you listen to me? My name is George Castro. I've been working on Ubuntu since the very beginning. I've been with Canonical for six years. Prior to that I was a system administrator at Oakland University in Michigan, where I was a Linux admin. And back then, pre-cloud and things like that. And I remember some of the problems that I used to run into. And I'm joined here by Dr. Mark Mims. He's our lead ecosystem developer guy. So he's gonna show you these charms and things like that. Yell out whatever service you wanna see, we'll deploy it, we'll open these charms. I'll get to what a charm is in a second. And we'll actually show you how it works. So if you feel like writing your own, we can do that. And we can go as deep in technical detail as you want when we get to that part of the talk. So what JuJu is, so if you think back when I was a system administrator, the subhuman thing over there on the left that was like me, right? And back then we had like bare metal, right? And someone would say, we need to do this, right? And then I would put in a PO and I would get a server and it was cool, here's my new ProLiant. You would open it up, get that smell, it's awesome. And then you would rack it and get your cables all right and stuff and then rip it all out and stuff. And life was good, I didn't know any better. I would just go rack things myself and do all this kind of stuff. And then at some point, hardware got to the point where somebody invented virtualization. It's like, hey, you don't really have to go around buying all new servers all the time. You can consolidate them all and have as much usage out of a server instead of having them idling around and wasting power. So virtualization came around, everyone was happy. I was still kinda doing the same things but now I was doing them with VMs and the first time I saw a live VM migration it was like the coolest thing ever, right? And then at some point cloud computing came along and it kind of allowed and actually looking back now, you know, we're kinda spoiled, it's like, wait a minute, so you're saying a developer needs resources, he just has an API call, magic happens, and then he can get work done, that's crazy. Like, and then people started to like that and they started to like that a lot, so much so that now you have a ton of stuff running in the cloud and how are you gonna manage that? So configuration management came along, right? And you have your Puppeter Chef script that this instance is gonna be a web server, do these things. We believe that things are still moving forward and the next step is gonna be service orchestration. So we sit at a level above configuration management tools and we are all about doing service orchestration and not really caring too much about the individual machines because other tools do that. So I'll get into more detail of what this is. So this is where we feel Juju sits on the stack, right? And just like cloud computing depends on virtualization, depends on bare metal, right? It's, you know, you're not throwing anything away and starting from scratch, we're building on the lessons that we've learned. Then eventually, we'll all get so smart, there'll be robots and we all die, you've seen that movie. So something I want to talk to you today, especially at the OpenSack conference, so this is gonna seem kind of like heresy, right? Is like you go to these talks like Juju Workshop, right? And I was like, this is what you need, you need Maz and you need Juju and you should be running on Ubuntu and then you go to the next booth, some other dude is like, no dude, you need this, this, this and this.  Of course not. And like we get like really into this stuff where it's like, well I need this and you gotta need that guy, that guy can help me out do this, blah, blah, blah, blah, blah. But let's back up for a second. At the end of the day, nobody except the people at this conference sets up a cloud for like no reason, right? Check it out. I set up an OpenSack cloud. What are you gonna do? Did you just have it sit there? No. We've been deploying OpenSack with Juju for a long time now, at least the last three releases. And the whole deploying and configuring and getting OpenSack up, that's great, you need that, right? You need all those tools that I just mentioned. But we tend to forget we are making these clouds and stuff to solve problems, right? Customer problems. And at the end of the day, it's all about services. So Juju, instead of going bottom up, we take a view that we're going top down. Start with the service. Start with what you need to provide. So like, don't lose your mission. Ah, bummer. So I'll show this live in a minute. Sorry, it didn't scale, right? So what we're building here are cloud Lego blocks based around the services, right? So you say, I want a MySQL database. You get a MySQL database. You say, I want PostgreSQL. You get PostgreSQL. You don't have to manually build up, right?  And that's great. And you're thinking, oh, okay, but I've got automated things already that kind of do that. Where Juju fits in, and Mark will show you this by example, are those fields of the lines that connect things together? It's all about relationships, right? And what Juju was written for was to define the relationships between services. So you ever do this before. I'm setting up a blog WordPress. You got one machine. Let's say you have two machines because your blog will be really popular. You install your database and then you install WordPress, right? And then you have to wait for this thing to finish installing so you can put its configuration option into WordPress and then you get to move one more step. And then you have to click the button to make WordPress see the database so you can get, and you're zigzagging back and forth until finally you have a working blog. Wouldn't it be great if that relationship could be expressed and you figured that out? And then when you do that, you get horizontal scaling for free because if I know exactly how WordPress is supposed to talk to my SQL and I manage that at the service level, all of a sudden when my blog gets really popular, I could just add three more units of WordPress and those machines will know exactly what to do because I'm defining the relationship at the service level, not at the machine level. So that is what Juju does. And you write these relationships and install hooks and things like that in a set of scripts that we call Charms and that's what Mark will show you later. Charms are awesome because you can write them in any language. In fact, when you get back to work on Monday or whenever your hangover is done, you will, your custom deployment script thing that you've been carrying around in your back pocket for the, you know, between jobs and stuff. Yeah, I know how to deploy that stuff.  You're already probably most of the way there. What's really cool is we take all of these Charms and we stick them in a Charm store and let anyone be able to deploy them. So right now, today when you get home, you can deploy any one of the 120 plus services that are available there. And we really believe in making Charms reusable and shareable and usable by the community because that's what we use.  So it's really great to say, you know what? I need a load balancer. Click drag and a load balancer just happens. You just build your deployment. And that's what it looks like. And Mark will show you later. And this is a zoomed in view. So if you, when you actually do care about machines, you can go in and see what's going on. We'll show you that later. But I'm a command line junkie. So how many of you are familiar with App Get? Right? So a little bit of heresy here. App Get is a great tool. But a lot of people also say, you know, App Get is so great, you know, it's amazing. But when you think about it, App Get itself is really not that sexy. You know what's sexy? Is that it's connected to the huge Debian archive of all the stuff you need, right? So in a way, Juju itself, yeah, it will deploy this stuff for you. But the real goodness here is in WordPress, MySQL and Memcache Charms. So here is just a quick, simple command line, you know, how you would deploy WordPress. You do Juju deploy WordPress, OpenStack, and HP go get me a VM. And I can tweak that with parameters, of course. And then it will go and install WordPress. Now, I got some experience helping a really busy blog site scale WordPress. And it could be, who's here trying to scale WordPress and you're not a WordPress expert? Not a good time. So as we started learning how to scale WordPress, Marco who wrote this charm, he started putting all that goodness into the WordPress charm. So it comes with that caching and it does all the good stuff. So what's really great is, I'd actually, I don't really care, right? I'm getting WordPress and I'm getting a WordPress that's been peer reviewed and checked out by the community and really tweaked. Next, I deployed the database. The guy who wrote the MySQL charm is a really, really good MySQL expert, right? So you know when you go to your user group and it's like, hey man, can someone come to my job and like help me tweak my database, right? And then the guy gives you like tips. He's like, buy my book or whatever, right? What you really want is to get that guy in your server room, right? To kinda hook you up, right? So it's nice.  Now installing MySQL is relatively easy now. People can do it. But this is just a simple example. So when you think of things like Hadoop or things that are really hairy, when you get the book and the first half is how to install and configure it, right? I hate that. So this is what we do. We WordPress, MySQL, WordPress with memcache, scale's really nice, so we add that as well. And then those are the relations that we're talking about, right? WordPress, MySQL hooks fire off and he will show you these hooks. And then WordPress talks to memcache and it configures memcache to work with WordPress. Which really great here is when you do things at the service level, you get horizontal scaling for free, right? And that's what's so great about the cloud. When my blog becomes really popular, I can just add as many WordPress as I want, right? And I don't have to go back and talk to the database again or memcache because I've done that at the service level, which means I can just hit up, enter, up, enter and scale. And that's really nice. And I can go back down as well. So other things we ship, you can do like cool little options that are in the charts, right? So Marko thought it was a good idea. Wouldn't it be great if the WordPress content, you could stick that in a repo somewhere, right? So we add an option for it. You can do set debug, yes. And your logs will start to get filled up with stuff. And he also did, this is one of the coolest things we do is he made really opinionated decisions of what a good WordPress deployment would look like. And they're not, it's, the best practice is, it really depends on your needs. So the tuning, by default it's single, that means I'm an individual blogger here. We have optimized tuning that tweaks some of the variables and in the database and in WordPress itself to make it really fast. And then bear where it's like, hey, I'm an expert, I don't want you making any decisions for me, I'll do it on my own, right? So wouldn't it be cool if anything that you do deploy to the cloud, you can do things like this? So I'm gonna, we're just gonna show you because that's cooler. Any questions so far, anyone lost? Anyone having, anyone not having a good time because we still have beer, right? Okay, excellent. Yeah, why don't you show the gooey and I'll step off here. I'll let you talk. Okay, let's come back to that. Okay, so George and I are, as he said, we're an ecosystem team. We're really worried about building services on top of Juju. So we care about charms, we care about what it takes to get charms working and what it takes to get more charms in the store to offer more services that are deployable on Juju and Ubuntu. Before we, and we'll spend most of this talking about charms themselves, let's talk a little bit more about Juju and go through a couple of examples of that first. So, and that's kind of what I'd like to do for the rest of this time. And please, you really, really don't want me to appear just talking for a couple of hours. I'll be happy to do it. I taught college for a while, you know. So please interrupt, please ask questions. It's a little harder in this size audience but ask questions from your seat, we'll repeat them if you wanna just walk up with a mic and catch them on video. Stop me at any time, please. Anyway, so we'll talk about Juju, we'll talk about charms and charm development, what it takes to get charms done and then we'll talk a little bit more about Juju and I'll talk about some of the advanced features that it can offer you as a charmer. So when you're trying to charm up the services that you care about, what would that look like? So, okay, hello, and a TPP. So in this part we'll be completely driven on the command line so we will have connectivity issues at times and TPP issues at times. Okay, well, our agenda. Let's take a look at what sort of the basic point of a lot of this. So George showed a couple of pieces of what we're looking at and let me fix the eye so that we can all see this. Okay, brace yourself. It's ugly, but it's readable. Back of the room, see all that, see that, okay? All right, cool. So it's an example of deployment for something like Hadoop, which actually a decent-sized Hadoop stack is not trivial to deploy. This is an end goal for Juju. Juju deploy Hadoop master, Juju deploy Hadoop slave. Relate the two in a role that you'd like them to be and you want the master to read the name node and in this case we're gonna go ahead and deploy it so that the master is also the job tracker and stick the task trackers and the data nodes on the slave nodes. And I did this, I added a multiplicity into my slave deployment. That's the little minus N12. So this actually brings me up a 12 node Hadoop cluster. That's a pretty simplistic example because it's just Hadoop. In real life we've got more things going on than just the primary service we're dealing with in production. So we'll come back to the fact that there's a bit, there are more pieces involved here. Here's a simple stack. So deploy my SQL, deploy media wiki. I'm gonna deploy it with an alias here. The end of this line here, my wiki is actually just an alias for the service that I'm deploying. Excuse me. So we have two services at this point, my SQL, my wiki, we relate the two. We relate the two and I'm sort of telling, there's a little bit of ambiguity between how you might relate to media wiki and my SQL and we'll see that in the charms in a second. And we're just disambiguating that on this particular relationship here. And then I expose it. So this is the provider-definite behavior. Juju handles a series of different providers. It handles OpenStack, as we've discussed. It handles DC2. It handles a local provider, an LXC-based provider. So you can build little micro-clouds on your laptop. Sorry, buzzword micro-cloud, but forgive me, please. The little, but it basically uses LXC containers, lightweight Linux containers on your laptop that works well for an Ubuntu solution. But the, sure, absolutely. Let's, if you, I need to keep going, let me know. Actually, well, let's keep it, I'd like to show a bit more something on screen. Anyway, so there's also, there's OpenStack provider, EC2 provider, local provider. There's also a mass provider, Metal as a Service. It allows you to use Juju to deploy on bare metal and treat metal more ephemerally, like cloud instances. So what that relies on rapid reprovisioning. We're good at those things as an operating system. So, you've seen the mass story probably in the past where we talk about, we use Juju to take mass, the use of mass, Juju on a mass provider to deploy OpenStack itself, and then use Juju on that OpenStack instance to deploy workloads in that. And actually, we didn't mention that here, but we, could get some water when you have a chance. We, in this conference, we could very much just be talking the whole time about taking the mass provider, using Juju to deploy OpenStack itself. George and I are more interested in providing, in talking about services, the types of applications that you are going to be providing, or that you're going to be providing to your customers. The workloads that will run on top of the cloud you're dealing with. We can get into detail of the actual OpenStack charms, and we can talk more about doing that on the mass provider, but most of what we're going to talk about upfront here, at least, is going to be the workloads that you're deploying in the clouds that you've built. Okay, my point in all this, coming back to Expose. So the individual services requests, requests the ports that they're listening on, and they make that request to Juju. However, nothing is open until you pull the crank to Expose it, so Exposed is what adds rules to the security groups to allow inbound access on the ports that were requested by the individual services. That's important, there are tons of situations in initial configuration and startup where compromises are possible in a partially installed system, and this is a really nice way to sort of gate that, so that's the idea behind this. So, before we go into a bunch of other concepts, I'd like to show you something that's a little bit, it's hard to fit on one screen, but the point here is, and we'll go through a whole bunch of the details on here, the point in looking at the stack that I'm describing with this particular set of scripts, or this particular script, is that Juju, this is not just for toys, you'll see a lot of toy deployments because we can fit it on one screen. That's not all we're dealing with here. We have the notion of subordinate services that can manage infrastructure aspects, like log aggregation and backups and things like this. We normally don't show those, those aren't in the brochures and things like this, but don't think that it's just a toy because we're not showing those in the things that fit on one screen. So this is an example of, and I'm just gonna hop down to the bottom here, the idea behind what I'm gonna do is I'm gonna deploy two wikis. And the idea, the goal of this is, let's say I'm a Wikimedia Foundation contributor to Wikipedia, and I do, right, they have Wikimedia Labs, they have an open stack install there. If I'm a developer in that group, I'd like to be able to test things the way they're gonna run close to in production before they get to production. So what you wanna be able to do is take, is take media wiki instances or wiki stacks and deploy them in production-like environments that's more than just like on your laptop or something. So that's an example of where you would use such a, this kind of a stack. So deploy wiki, pulls in. So that's what I'm doing here. Let me, let me go over the setup. Deploy common services. We'll pull a MySQL master. I'll add in two read-only replicas. I will relate them as a master and slave. That's what that's doing there. I'm going to also, am I doing something wrong with the buzzy? Okay, so I'm going to give me an upload store. When you spread, you know, we're gonna put it behind an HA proxy instance or varnish, I don't remember which one. But we put it behind a head. When you spread your wiki instances, wikis have uploads. So those uploads, you'd be out of sync if those didn't share an upload store. That's what the NFS server's there for. Memcache server, deploy ganglia to watch it all come up and then expose ganglia in the step as well. That's pretty safe to do right away. Anyway, so the script starts out by deploying all of those common services. We just talked about. And then for each of those wikis, I'm going to walk through and I forgive the shell, it's the bane of my existence. The point of showing shell and using shell is that it's easy to read and use in common. Right, so Juju has a notion of an environment. Oh, sorry. So the question was, I'm talking about deploy-deploy-deploy and I talked about an example where a developer might wanna deploy something, but I didn't show you how to set up an environment in Juju or even talk about what environments are for Juju. Great question. So Juju supports multiple environments, even multiple environments on the same provider on the same account. I'll show you some examples of environment configuration in a second here. But you would basically have an environment configured, the developer would have a development environment configured or something like this on whatever provider they're working with, be it local on their laptop, if their laptop's beefy enough to handle LXE instances that can do a whole bunch of stuff, or you just test accounts on a cloud or a private cloud, perfect example. And like I said, Wikimedia Labs account is a perfect example of that kind of development situation. You'd have a different environment for production and the developer probably might not even have access to that, might nowadays the continuous deployment.  And this actually, this part of the script's a little old, the way I would relate to Ganglia is to put a Ganglia emitter as a subordinate service on the MediaWiki instances, but this works too. So I'm relating Ganglia to the Wiki, I'm relating the Wiki to the database, and here's an example of what I was mentioning before about there's a relation, but there's also a notion of an interface, so Juju has the notion of an interface for relations, so relations implement interfaces, and this is an example of a disambiguation of that. So the relationship between a database and the Wiki is scoped in this case as DB, and for the slave, it's explicitly telling the Wiki that it's a read-only slave. MediaWiki, the charm, supports the notion of being related to a read-only slave or replica. Not all charms do that, and it's a good chance to point out, we have, when you look at the set of charms we have, there's a variety of sophistication in the charm, some are just rock solid, and offer all these configuration options, and we'll go through some of them, and some of them are really new and placeholders, and an example of that, and there's some here that you'll see that I use just to show people what's going on, that I wouldn't want to run in production because I'd want to make that more maintainable, I'll show you in a second what I mean, it'll be clear, they're shell script. And then there are also some that are placeholders, a perfect example of that is PostgreSQL, the charm. I wrote the first version of that, not really good with Postgres, I wrote it as a placeholder, and over time we had someone who really knows Postgres really well own the maintainership for that charm, and he's taken it and done Postgres 9 replication and all sorts of great stuff with that charm. But the point here is that MediaWiki has the sophistication to handle a read-only replica, and that's what we're doing there by disambiguating that particular interface, spelling out the interface explicitly. And what happens is when you do this ad relation, Gigi will gripe at you, if there's ambiguity that you need to resolve, it'll ask you to resolve it and present you with the options, we'll see an example of that.  I can't see very well by the way. And then expose that particular head, and do everything we just did for each of those for two different wikis. Maybe I'd want to get fancy and pass a parameter in that size and something like that. The point here, this is a relatively mature, sophisticated stack of services that has closer to what you're going to be seeing in production, not just a toy from that perspective. One thing that could be misconstrued from looking at this is that there's a cinch in which these scripts are cool. It's really neat to kind of see that all built out at once and have it happen, and the time it takes to do this is really kind of ridiculously. And it's funny, the more capabilities we get, the more we want. So I spend my whole life waiting on clouds to provide, to respond to me. And damn it, it's just taking too long still. I wanted to go faster. But when you think about what it's actually doing with that script when that runs, it's ridiculous the power that that gives you, the leverage that you have as an individual doing that. But the thing that can be misconstrued here is that this tool is only for starting these things up. And that's not the case. I could deploy it without the, and here's an example of, like I might start with a wiki, that's just basic, wiki database, connect them, go to town, expose the wiki to the outside world. Three months later, three years later, at some point there's nothing that's saying all that stuff has to happen at the beginning. If you didn't do it at the beginning, you're screwed. What happens, at some point later I can spread that out a bit. I can do something like, so this is what we were looking at before, deploy my SQL, deploy the wiki, relate the two, expose the wiki. Maybe later though, I might wanna do something like, I wanna add more units to it. In order to add more units to it without going down, I need to add a head. And this is where it would have been nice to think about ahead of time of, oh, we should have deployed it first with a head and a wiki and a database and attached our IP address or public IP address to the head instead of the wiki itself, but whatever. So I deploy the head, I'm going to add a relation between the wiki and the head and then I'm gonna do something like, unexpose the wiki, expose the head. You might wanna do this.  You could do your DNS so that this could happen without any actual downtime. You would have to probably reverse these order, right? You'd expose the head, then you'd unexpose the wiki, but whatever. And then you'd basically spread it out. Spreading out in downtime depends upon the charm itself. Yeah, and I don't know many wiki well enough. We can dig into what it looks like under the hood to answer that question, what it would look like and whether there would be downtime when we're adding units. It shouldn't because the units would, actually it shouldn't because the wiki would come up. The way these particular services work is Varnish and HAProxy, well HAProxy in this case. Once the HAProxy will be up before any traffic is routed to the back end, the back ends will be up before any traffic is routed to them because what happens is the, and this is the hook order and that gets called.  I'll cover more of that later. Okay, so Juju, the infrastructure you're dealing with can be manipulated throughout the life cycle. It's a life cycle management tool. It's not just a deployment tool. And it's not just a toy because I can do more sophisticated stacks of services like this. That particular stack we can see here has, and this is a GUI representation. How legible is that from the back? Is that pretty clear? You know, I don't know. I can and then I think I can. But you know what? When I zoom back to fit the whole thing I just undid what I did. Let me see if I can squish it up a little bit. So the GUI is under development, active development still right now. There's a couple of aspects about it. Infrastructure-wise they're pretty interesting and I'll talk about that. That's kind of squished open. So there we can kind of see some of the stuff we were just looking at. What we're seeing in the relation, so the relations of the lines, what we're seeing in the text for the line is actually the relation name. So that's the, yeah, it's relation because it would be HTTP otherwise. So anyway, so we'll make sense of what those relation names map to as far as hooks in a second. Yes, sir. So what? Oh, yes, thank you. I've not told you how do we find out what sort of relations that a particular charm supports? There's a bunch of different ways. We're gonna look at the source but there's also other ways here. So I wanna say, there's one that's gonna have multiple ones. This guy I think I can, nope, not that. There's a way to do it here in the GUI. There we go. Build a relation. And it basically highlights out everything that can't be related to it. That's not everything, yeah. But that's one way to do this is you can actually map the relations between the things. We can probably do, nope, so click. Yeah, okay. There's a, so that's one way is through the GUI and we're working again under active development. That one's still being worked on. In other ways, we could actually kind of see what, we search for the charm here and rather than deploy it, I wanna look at, we can get the description, we can get the interfaces that are there from within the GUI. It tends to actually be cleaner at the moment. There's a jujucharms.com and I can go to SQL. Get the, so what we're looking at here is a set of interfaces. The interfaces that the charm provides and then the interfaces that the charm requires and what it's doing is it's actually listing the relation. The relation's called Munich. The Munich node is the interface name. Here's an example. The DB is the relation name. MySQL is the interface name and when you click on those, it shows you all charms that implement that particular interface. So the AWS relational RDS charm implements that interface. Galera does. There's a bunch of MySQL ones and then there are people who, oh here's charms that consume it. So outflower uses MySQL, cacti, chive, Drupal, et cetera. Etherpad, GitLab, Hive, Juma. There's a bunch. OpenStack, the dashboard. OwnCloud, Rails, so you can kind of start, there's a little web of dependencies there. I'll show you in a second where that's specified in code for each charm but you can certainly get pretty much any information you need about a charm. You can look at the set of hooks it implements. You can look at the set of relations that it provides and requires. And then there's usually a good readme that shows examples of how to use it as well as the basics of how to use it and typically good readme's, let's find the Hadoop readme is particularly a nice one because it shows you how to do the basics when you're doing it like we just saw in the example before where your name note and your data note are in the same. Or your name note and your job tracker are on the same master. That's not necessary. You can split stuff up as much as possible. He talks about rolling upgrades with HDFS and things like that in here. So we try as reviewers for the charms that get blessed in the store to make sure that they have a really good readme that kind of show you what's going on there. But the future of that does absolutely involve being able to do things like, I keep moving it while I'm clicking the hold it is the problem. I don't know to be honest with it. So the question was, what's the relationship to heat? Where do we see that relationship going forward? My understanding is heat in triple O is specific or specific to continuous integration. They're not sort of planning on dealing with packaged things and stuff like this. And that's my understanding. I might be there. But there's, so juju can be, we'll talk about integration with other tools in a bit because there's a lot involved with that. Short story is juju can be top dog in some of those situations but also can be driven by other things. So it's a juju API. And actually one of the, the Niskuli is a backbone JS app that drives a juju API over a web socket. So that's a, the idea going forward is that there's a lot of flexibility there. Okay, so we can get a lot, we can get a lot of information about what's going on with Charms. We can sort of, there's a couple of tools that let us manipulate things and work around that. One thing I wanted to do before we go too much further. So we saw some examples of how to use juju. Yeah, I'll get to a charm. Oh, so architecturally, let's talk for a second about juju itself. So the idea, you know, the probably the most weight in juju honestly is the providers. Juju at its heart was created to scratch the itch of distributed coordination or coordination in distributed configuration management like George described. I always have these problems of where I need the information at this point of the configuration that I need from this one. I gotta wait for this sort of come up if it ever comes up and get the information out of that and stick it over here. Juju was created to scratch that itch and that itch pretty much alone. So as a solution to that problem, what it is is it's really a declarative specification of your infrastructure. Currently, this is done in ZooKeeper. Put in what you want your infrastructure to look like. At the service level, so you specify basically enough information to see this in ZooKeeper. And then juju is on top of that, a handful of agents that subscribe to topology changes in ZooKeeper. So you've got a declarative specification of your infrastructure that you manage over time and agents that pick it up. Those agents can be considered part of the operating system where an operating system is easier at packages for it. They're actually, I think, in universe. Is it juju and main or universe? Yeah, it's in universe. So it acts like it's part of the operating system. There's no real difference at that point. Anyway, so that's a, and the agents are busy and they make that happen. And then that's all communicating with you when you're looking at things. Most of the state information is stored in ZooKeeper itself so in that state. It's really a very thin, what is that? That ends up being a very thin eventing mechanism, a very thin coordination layer. How, the key is that it's calling the right, it's calling hooks within the charms at the right time. When you do an install, it calls the install hook for that service. When you relate that service to another service, it calls the relation hooks for that service. And there's a little bit of a relation life cycle in where there are several hooks for every relation. There's joins and then there's change. And essentially the relations end up being a communication channel between two different services or rather all of the units on two different services. And that communication channel is open as long as either side continues, and change hooks will continue to be fired. And we'll talk about that in a second. So, and as we saw here, it's really this set of hooks is what the charm is. And that's all a charm is. If I look at the structure for, there's a charm. My SQL's a longer list because it had more hooks in there. But at its base, this is a really pretty basic charm. At its base, it's got configuration. So a charm tells you how to externalize the configuration parameters that you might need to externalize for a particular service. We'll look at that in more detail. Copyright, so let everybody know they can use this. Hooks and then some metadata in a readme file and a revision and let the charm store kind of manage what's going on there. The metadata is the first thing to look at. And so let's look at, it's really pretty straightforward. There's a maintainer. There's a summary description and then there's, I have a MongoDB relation. So this is actually a charm that deploys applications that are based on node.js. It's really trivial charm. It's not meant to be a particularly sophisticated one. It's more meant to show you examples of a really simple charm. And what does this one do is the node apps that this one would take would be a website. On the other side, MongoDB, close that full back. I've got a, it provides Mongo, it provides a database, provides config server interface. It requires other Mongo config instances. It requires a Mongo sharding services. And it does a replica set for peers. So there are potentially interesting deployments I can do with MongoDB, the charm. It offers quite a bit and it offers quite a bit of configuration as well. But what's important there is notice that this guy's requires matches up somehow with this guy's provides. They requires what's the Mongo with an interface MongoDB. And as long as that's the case, I can glue them together like Legos, okay? And so let's look into what actually happens in that particular relationship. Or we'll see that when we're digging in the hooks. Notice that for, sorry? Yes, sir. Are the relations always? You can have a generic interface that can have lots of providers. There is no notion of interface hierarchy at the moment. So it would make sense that MySQL and PostgresQL should be children of a SQL interface, say. There's no such notion right now. What we're doing right now and the way we have it specified is actually we should go back and fix that in the store at some point is MySQL and PostgresQL, all you'd really need to do is take each of those interfaces, turn them in the SQL and then extend that common interface with a type field that whatever you're connecting, whatever is consuming can then load the appropriate driver based on the type or something like that. And that's where you were seeing or when we click here of what's, so let's go look at that. Node app requires MongoDB. That's that interface there. That's where we get all the, that interface is provided by MongoDB charms. And you're looking at different, you're looking at different owner. This is actually not just looking at the official charms in the store, that would be the one owned by charmers. Here you see charmers on Eric MongoDB and it really irks me that we, so internal to Ubuntu really, we refer to releases of Ubuntu, not by their numbers but by their names and outside before I joined Canonical, I was like names, what are names? I don't care. I just want 1204 or et cetera. Anyway, so what you're seeing here two store owned charms, this one and this one, one for Precise and one for on Eric 1204 and 1110. The, correct. Sorry, the question was, we're taking responsibility for name conflicts in the official versions in absolute, yes. And series as well. Notice that there's not a quantal or a raring one. With Juju, it's really, what I do typically is I will have heterogeneous environments. What I recommend you do is use LTS only. There are times when that doesn't work, when what you need has not been back ported to LTS and it is perfectly reasonable and it's done all the time to have heterogeneous environments. You don't care what series is running for that particular service. You care that the that MySQL is running in the best way that it can possibly run. So if that's a quantal instance and other things are running on Precise, that's perfectly okay. There are ways to kind of specify, okay, no, I can only because we're whatever working for the government or something, we can only run on Precise instances that have been approved, blah, blah, blah. So that's an option and we work really hard to maintain the LTS versions in the store and then there are situations where you really need the one that like Ceph is a perfect example, Ceph really needs the later versions of the charm or the versions of the charm that work on later series. Although James may have had the back ported to Precise at this point, I don't know. Anyway. And so here we see, here's the, you know again the charms that provide that MongoDB interface, the charms that consume that MongoDB interface and you see in the GUI, we're trying to make that a little more automated so it's easier for discovery and search and things like this. The website is the best and authoritative place to do it right now. As you saw in Mark's keynote landscape, we'll have it integrated but through the Jujugui that we're looking at here. So that's all single, single source. Okay, so we see some hooks here, we saw the metadata file, we saw that the metadata as long as the provides and the requires match, then we can glue these together like Legos. We've got a dependency graph, if you will, between charms and services. Interesting point, we're actually doing some charm testing. Some of the ways we do charm testing is to walk that graph as widely as we practically can and spin up little stacks of services in an automated manner that spin up the service and then anything that consumes it and anything that it consumes, one step out, sometimes we go to two if we're feeling naughty. For every one of those combinations, it has to pass, everything has to be up and then related in order for the charm to pass that test. That's our automated tests. We've worth mentioning here. It's landed since this has been written. Every charm at some point soon will require tests within the charm itself before it gets accepted into the official store. Also, we were looking here at, and ggcharms.com, if you go the GitHub way, there's a charms repo here, George's big smile and mug. github.com slash charms and what we can see here is just, we're sort of the official store versions of the charms only. What we're looking at here is both the store versions of the charm as well as the, so where, what do you call it? Private charms or not private charms, but charms owned by people, particular branches. Juan Negron has a MongoDB charm here. So, we say, go ahead, yeah, absolutely. So, his question was, I see the charms run a lot of app-get commands. Does it ever make sense to just do puppet apply? Absolutely, we have charms that do that. Okay, so one of the things he didn't mention is the charm store, we don't freeze it, which means some people, for their charms, they need a new version of something. Actually, some charms have deploy options where you can say, we default to distro and then that hook is app-get install, whatever, right? An option might be second fig, upstream source equals github and then it'll pull it, build it on the spot and deploy that, or another option might be use a certain PPA. So, what we want to do in charms is basically give people options, right? Because it is pretty cool, especially as LTSs get older, right? You start aching, you're like, okay, so this version of this thing was good for me then, but now I'm kinda, you know. So, charms themselves, a charm is a script that runs at root as root on the instance when it launches, right? It just executes that stuff. Whatever, however you wanna get that software installed is totally up to you. Some charms have an option to build from source on the spot, some deploy a custom configuration. The whole thing could be a binary block. I don't really care. What we do care about is giving people the option to deploy what they need in the cloud and sometimes it's really nice to be able to just pull your entire stack from github, right? So that's cool. But generally speaking, in this store, by default when you're doing a deploy, if there's a package available, it's actually a policy, you default to the safe thing that was shipped in 1204. Over time, we want every charm to give people deployment options just like that. And we talked about store charms as the sort of official blessed ones and the real workflow there that's evolving is that you don't typically care about MySQL. You don't typically care about HA proxy. I don't, I just want them to work. I want enough config options externalized so that I can tune it the way I want if I want to. But I also want same defaults to be chosen for me for those services. I just don't care. I just want them to work. However, my application itself, the charm that I've written for my Node app or my Rails app or my Django app, I really care about. So I might not use a generic version of those. I might fork and maintain my own Rails charm that then is tweaked with all the special sauce that I want to do for that. So it's perfectly fine to have environments that have mixed charms that deploy either directly from the store or directly from your frozen branches that you keep in a repository that you're then deploying from those depending on how you roll for production.  Yes, sir? Depends on your particular service set. So his question was we're doing development on Quantum. We're gonna run into problems interfacing with charms that are running on precise. Usually not at all. It's your particular, what you would do in that case is take your charm that you care about and make that based on Quantum. It's really easy to deploy a Quantum-based charm. Did you deploy a charm store, or an ordinary even, you know, so I could specify a particular series. I can even make it, I specify the owner from a branch and launch pad or something like that. You can't quite do Juju deploy GitHub URL yet. I'm working really hard to push that through. But, and we'll get back to the question of how do you integrate with Chef and Puppet on that because there's good answers to that and I'll show you examples of it. The hooks we're looking at can be written in anything and the things that you care about are probably a limited subset of the services that you deploy. You probably deploy a whole lot of services where you want whatever the experts, the community, the upstreams decide or the defaults or the best practices for, you wanna be able to take advantage of the fact that that charm encapsulates those best practices and deploy it that way. HAProxy, my SQL, yeah, I care about my app. Yes, sir. Yes, there's a couple of tools that we've, yeah, sorry. Thank you.  Is there any sort of format I can use to describe what's going on here and then slurp that in? And absolutely, we have a couple of hack together versions that work with a previous version of Juju or a stale version of Juju that is being changed dramatically right now. We have the notion of Jitsu export and Jitsu import that'll allow you to pull out a one cloud and move to another cloud and we demoed that at OSCON and stuff. What I do in this case to answer what you're looking for is Juju API. So the Juju API can be programmatically driven. You can easily put together a JSON description of what sort of infrastructure you want and then call the Juju API according. That's the idea going forward with that and that is working right now. It's not quite as clean as I would like it to be for production as it will be for production soon. And what I mean by that is just that in order to get that server, you actually have to deploy that server within your infrastructure. It doesn't come as part of when you bootstrap an environment. So I want the GUI, the API server, all of that will be part of when you bootstrap an environment. It'll be there right now. You gotta add it extra. If you look at the... Yes, sir. Excellent question. So he's asking the shell scripts are sequential. Does that imply that the deployments are? No, not at all. They're just done that way because we look at them that way. You obviously have to deploy things before you relate them to make the infrastructure aware of it. If there is any sort of ordering in the relations or there's gonna be no ordering in the deploy services install. So you can deploy in service before another service or vice versa. That doesn't matter. They're deployed on different nodes. The install hooks are isolated. It's the relations that could be ordered dependent. And we are really careful in reviewing charms to ensure that they should not have any order dependence. And in practice, whenever we've found order dependence, we've been able to resolve that with that and not do order, I think. For everything I'm aware of in the store at the moment. And I'm one of the main reviewers. Yes, sir. So they're... Yes. Can I get any information about where my services get deployed in my infrastructure? This is very relevant to a lot of things. And the answer is kind of. Right now, so there are, and I think this, no, this landed in core. We were writing, we were doing this. We've been doing this for a while with this Jetsu external set of tools. This landed in core very recently. I don't know that it's been pushed up to the archive yet. The notion of deploy to a particular machine. So you can control things that way. The way we're addressing that mostly is to be able to do things like specifying constraints. You don't wanna put a database on a thin instance. You want your database to be fat. So I wanna specify constraints that this particular service, when I'm deploying it, do you deploy minus, minus constraints? Is what, I'll just actually, sorry, self promotion, but it's not really, I'm just going to the config. So you'd see deployment lines that look like, the middle there, Gigi deploy from a repository with constraints specified here. In this case, I'm doing an instance type, but I could certainly specify ram greater than four gig comma CPUs greater than five or two or whatever it is for four gig. And Gigi will do its best based upon the underlying provider to match that. However, if you know your provider has the ability to specify it through an instance type that's supported, but we're trying to get away from those, trying to keep it as provider independent as possible. That's a provider dependent constraint. Bad. But that's possible. Yes, sir. Correct. The by default, so this question is by default, everything is deployed to a server of its own. Everything, yes, everything gets a dedicated instance for primary services. There's also the notion of a subordinate service like a monitoring, a log aggregator or a backup service or an NFS client service or something like that that gets tacked onto the same container that your primary services is on. But every primary service gets its own container by default and any variations from that are doable. We do it all the time, they're just forced. And that's the deployed to is the way that we've done that. Situations, so there's a bootstrap node. When you do a juju bootstrap, it takes up an instance. You're paying for that. If you really care what you're paying on for that, you can actually go ahead and deploy your database and your WordPress server onto there and then spread it out later to different instances. And that's where we've kind of hacked around to do that initial deploy. Most of the default behavior is that it's gonna take up that bootstrap, it's gonna take up an instance just for the management itself. And more than an instance if you're doing HA, right? So you would do either a zookeeper quorum or the new, the rewrite of juju involves HA, that's by design. The older version is a hack on that. We've gotten working. So we can deploy with constraints. We can deploy with configuration. Let's see, what else do we need to talk about? Let's go in and look at what actual hooks look like. So, and this is, here's a great example of, so notice that I'm deploying here a, can y'all see this at all? I'll try to get that a little big. This is a juju deploy line and what I'm deploying is Hadoop. And this is where we start to try to get big to do some scale testing. But notice this constraint line that I have sort of configured there, or highlighted there. There's also the notion of config. So when I'm deploying a service, we see that these services offer a, in this case, MongoDB, and this is not the, let's go to, MongoDB has a whole slew of config options down here that you can do. And things like the number of backup copies kept, the replica set name, verbosity, MMS interval, bind IP, OP log size, there's a lot of stuff here. The good news for me when I'm working with MongoDB is I really want someone else to make those decisions for me and Mongo does, the charm does. So, but they're all there as config. How do I specify that config? I pass in, when I'm deploying, is one way. I can pass the config as a group with a YAML file. And that looks like, here's the config for Hadoop master, for instance, you wanna set the heap size, the DFS block size, the fact that I'm going to mount on in this case, whatever the local store was for that particular provider, I don't want it to go off of the root file system. Slave is a little smaller heap size, there were mediums. I do set the block size. I set the number of parallel copies. There's a lot of configuration parameters I can set. I can set that at deploy time via the deploy with config here, but I can also set that along the way with juju set, so I can do something like, and this depends very much on the service you're talking about. I can do a, literally the command is juju set, so let's see what we have deployed. Do you know a good setting to play with on? Do we have a WordPress deployed? I don't think we do. I'm sorry. I'm sorry, can you restate that? Change IPs, I have no idea. That's provider related. That wouldn't be a specific thing in the charm. That wouldn't be a configuration for the service itself. That would be a configuration for the provider, right? Am I misunderstanding something? I don't know what you mean by swap IP. Yeah, what happened? I'm sorry. So, well let's take a look at that. I mean I do see, when you do a NovaList, so I mean that's worth investigating. If I do a NovaList for what I have up right now. 10 minute warning for what? Oh, I think I have my region setting correctly in my NovaGloin. It's different from what I'm deploying right now. Yeah, it is. And I can't do that on the screen with everybody watching. Yeah, so we're not gonna do that through Nova. But we can do a juju status as a way, when I'm wanting to know what's deployed. And really what I'd like to do, well we'll just scroll back through it. There's a way to pass filters when you're interested in only a status of a particular service. You don't have to look at all this craft. But let's see what we're doing up here. So we're getting what looked like public IPs up here in DNS name. Those are machine states. We have some that are still pinned. Oh, you know what? I'm over my account one, I bet. We're doing, yeah. Anyway, so there's some of these, like be able to classroom that I just spun up that didn't come up, because I'm already over my account on my instance limit. The, you see here the name of the service. I'm trying to do VI, I'm writing. You see the name of the service here. You see the name of the unit. So a service can have multiple units and I have my little unit ID at the end of the service name, slash zero, slash one, slash two. And then I see that it's an age, say pending and I don't have an address because it didn't assign one. Ganglia, however, I spun up from the very beginning. We can see that it's in a started state. It's got a machine associated with it. The machine was the list we were looking at above. And then the public address is associated here. The GUI, yes. So depends. If the, yes. He's asking, when I see pending, how do I tell what's going on? How do I debug this? How do I figure out what's going on? There's lots of, I'll show how to debug charms and charm hooks. We actually have a debug hooks that is really, really cool and really, really useful and essentially breaks you. Gives you a teamwork session onto the instance you're trying, onto the unit you're trying to debug. And then it'll pop open a new window and break you at when the particular hook is called. And so you then have a choice to manually run through the hook and then exit. And so it's a nice little debug session based on Tmux. That's when it's up. In this particular case, the problem is, so there's lots of modes of failure. One is that the provider is saying, no, sorry. And we see that because the machine IDs were still pending up top. There's, those could be up and running. And then the services in a pending state, I doubt we have any of those because I haven't deployed anything for a while. The latest thing I deployed was WLB Classroom. But it's, but when the machines come up, they'll be in a started state, but Juju's still running this install hook. And that's, the agent state is pending. Agent state here that started here is pending in that situation. If you wanna watch what's going on there, the best way you can, of course, secure shell data, these machines, it's easy to do a Juju SSH ganglia zero. It takes me straight to that machine. I can do it by name. I don't really care what its IP address is. It injects the keys from the environment that I'm running Juju commands from. There's an environment option to add additional keys that you wanna inject to the instances. There's some production best practices around key management stuff we have written up. You can SSH to the machine, but there's also the best way to do it is a debug log that essentially will, it's a global log of what's going on in the interface. There's a way to, there might be a way to scope it. Like status, Juju debug status only for ganglia. I don't care about the rest. I just wanna see the status for ganglia. There might be a way to do that for the debug log where I'm only going to get, so I can specify warning levels. There's an exclude, but I'm not sure what the include and exclude are. Yeah, thank you. Oh, there you go. Yeah, for individual agents. So yes, we can call that. We can do Juju log, or debug log that's not just the global mess that this is. It's great actually when you're watching things come up though because you get what's going on. The newer version of Juju actually just uses our syslog for this. So that, yeah, and there's work underway to ensure that that's scaling to the degree that we need to go and figure out what are the alternatives as we get to 15,000, 20,000 nodes. And what is this five minutes until what? Till we're done done? Yeah. Dude, I don't know we're near done. Yeah, all right. Let me cruise through a couple things here so this is an example of the node apps install hook. You can see this in jujucharms.com yourself. You can walk through this. But the main points I wanted to get to is you can pick up the config that specified. Some charms are written so that you can specify the config via juju set on the command line at any time throughout the life of the charm or the service. Some charms are written so that you specify it as a mutable initial config. This particular charm, and that is less optimal. We recommend that people, when we're reviewing charms for acceptance in the store, we don't want a mutable config like this. We want config that can be changed throughout the life side of the service. So, but that behavior is very dependent upon the particular service itself. You get a lot of differences between services. Yes, sir. Right, right. Let me show you that real quick. So, his question was, when I have two different services and I'm dealing with relations, how do they share configuration information? And that is really the point I'm trying to get to the whole talk. Here's how I get config from the YAML files that I passed when I'm deploying, is I just kind of pull it using a config get command. And in this case, I'm gonna do things like, I can install for PPAs. And frankly, so as in Ruby development, for instance, I'm sick of hearing, oh, don't use Ubuntu, or don't use the Ubuntu packages for Ruby development because they're so old. You install Ubuntu fine, everybody does on the cloud, but you need to install Ruby from packages or from upstream or something like that or build the enterprise Ruby version or something like this. Juju, in the charms, anything goes. In the store charms, we have some preferences, but in a charm, you can do anything you want. I can do RBM, I can do gem install or bundle install. I can do what I'm doing here. I'm actually pulling from a repo that I've specified in configuration. And then I'm doing an NPM install, NPM install on the package.json that's specified there. Yes, sir. Trusted or not is the signatures, right? So you can't do an add app repository would fail if the keys are bad. So, yeah, right. And so we can tell here that we would know if it didn't come from Chris Lee, but as a charm author, I'm making the decision here to pull it from his VPA. What we try to do in review is make sure that that's only coming from trusted sources, right? Yes, and anyone can write a charm that does anything you want and push it up. That won't necessarily get in the store, though. Who's Chris Lee? Yeah, wait, who's Chris Lee? So in this case, in this case, actually. So this is community police. Yeah, actually Chris Lee is a well-known node guy and people are using his PPA in production and things like that. So I personally trust the people that tell me that Chris Lee is a good guy. However, in the store, by default, like we say, we want, you know, you'll get, by default it would say app get installed, node.js, and you would get an old version. That's why we want charms to have options. Wouldn't it be nice if you could just pass any node PPA from whoever, or the one you built for yourself that you've checked. So, let's move on to relations real quick. Yeah, if you were to submit this, he would yell at you. Yeah, he would say, hey, wait a minute. It would get knacked for, and this one's got all kinds of things that I would change, for instance, HearDocs and Bash. You know, I'd use real templating tools. And this is where a puppet would come in handy with that ERB or even Cheetah or something like that. But repo, it is, but this is better than the curl pipe to sudo. That's pretty standard in the node community, but anyway, the point here is this is the install hook. It's really simple. It's written in shell, it doesn't have to be written in shell. It could be compiled binaries, it could be Ruby, it could be anything with an Ubuntu runtime. You can write this. And in particular, you could just, we have charms that just do puppet apply install manifests or the equivalent for Chef Solo recipes, right? This is a great situation in which to use awesome DSLs for config management. No problem, it's great. The difference is that you'd pull from config and shove in a factor, right, before you're applying that manifest. And that's easy, and that's exactly what the charm I'm talking about does. I can show it, but we're really don't have time to do that. Let me get on. So this is, you know, the install app, it pulls node and installs any dependent node dependencies. It sets up in a NIT script, does some basic configuration for the app. None of this is related to a relation. However, the MongoDB relation is where, and this answers your question, sir, about how do they share config information between the services? And this is where the node service is in this hook, gets called when they're talking to each other, and there's a channel that opens and either side can do a relation set or relation get. And relation set passes information over that channel, and relation get is what allows this guy to pick up that private address. So this is what allows me, as a person that's managing this junk, to never know the IP address of the services I'm dealing with if I don't care about them. I mean, I can get it from status, of course, and I can secure shell there, and even, you know, I showed you an example of I was secure shelling with the instance where the service name, whatever it was on the other there, but I could certainly just put in the IP address, no problem. Ubuntu at 15 dot, whatever. But I don't have to, because I have this channel where they can really talk amongst yourselves. Yes, sir? Absolutely, absolutely. And this is where the sequencing of events, and I really thought we had like two more hours. I'm really sorry. This is where the sequencing of these events happen, and you will notice that the tree for node app hooks and the tree for MongoDB hooks, crap. Let's put that next to each other, maybe. Nevermind, that's not gonna work. The notice that there's MongoDB database relation joined here, and then there are MongoDB relation changed there. So join gets called on both sides first, typically with, and this is dependent upon the particular services themselves of what makes sense. A database typically, when it gets adjoined, it takes the service name, creates a database, creates credentials for that particular service to join, and then that's passed over the channel, and the application that's consuming it will pick that up through relation gets. It gets that in a changed hook. So there's sequencing associated with those two particular relation, the hooks associated with a particular relation. Join, change, change, change, and change will keep firing while, as long as either side is still doing a relation set, you could play tic-tac-toe, no problem. And that's it. So seeing, as a term author, understanding that sequencing of join to change is what the magic is behind there, in addition to the ability to pull something from the relation. And in this case, what you're doing is actually, when it comes time to configure the app, I'm gone, I'm gone. When it comes time to configure the app, in this case, you're just doing really simplistic stuff. Again, this is not something I would use in production because it's just setting a config file. In production, I would use a template, like an ERB template, and apply the changes to that template. But it could be as simple as setting a config file with the changes that you just got from the other side of the relation. Police them, I don't, yes, in review, we very carefully walk through that. At runtime, the thing is, I don't think circular dependencies, yeah, there's both a static review and dynamic reviews, so we do test that. But it hasn't been an issue to date, honestly. So there may need to be, we do have the automated testing though that would fail the test. So it would poke out, hey, we have some circular dependencies here. So that's it. So before you leave, I've got t-shirts for you guys. Thank you for staying for the whole way. Did we float the keg, or did we make a dent back there for our tender guy? I don't drink more, I guess. Yeah, so head back in the corner. There are surveys on your table, and we are very keen to find out what kind of workload you guys are doing. So if you wanna do that, and drop it off here, drop it off at the booth, doc rooms will be at the booth. You know, if you wanna ask questions. The shirts are ordered from smallest to XXL up here. I don't know if I have enough for everyone, but hopefully I will. Little slides of the talk. Thank you very much for coming. I really appreciate it. Thank you. Thanks for coming out. I have no idea.


This 2020 Ram 2500 Crew Cab Short Box is stock number 12198Z. We are here at Summit Automotive in Fond du Lac, Wisconsin, you're new and used heavy-duty truck and Ram headquarters. This 2020 Ram 2500 has the 6.4L V8 Hemi engine. It has been fully-safetyed and inspected by our service shop as a fresh oil and filter change. All the fluids have been checked and topped off and this truck is 100% ready to go. Bright white is the color and we shoot all of our videos in 1080p, 60 frames per second. So if you have HD capabilities on your computer, tablet, smartphone, or television, turn them on right now. Because it is definitely your best way to check out the quality and condition of the vehicle before seeing it in person. Also the options on the vehicle too. And if you like the video and want to subscribe to our YouTube channel, the upper right hand part of the screen is a link to subscribe to our YouTube channel. Click that and then click the bell notifications. You'll get updates on the videos we do each and every day here at Summit Auto. This one has the steel rims with the chrome covers and it has Firestone Transforce HT Tigers. These are LT27570R18s and they are brand new. We put those on in our safety inspection. You can still see the little wear knobs on there. Frame and underbody is an exceptionally nice clean condition. Driver's side front fender, no dents or dings on there and the headlight lenses are nice and clear. You can see the front bumper is in great shape. No dents or dings on there. And the grille is in nice shape too. Didn't see any dents or dings on the hood. And the passenger side front fender looks really good as well. Passenger side rim is nice. That tire is brand new as well. And as you go down this side of the truck, take note of how clean the body is, how reflective and mirror-like that paint is. We take these HD videos so if you are far away or even if you're close by and just cannot make the trip down but you're still interested in purchasing the vehicle, you can see the truck, hear the truck and have confidence in the vehicle that you're looking at before you even get here. So when you do get here, there's no surprises and you can make a smart and informed buying decision from wherever you may be watching this video or checking out our ad. Very nice down that side. This back rim is in excellent shape as well. And the back tires are brand new too. Frame and underbody looks like it is brand new as well. And it's like that all the way underneath. Looks really good as all the remaining factory exhaust. Lower rockers all look pretty good as well. And it comes with some nice chrome step bars. Coming around to the back of the vehicle, rear bumper is in nice shape. Didn't see any major dents or dings on there. Really the only imperfection I saw on the whole truck is there is just a little bit of a ding on that tailgate there. But other than that, this truck is very nice back up camera. Doesn't affect the function of the tailgate though. Drop in bed liner, you get some nice side box tie downs as well. Tailgate's really light, easy to put up. And as you go down the driver's side just as clean as I passed on your side. No dents or dings on the box. And for full disclosure, this back rim is nice as well. And that tire is brand new too. So four brand new tires on this truck as you go down this side. Doors look really good. Does have the heated fold up tow mirrors. They fold up like so. They also fold in. And they have built in directional signals in them as well.  Arnold rips or tears on the seats. They're in pretty nice shape. Factory all weather floor mats. Auto headlamps. Power windows, power locks, and power mirrors. Hop inside. Check out the miles and the radio. Turn the lights on to dim it down a little bit. You can see that this truck has 43,314 miles on it. You do get the digital speedometer, compass display as well. Instrument cluster is very nice and clean. Steering wheel I believe is leather wrapped. That's in nice shape as well. Cruise controls, gear selector, Bluetooth and information center controls on the left. Push button start has the eight speed automatic transmission. This one gives you the 8.4 4C radio, AM, FM and Sirius XM radio capabilities on this truck. You do get a backup camera on here as well and that is an HD backup camera. You can see just how crystal clear that is and you can zoom in over the receiver hitch to get hooked up to your boat camper trailer the first time every time. You also have all your different apps including projection manager. You can project your cell phone to the screen via Android Auto or Apple CarPlay. This one doesn't have nav but if you have a navigation system on your cell phone such as Waze or Google Maps project it to the screen and use it right through the screen. We actually did a demo on that on Android Auto in the upper right hand part of the screen if you want to check out that video and all the functionality that you can do with that. Down here is your turn dial 8 speed automatic transmission, factory brake control or stability control, tow haul and your manual climate controls. Down here you get two USBs, two USB C's and an aux jack and your 4x4 shifter is right there. You get your keyless entry right there. Pass and your side format and seat are in excellent condition. No rips or tears on there and the headliner is in nice shape on this truck. I didn't see any rips or tears. It smells very clean inside here. I don't think it's ever been smoked in. You do get map lights up there and your power sliding rear window button. We'll take a quick look at the back seats and then we will start it up and check out under the hood. Back seats are just as clean as the front seats, no rips or tears back here. It does have the latch child safety system for any child car seats you may have. Power sliding rear window with the built-in rear defrost. These seats fold up for extra storage and you get that all weather format back here which keeps your carpeting looking good. Side current airbags on this truck. Child safety locks on the back doors and the bottoms of the doors all look really good too. We'll start it up and then take a look under the hood. Starts right up. No check engine lights or anything like that. You also get a temperature display in that EVIC. I would personally like to thank you for checking out the video today and hopefully from this HD video you will have been able to verify the quality and condition of the truck inside and out. Under the hood we have the 6.4 liter V8 Hemi engine. Engine base very clean, runs very smooth. Once again this truck has been fully safety and inspected by our service shop as a fresh oil and filter change. All the fluids have been checked and topped off. And this truck is 100%. We're ready to go. There's your emission sticker. Four brand new tires as well. I'd highly recommend this truck from a quality and condition standpoint and to see more pictures of this truck or one of our other 450 new and used cars, trucks, SUVs, minivans, Wranglers, half tons, three quarter tons, one tons, you name it. We got it. Go to that website right there www.summitauto.com. Full pictures and descriptions of every single vehicle from two locations all at www.summitauto.com. If you'd like to check out more HD videos you can go to www.youtube.com slash www.summitauto.com. Remember to like, subscribe, and share on this video and all the videos that you see there. In fact, in a second you'll see a link to subscribe to our YouTube channel on the upper left. Link to more Ram 2500 truck videos like this one on the upper right. A link to this vehicle on our website on the lower left. And a link to one of our latest YouTube videos on the lower right. Click those, check us out. We're super excited to help you with this ultra clean 2020 Ram 2500 Crew Cab Short Box with the 6.4 liter Hemi tradesman in bright white clear coat. Thanks again for checking out the video.


Dude throwback. I was a bit better than that's growing up love Jason kid I don't know what's going on with the hair, but as you can see I had no forehead then and I still have no forehead now


What's up everyone it's your girl Kenzie here from Lane Sports Cards and I'm ripping some personals here for Shawn D. We've got a total of 27 boxes of 2022 Star Wars chrome sapphire. We've got seven boxes here and then we'll get into the two sealed cases. Thanks as always Shawn for your support and for hopping back in. Let's see what we can get you out of these 27. Good luck. This may be the only personal we've got time for. It took me a bit to get everything gathered and ready so we'll see. Get through as many as I can. One Star Wars sapphire that was so sick. Beautiful looking cards. Got orange to 25. Card number 45 the lightsaber. There you go. Good start there for Shawn to 25. We'll mag that. Green coming. Oh not green. So 75 card number 106 a message from Princess Leia and card number seven Darth Vader. All right and that'll do it for box one. Make sure these just uh just top load. Okay yeah the color just make sure it gets it gets top loader right away along with any card numbered one through ten. I'll be sure to set them aside. We've got purple to ten amazing robot seat three PO card number 66 nice purple to ten. We'll mag that one. You and me both. Card three uh R2D2. Card 10 uh Princess Leia is captured. Aqua to 99. Card 43 the um let me read that. Luke prepares to swing across the chasm. Luke in disguise. Yes Nick. Green to 50 the lightsaber to card 45. Card two Grand Moff Tarkin. Card 52. Green to 50. Who will win the final Star Wars or yeah the final Star War? Luke in disguise. We've got to 75 the villainous Darth Vader. Card number seven. Nice hit there for you Sean. We'll mag that. R2D2. Card three. Aqua to 99. Card 131. Spectacular battle. Card seven Darth Vader. Card number two C3PO and R2D2. We've got Aqua to 99. Card one eight one sixteen a hundred for their heroism. Okay give me in the ribs now. I'm just starting to get higher and higher. Got Aqua to 99. Card 31. Sighting the the uh sighting the Death Star. Same gold for you the other day Sean. I said it was a color match with this skin color but the gold and gold nice hit. We'll mag that one for you. That's sick. Do you imagine saying that at the Star Wars? Obi-Wan Kenobi. Card six. Card four Han Solo. That was a nice pack. Um nope this is Personals now. We are uh ripping Personals and Personals is full as well. Over full. We've got card 11 to 50. R2 is imprisoned by the Jawas. He said that really fast. No. Yeah. Card number one Luke Skywalker. We'll mag that. Start tomorrow at 10 a.m. Eastern though Tyler. You can check out latensportscards.com under the break schedule tab and you can see all the breaks available. We do have a new release of Saw for that forest. We'll be ripping tomorrow. C3PO and R2D2. That's crazy. Aqua to 99. The victors received their reward. Got card eight grand Moff Tarkin. To 75. Card number 41. Facing the deadly Chasm. Now we'll get into the faces. Card eight Moff Tarkin. Aqua to 99. C3PO. Nice one there. Card number 132. Green to 50. Lord Vader and a soldier. To the cases. Here we go. Good luck Sean. Case one. Box one. From the case. Got the 75. Card 123 of Solo Blast of Stormtrooper. Luke into Skye. Yeah Anthony it's sick. This uh uh Star Wars Sapphire is a beautiful product. I love that it's all from one set too like the OG Star Wars you know. Card three little R2D2. Card 10 Princess Leia is captured. Aqua to 99. Card number 81. Weapons of the Death Star. Obi-Wan Kenobi. Card five. Hey Sean um yeah absolutely. Let me do that after this box. We got the 75. Card 57 um Mark Hamill as Luke Skywalker. Nice one there. We'll mag that. Of course we appreciate all your support. Yeah he said yes that's me. Card four Han Solo. Sure thing. I can recap um all the cards that we're gonna mag here for you. If that works for you. Some of the better hits so far. Um Rebels Defend their Starships. Card nine. Yeah definitely appreciate your support. Luke Skywalker. Card one. Mag that. You gotta have a crazy collection. I'm sure of all these cards. The 75. Card 93. Stormtroopers Blast the Rebels. A long time. Oh yeah because every single card does get a season top loaded. Yeah uh Princess Leia. Card number five. There's usually a couple people working on it. All right let's do a quick recap here of what's gonna be mags. Do we have any mags there? No. Okay so this is the um the bigger stuff that we've hit so far. We've done seven. We've done nine total boxes out of your 27. The Luke Skywalker um to 75. Mark Hamill as Luke Skywalker. Another Luke Skywalker base. We had the gold CP-3-0. And I was saying earlier I believe I hit this one for you um over the weekend. So you got two of the gold CP-3-0. They said it's a color match there with the gold. That's a nice one. Uh the 75. Card seven. Darth Vader. We had the purple to 10. Card 66. Amazing. Robot C3PO. And then the orange to 25. On card 45. The lifesaver. There you go. So that's all we've got for the mag cards so far. Got some other nice color for sure too. Those are the bigger ones so far tonight. Yeah I bet that's awesome. Keep on keeping all. We got purple cards and card nine. The Rebels defend their starship. Perfect timing for you to come in. You see the back of it? Yeah you see a little starship there. That is the back. We need it for QC with tops. Gotcha. Have you been able to put any of those pictures together on the back? For the back on stream for QC purposes. Yes for the ones that are going to get magged we got Luke Skywalker card one. We'll mag that one. Are they? Oh wow. I'm glad they're able to do that. Tops is usually pretty good on replacement stuff especially with a card that big. My goodness. Got Princess Leia card five. Awkward at 99. Facing the deadly Chasm card 41. We've got green. Card six to 50. Nice hit there for you Sean. Nice Obi to 50. Mag that one. And 95. Yeah that was the purple for sure. I know those those first 10 cards are the ones that are really to chase in this product. And then color of them is just a bonus for sure. Big bonus. Box four the first case. Got green to 50. Card 79 preparing to board solo starship. Lightsaber green. Huh. That one looked pretty good though to me. I did show the back for you. Good to know. I have here gold to 15. Card number 31. Sighting the Death Star. Another gold. Here goes Sean. We'll mag this one. Nice. Yeah sure thing Sean. I don't know Anthony. I don't know if he will. Seems like a pretty pretty big Star Wars fan. Got card two. C3PO. Looking for looking for the highest offer. Yep. I could just imagine what some of the offers on that card you're getting Sean. One of ones are super hard to pull out of this product. We've got awkward at 99. Card 114 planning to escape. Got the base behind and Darth Vader. Card 7. Absolutely. Yeah that's that's what I was thinking. You might card on 90 out of 75 honoring the victors. That's like a card that like you could keep it and it would want to lose its value. I highly doubt Moff Tarkin. Card 8 will always be a desirable card. Yeah. We've got awkward at 99. Card 14. Luke checks out his new droid. Nice low numbered color there. Next Fox. Oh yeah for sure. Absolutely. I 100% agree. Like I've been saying like most of the other Star Wars product that's been released it's like a mixture. We've got green to 50. Card 88 stormtroopers guard solo ship. I can kind of see what you might you might be looking at on the rolls there. Is that what you mean? Guard solo ship card 50 or out of 50 sorry. But yeah how it's all like the same set in the other Star Wars like there's some from the animated series. There's some from the original movies. So it's a mix that with this it's just all from you know the original. We got C3PO and R2D2 Card 7 Darth Vader. So we can we can send a few boxes with you Sean if that makes it easier for tops to replace them. Out of 75 card 49 the Rebels monitor to raid. Just let me know how many how many boxes you want. We'll be sure to set some aside to send with the cards of the first case. Card 8 Moth Tarkin. No okay just making sure. Didn't know sometimes it's really panini I feel like it's picky that needs the wrapping and all that when replacing a card. To 75 card 99 Ben with the lightsaber lightsaber. Yeah very true. Very true. I guess that's enough proof huh. Card 3 R2D2. Luke in the skies. We've got green to 50. Card 96 the droids on Tatooine. Tatooine. Can't really see anything on this one. It's kind of it's harder with the yellow to see anything. I don't really see anything on this one though that's a good thing. Over 50. Card 10 Princess Leia captured. Oh my goodness. I guess with the first of its kind it still has some some things to work out if they do this next year. Yeah well that's good. Yeah you've probably seen your fair share of these so you already know. Got Aqua to 99. Card 55 Han Chewie in Luke. Card 6 Obi Wan Kenobi. Yeah we haven't seen red yet that's right. I think I pulled you one red out of your four cases on what was that Sunday. Out of 75 card 46 the desperate moments for Ben. Those are pretty pretty hard tools. Yeah card number one of Luke Skywalker. Card four Han Solo. Box nine of the first case. What's the lowest number that you have of that card number one? We've got card three R2D2. I think I pulled you an Aqua number one too on Sunday. Aqua card 21 the Tuscan Raiders. Scary looking people. Han in Chewbacca. Red coming. Let's do it. We got Princess Leia captured. Card 10 Luke in disguise. I feel like that's another desirable card. A lot of Luke Skywalker cards are for sure. The gold was that was card one. Oh wow. I think they were saying that as well. Like dude you called that. You called he said red coming. I'll pull from the back here. All right come on show us a big big low numbered one. Good luck Sean. Here we go red. Card number 38 Luke and Han in the Refuse room. Yeah 38. Wow beautiful hit there. A little bit of that strip across the back on that one. It looks like you can see. Let's see that right there. Big card there though. Two of five. We will mag this one for you. Yeah print line on that one. Yeah like I said with it being a first first product of its kind. Hopefully they can learn and improve on it next year. Got Aqua to 99 card number seven. The villainous Darth Vader. Nice hit there. We will mag this one as well. Tops. This is a tops product. Sorry I should have showed before I sleeved it. It's like a little dimple right there. Or is that. Nope that is on the sleeve the little dimple. It is not on the card itself. Yeah great box. Great box. All right number 10. Then we'll get into your second case. Feet Moff Tarkin. Got orange coming up. Orange card seven to 25. The villainous Darth Vader. Darth Vader card seven. We just pulled the Aqua right. Yeah Aqua and now orange. Beautiful. Gotta get the glare. Pretty good on the back there. Pretty good. Trying to show it's hard to get the glare on some of these sometimes. Pretty smooth looking card. Very nice. Three of 25. We will mag that one for ya. Great end to this case huh. Fantastic. Feeling another red in the other case Shawn? What are you feeling? We've got Aqua to 99 card world two. Rubbles prepare for the big fights. Make sure any big tips please show the back. Gotcha. Okay yeah anything we're gonna mag I'll be sure to show the back on on the stream for you. Absolutely thanks for popping in. It was nice talking to you Shawn and again thank you so much for all your support. These have been fun to rip. You've been pulling some really big cards out of here. Feel black in the other case. Oh man that would be uh that'd be something. All right so we got your loose uh seven boxes. Your first case of ten and then the last ten here for ya. Good luck. You got it man. Have a good night. Have a good one. Thank you. I hope we pull the black for ya. That would be crazy. That would be crazy. I don't know if I'm as cool as Caleb though. Caleb's the one who pulled the one of one I believe. Yep. Yeah. Case two. Good luck. Card six Obi-Wan Kenobi. Luke in the sky. We've got to 75 card 121 Han Solo in Chewbacca. Cool one there. Oh that is. Wow. You've got gold coming up. Gold card 117 13 of 15 Chewbacca poses as a prisoner. Do everything in your power to not get the glare. It's like a little indent right here. Card 117 on the gold. Box two. Yeah. I didn't see the other seven that's why. Obi-Wan Kenobi card six.   Oh my gosh. Awkward and 99 card 121 Han Solo in Chewbacca. We've got green card 83 to 50 aboard the Millennium Falcon. There's definitely some enrolls on the back of that. That one's pretty bad. Wow. That glare really does. I guess sometimes it helps. Yeah. You can definitely see him with that glare. Yeah. Goddy came and shot and told us that for sure. Okay. And then we put these in that same bin. Card seven Dark Vader. Yeah. It wasn't the base it was the other guys wanted to. Then they knocked it all out. Card eight Moff Tarkin to 99 Aqua. Card 79 droids to the rescue. To 75 card 119, 3PO, Bell, Ben and Luke. It's hard to read all these cards. All right. Box number four. Box number four to 50. Card 125 Luke in the skies. Nice one there. Mom shows a clean back. Pretty clean for the most part. There you go. Nice looking back there. We will mag this Luke. Card seven Dark Vader. Got orange. Card six to 25. Obi-Wan Kenobi. Boom. Sean killing it tonight. Very nice card. And clean on the back. Very nice.  This one for you. Moff Tarkin card eights. Haven't seen that card one in a while. It came in half a case. It's almost a whole case. Yeah. Card one. It's been a while. We pulled a couple of them back to back. Yeah. Box five. Case two. Got another red. Got another red. Pull from the back. Good luck Sean. Red. Card 93. Four of five. Stormtroopers. Blast the rebels. Boom. Another red. Much better shape than the last one. Trying to see what that is on the top right there. A little roll there. A little indent on that one. Four of five. Another red. Two reds so far. Come on. Let's see the black. Sean thought there was going to be a black. He called the red out of the last case. Let's see if he can call the black. That would be crazy. Crazy. Crazy. We've got the 75 card 35 Blast of the Laser Rifle. Card four. Han Solo. There it is. Luke Skywalker. Card one. We've got purple coming up too. Mag the Skywalker. We've got purple. Card 50. A day of rejoicing. Five of ten. Nice purple. That's a cool picture on the back there. A little indent right there. Same indent on a lot of cards that I've seen. Left. Ten. And card number nine. Rebels defend the Starship. Card seven. Darth Vader. Aqua. Card number 36. Blast of the Laser Rifle. Aqua to 99. Card 83. Abort the Millennium Falcon. Card eight. Moff Tarkin. Block seven. Card two. C3PO and R2-D2. Got to 75. Card 21. The Tuscan Raiders. Aqua to 99. Card 70. Special mission for R2-D2. Number 80 to 25. Where has R2-D2 gone? It's a cool looking card. It's good on the back there. Nice. A little Magnus one for you. Card 130. Pursued by the Jawas. Pretty clean too. A little indent right there. What a 99. Card 44. Han and Chewie shoots it out. Card four. Han Solo. Card nine. The Rebels defend their Starship. Card eight. Got Luke Skywalker. Card one. We'll mag that. To 75. Card 19. Searching for the droid. Box nine. Luke Skywalker. Card number one. We'll mag that. Aqua to 99. Card 79. Preparing to board the Solo Starship. Card four. Han Solo. Card nine. Rebels defend their Starship. Number five. Princess Leia. Green to 50. Card 25. Luke rushes to his loved ones. Pretty good for the most part. Last box nine and then we'll get into your final box for you, Sean. Right. Last box. Card seven. Dark Vader. Smoff Tarkin. We've got to 75. Card 44. Han and Chewie shoot it out. A full card. Come on. Show me a clean back. Looks pretty clean to me. Big hit there. A little indent on that same spot. Big Luke Skywalker. Out of the last box too. Mag that of course. And last pack. Alrighty Sean, that'll do it for your 27 boxes. Thanks again as always for your support and hopping back in and we'll be getting these right out to you.