Hi everyone, my name is Will and today I want to talk to you about simulating our Meso's Framework Cook. My plan for this talk is first tell you what simulation testing is in general and give you a sense of why you might want to do it and then tell you a bit about Cook, our open source batch scheduler that's really designed for the case when you have a lot more resource requests than you have capacity to fulfill it. Tell you why we built Cook in the first place and then tell you some of the more recent challenges that cause us to consider building a simulator. And then tell you a little bit about how we built the simulator and the trade-offs we made and go over one of the cases in which we use a simulator to improve the system. And so to start, it helps to have kind of an example system in mind when thinking about simulating a system. And this is kind of a generic system that you might have at your own company. It doesn't actually matter what it's really doing. We have some clients sending to a service. Service puts some data on a queue. Workers pull off the queue, do some processing, put it back on the queue. The service either writes it to a database, sends it back to the clients, maybe puts it back on the queue.  And the point here is that it's, even this is pretty complex. Like there's no way that we're going to be able to just understand by looking at this or even by inspecting some metrics. What it's doing, holistically, and if we make any changes, what those changes are going to do? We might have a sense, you know, we'll have mental models for the system. We kind of have a sense that, like, if we add more workers, they'll probably scale out linearly, hopefully. But we don't really know. And so if we wanted to start answering these questions, one way we could do it would be to just make the change in production and, you know, just see what happens. But that's probably, you know, that might be fine, but oftentimes it's not really what you want to do. You can't even ask, you can't even really ask the opposite question, which is, what would have happened if I didn't make this change? You can't ask the counterfactual. And so, you know, we have this case where we want to test a hypothesis. That's really what these questions boil down to. Because, you know, if I add more workers, my hypothesis is that I'll be able to scale out linearly. But we don't have a way to experiment with our system. You know, we can put it in production, but our production system isn't really a controlled environment. So what I mean by experiment is a scientific process to test a hypothesis. And so, you know, we can have a hypothesis and we can test it in production, but it's not really scientific. You know, there are lots of things changing besides whatever change we made to test our hypothesis. The workloads are going to change as we run our experiment. We might have an automated process or somebody that's logging into those servers and like updating things. And then what if that process fails or that person goes to lunch? And now, like, your system's in this weird half state. And you know, this is not the controlled environment you want to run. You want to be running experiments. And if we were a physical scientist looking at a physical system, like, that would definitely get thrown out in an academic review process. So you know, can we get closer? Can we get closer to running real experiments in a scientific manner? And so this really boils down to can we control the environment that our system runs in? And well, half of the answer to that is, well, yeah, I mean, we could just stand up another copy of our production system and just don't tell anybody about it. If no one's touching it, then it's pretty much controlled. You know, nothing's really happening that doesn't happen from just the system's own processes. But this isn't a really interesting system. There's nothing happening. But if we just send it production traffic, we're back to what we had in production, right? We've now lost any control we had. So what we can do is we can stand up synthetic clients. Now we control the entire environment. And what this means now is we can change any one piece and run a real experiment. Now, I should say that this still isn't fully controlled, right? Like, servers can still fail. There can still be network partitions. But it's way better than what we have in production. And so, you know, let's say we change this lead service. You know, we can start looking at, well, what happens when we upgrade our service? You know, hypotheses, like, does it remain correct? Or what if we changed our workload? What if we get twice as many requests as we normally do? Or we switch from a read workload to a write workload? Or, you know, what if something fails? You know, now we can start to ask these questions and run these experiments and really do this in a scientific manner. And so, this is kind of the heart of simulating a system. Is putting your system into a place where it's fully controlled. And this gives you a way to principally experiment on your system. This is probably a good time to point out that when you do this, your experiments might not always match what happens in production.  And you're like really happy and you do it in production and it doesn't scale linearly. And this is because no matter how close you get to a clean environment that's controlled but still like production, it will never be production. But I think that's still okay. You know, even if a model is not perfect, it can still be extremely useful. And so, we can now start looking at this, our controlled environment, our simulation, where we have our system and we have these requests and we start having them flow through and then we change something. And this is our way of experimenting. But, you know, so far we've talked about our system as just a copy of production. But you actually have choice here. And it's really on a spectrum from high fidelity and high cost to low fidelity but also lower cost. And so, the high fidelity side is you just take a copy of your production system. You know, the results you're going to get from these experiments should match well to what happens in production because, well, I mean, you're looking at effectively your production system. But this is going to be expensive, both in terms of dollars but also in terms of time. So, the dollar side probably makes sense, right? You need boxes to run your entire system a second time.  And if you have a very large system that's taking tons of requests, you're going to need a lot of boxes to be able to simulate this. I think the time part is a little more subtle though. Since you're doing actual network requests, actual disk writes, actual database reads, you can only run this real time. So, if you want to simulate over seven days of data, it's going to take seven days to run your experiment. And this means you either have to run very few experiments or you have to run very short experiments. And both aren't great. I mean, on the one hand, you don't get to get that many results. On the other, your results have to have this additional caveat that, you know, here are these results, but they only account for 30 minutes of data, let's say. And so they might not generalize to a longer running in production. On the other side of the spectrum is building a model of your system, either a mathematical model or you can, you know, write some code that approximates how your system works. And this can be much cheaper. You can often run in a single box on a single process and you can often run it way faster than your production system. You know, in this case, you know, you're not making any network requests, any disk writes, no database reads. I mean, this you can run way faster. And so you can run either much more experiments or much longer experiments. And so in that regard, you know, you could feel more confident about your results. The problem is that, you know, you're now taking approximations of your system. And so you might miss entire classes of behavior of your actual system so that when once you have your results and you try to apply them to production, they don't actually apply. And so, you know, you now have the caveat that you were working with a model of your system. And so your results might not generalize. But again, a not perfect model can still be useful. And there's really a full spectrum here where you can make models of small pieces of your system to get somewhere between these two extremes. And the choice we made with Cook was to take the part of our code that handles scheduling, the part that's like the actual Mesos framework, and use that code, but then mock everything else. So we run all of our databases in memory and we built a mock of Mesos so that we could run our system much faster in a single box. And so we'll see that in a little bit. But now, given that we have this setup, you know, we can start to ask, we can start to run these experiments with certain hypotheses. So, I mean, I think one of the most basic ones is basically the foundation of testing. If I upgrade from version one to version two, my hypothesis is that my system remains correct. And we could just test this. We could either generate traffic that looks like production traffic, or we can just take a trace of production traffic. You know, maybe look at what happened last week and apply it to this week and apply it, sorry, apply it to your upgraded system. And now, if your system remains correct, either because you have properties that you would assert or you just check that it behaves the same way it did last week. Well, now you could be confident or you could be at least more confident that your system's going to be correct when you deploy to production with this upgrade. You might have another hypothesis that if request rates double, your latency stays about the same. That, you know, your system scales well or that a doubling of request rate isn't going to hit a knee in performance where, you know, you jump to a new latency for all requests. Or you might look at a hypothesis around server failures. So, you know, maybe if a server fails, your hypothesis is that your system doesn't fall over. And, you know, when we look at real-world examples of people actually doing this, you know, this is a pretty common thing to look at. And so, probably the most famous example of using simulation testing to test your system and specifically to look at faults is Kyle Kingsbury's worker, Afer you may have known him as, with Jepsen. And so, the idea of Jepsen is it stands up a distributed system and sends it requests and then injects faults, either network partitions or server failures and checks that the properties the system advertises actually apply to the system in the face of faults. And if you haven't seen his work before, I recommend you go look at these posts. They're amazing. They're a lot of fun to read. And also, like really terrifying when you realize that lots of systems that we depend on every day have some serious problems. And so, I'm pretty thankful that Kyle's going through and finding these problems and helping us get more robust systems. On the academic side, there's this paper lineage driven fault injection where they take a model of your system and have a correct request flow. And then, they have this tool that will look at this request flow and try to find places where if they inject faults, they can make the request flow incorrect. And once this tool runs, what you get back is one of two things. Either you get a case where your system doesn't have the properties it says it does and you get this nice lamp or diagram showing where things went wrong. Or you get a gold star that says, for this particular request flow, your system is robust to some level of faults. And so, if you kill all the servers, your system's probably not robust to that. But given some level of faults, your system will be robust. And that's a really powerful guarantee. This is the kind of thing that I hope we get to see more of. A different type of example that's not testing for faults, but instead testing to see if we can optimize our system is this boat paper. And so, what they did was they looked to see if you change configuration settings on Cassandra, could you reduce latency? And sure enough, they were able to find settings that reduce latency by about 3x. And this is by changing just configuration. No code was changed. The way they did this was they just stood up a Cassandra cluster and ran through some workloads, changed the configuration, did the same thing and found settings and found these improved settings. I mean, this is really powerful. So, we've looked at simulation testing in general and now I'm gonna tell you a little bit about Cook. So, I mentioned before, Cook is an open source distributed job scheduler and it's really built for the case when you have a lot more requests than you have capacity to run. And the reason we built Cook was a few years ago, this is kind of what it looked like at Two Sigma and it still looks like this. We have a bunch of users that want lots of compute and often during the work day at least, their requests are for way more compute than we really wanna keep on hand. And so, we need to share it fairly. So, Cook has two mechanisms to do this. The first one is that we order jobs based on DRF, the same algorithm Mesos is using to decide what framework to give resources to next. And then we schedule so that users that have low share get scheduled sooner. And this works well when we have an empty cluster and a bunch of users show up and we were able to provide them all compute. But what can happen is if like two o'clock in the morning, there's only a few users that want compute, we're happy to give all of them large shares of the cluster. But as the work day comes around, more users want compute. And then we have this problem where the users that are already running, their jobs are going to take a long time. So, the users that are waiting will have to wait a long time before they can get their fair share. And so we have another component that will rebalance the cluster. It will preempt jobs from users that have a lot of resources and give them to users that have only a little bit. And so between these two mechanisms, we were able to provide an environment where users could request resources and then be able to run their jobs and be confident that they'll complete in a reasonable amount of time. But recently we've started leveraging the public cloud and with it has come a lot of capabilities and with it challenges. And so the first challenge is around heterogeneous clusters. Now, in the public cloud, you can purchase machines that are memory optimized or compute optimized. And then the choice of what job to run on what machine actually matters. They might have better performance on a memory optimized machine, but it might cost more. And so being able to trade off a cost for performance would be, I think, really powerful. But at the moment, our scheduling algorithm will just schedule the job in any box where it will fit. Another challenge we've had is around scaling in the public cloud. So I mean, that's one of the most powerful components of the cloud, is that when you need to burst, you can. And then when you no longer need the compute, you just scale down. But this really begs the question, well, when should you scale? And right now we have some heuristics to do the scaling, but what we really want is something that's more integrated with our scheduling algorithms. And so earlier this year, we decided we wanted to revisit some of the algorithms we used to schedule. But we had this problem. It's the same problem that we saw earlier, where we want to run the experiment of if we change our scheduling algorithm, are we able to take advantage of these things and either improve performance or reduce cost? But if we do it in production, we have all these confounding variables, prices change, workloads change. We can't be really confident that our changes actually have the effect we want them to. And so we set out to build the simulator. And so now I'm gonna tell you a little bit about how the simulator is built, how it's designed. And to do this, first it helps to look at how Cook is architected both from an external view and then internally, and then revisit what Mesos looks like. So this is kind of how we run Cook in production. We have three Cook servers for high availability, zookeeper for leader election, datomic as our data store, and then Mesos of course for resources. Internally there's a bunch of components, but the three biggest components are a component to take the jobs that are waiting and rank them based on DRF. And then send this ranking to a component that handles scheduling. It also takes in new offers. And then once we have these matches, we'll send, we'll call launch tasks on the scheduler driver, on the Mesos driver, to actually send those requests to Mesos. We have a second component that handles rebalancing of the cluster. So this is the component that preempts from users with lots of share and gives it to users with little share. And this is also taking in both the view of the entire cluster and the ranking of all the jobs we want to run. And once it finds tasks that we want to preempt in favor of others, it will call kill tasks on Mesos to free up those resources.  We have a few Mesos masters for high availability. So you keep it for some state and to handle leader election, the Mesos agents for resources. And then on the framework side, we have some client library, the Mesos driver, that handles the communication between the framework and Mesos. And so I mentioned earlier that what we chose to do was take Cook and then mock all of its dependencies. And so what this means is that we will end up mocking both the Mesos master and the Mesos driver and then bringing our databases in memory. Thankfully, both of them have a testing in memory version. So when you build a simulator, there's some high level properties you need to think about that are gonna affect your entire design as well as how you even interact with the simulator. And those two are whether to have a simulator be deterministic or non-deterministic and whether to have it be real time or faster than real time. And so looking at determinism, having a be deterministic has some really nice properties. It means that you can be more confident in the simulation itself. So if the simulation runs and you run the same simulation twice, so you make no changes, the results should be the same. And if they're not, you know there's a problem. You know, there's some bug in the simulation itself. This also means that if you do make a change between two simulations, the results that you see, the difference in those results is strictly from the change you made and not from any noise from the non-determinism. The problem with having your simulator be deterministic is that your distributed system is definitely not deterministic. And so by forcing your simulator and because of this, your actual system to be deterministic, it means that you're missing a class of behavior that stems from the non-determinism. And so again, you have to add this caveat with your experiments that, you know, this assumes that there is no non-determinism, which may or may not drastically affect how you interpret the results. Another choice you have to make is between having the system be real-time versus fast-than-real-time. And earlier we said that if you have your production system and just make a copy of it, you're forced to have it be real-time. And so this is really talking about what you decide your system means. And so if you make it fast-than-real-time, you're able to run more experiments. And so what we chose to do is make our system be deterministic and fast-than-real-time. And so at the highest level, this is what our simulator looks like. We have Cook and our Mach of Mesos that are talking to each other and we have the simulation driver that's going to instantiate both, wire them up and then drive the simulation forward. And so we'll look at each component individually starting with the Mach of Mesos. And so we implemented this. We needed to Mach both the driver and the master. And so on the driver side, we needed to provide implementations for launch tasks, kill tasks, decline offer. I mean, if you've built a framework before, I'm sure you've seen this sort of stuff. And then the Mach of Mesos needs to be able to handle these calls as well as be able to call into Cook and provide resource offers, let it know when it's been registered and provide information when tasks move from move to running and to completed. And this was a lot of fun to implement but we don't really have time to go into it. Ask me afterwards, I'm happy to talk about it. On Cook's side, we made a really strong decision to try to change as little as possible. And this is for two reasons. One is the less you change from simulation to production, the more confident you can be in your results. But the other thing is more from a software perspective. The more changes you need to make to accommodate the simulator, the harder it's going to be to keep your simulator and your code in line. And so by making as few changes as possible, it reduces how much effort you need to do to maintain your simulator. And so the only thing we really did was provide a way to trigger each of these components instead of having them all be event-based or time-based. And so this allows our simulation driver, once it's instantiated and wired everything, to trigger each component individually. And this allows us to have the simulation be deterministic. And so first on each cycle, it submits new jobs. It triggers our MacaMasos, send new offers and send status updates. Triggers cook to rank, triggers cook to schedule, triggers cook to rebalance. And between each of these stages, it's incrementing simulation time. And handling time when you want your simulation to be deterministic is actually tricky and you need to think about it. You know, time is one of those things that's going to make your system be non-deterministic. And so one way to handle this is between each of these events to keep track of the wall clock time that maps to what the simulation time was. And then in addition to doing that, you also need to keep track of the inputs and outputs. So you know, at this simulation time, this component got these inputs and got these outputs. When you go to look back at the results of your simulation, you can see how everything flowed. The problem with doing this is that you end up keeping track of a lot of data, both what wall clock time's mapped to simulation time, but also all the data that's flowing. And you also need to have ways of getting that data. So right now, because of the way we're doing the triggering, all the data flow is still handled by the normal code that we use in production. If we wanted to be able to get that data out, we'd have to add deeper hooks into all of our components, which, like I said, we don't want to do. And so we did something I'm not entirely proud of, but it does work, which is just stop time. So we're running on the JVM, we'll use Jota time, and deep in the bowels of Jota time that I hope nobody knows exists, is a function that you can set current time millies. So any call to get time now gives the back the same instant until you reset it. And so we use this so that all of our components, when they get time, all have the same time. It's now just simulation time. So any timestamps that are written to the database also have simulation time. And what this means is that we didn't have to make any changes to get any of this data out. We didn't have to keep track of what simulation time was. All we had to do was change current time millies throughout the running of the simulation. And so, like I said, this is a little sketchy, but it means that we didn't have to make these deep changes we didn't want to make. And it also means that if we look at the actual database of two simulations that have no changes between them, they will be identical as well, which is kind of a nice property when you want to look at the individual events that occurred in the system. And so, I mean, there's a lot more details about the simulator, but this kind of gives you a nice feel for how it works. And so now I want to tell you about how we used it. And what we did was kind of similar to what that boat paper did, where we looked at if we could change configuration settings to improve how we do rebalancing. And what I mean by improve in this case is, could we reduce the number of preemptions and reduce the waste that stems from those preemptions while still keeping fairness about the same? And so, I mean, I've mentioned it before, basically the idea of how the preemption works is if we have some users that have a lot, a large share of the cluster, we have users that have a very small share and they're not getting more resources at a quick enough pace, we'll rebalance the cluster, we'll preempt from users that have a lot and give to users that have a little. And what we're really changing here is just configuration settings. And the two knobs we had at our disposal were the number of tasks to preempt at any cycle and how selective to be when we choose to make preemptions. And so this is how unfair the allocation needs to be before we're willing to preempt tasks. And we set up our experiment to look at data between May 1st and May 6th. And what we chose to do is look at reducing the number of preemptions and increasing the selectivity. And we had a feeling this would work well because previously I had run a simulation that didn't touch any of these preemption settings. But I had misconfigured it and it had it so that it didn't have the settings that the production system was using. And the results from it, when I was looking at it, I was like, wow, this is really weird. Showed that this combination of reducing preemptions and increasing selectivity would let us reduce the number of preemptions and reduce waste. Whereas doing just one, one of them would, either waste would increase or the number of preemptions would increase. And so this was kind of this non-intuitive result that kind of popped out just from, well, a mistake. But lots of good things come from misconfigurations and mistakes. So we have this experiment and the results we were looking for in simulation was that fairness and just total resources that we used over time stayed about flat or decreased only ever so slightly but waste decreased significantly. And so that's exactly what we ended up seeing. Total resource use stayed about the same, fairness dropped by only a little bit and so we'll look at that in a moment and then waste dropped by about 15%. So we were really happy with this. And then looking at the simulation results for fairness, we looked at what we call starvation over time and what I mean by starvation is if you look at how much resources a user would get if we just evenly partitioned the cluster per user and then looked at how much resources they got allocated by Cook. Anything under that fair, anything under that even allocation, we considered them being starved. And so when we look at this graph, up is bad and what we'll see is that our old production settings and the new settings that we went to, the new settings were always higher but they tracked very closely. And so we were pretty happy to trade off this slight increase in unfairness for that large drop in waste. And so once we had finished our analysis, we were happy to go to production. And what you'll see is we're looking at waste over time now. And there'll be a line showing when the change went in and after the change went in, waste drops and then stays down. And so we were happy with these results. We were able to have a hypothesis, test it in our simulation, find results that we were happy with and then apply them to production and get to see that they matched pretty well with what we saw in our simulations. This was really powerful. There's no way we would have been able to just guess around and poke at these settings in production, right? We'd have to wait a few days to see whether what the effect of these settings were and we'd like go back and reset them. This would be really painful to find without something like a simulator. And so we've looked at what it means to simulate a system in general. We looked at what cook is and what it meant to simulate cook and saw a case in which using the simulator helped us improve the system. I should say aside from just this case, we also use it to improve testing in our system. So while building it and while starting using our simulator, we found two really nasty bugs in cook that had been there for years. And then once we started applying it to our continuous integration, we've caught a few bugs now that we didn't catch from any of our other testing and would have made it further into our process of QA and release before we would have hit the bug in probably in production. And so before I wrap up, I wanna leave you with just this one idea which is simulation testing helps us understand our system more deeply. Being able to experiment with our system and being able to do it in a principle and scientific way helps us understand not only whether our system is robust and how to improve it, but also let's poke at these complex interactions between these components. And any tool you can have that can help us understand these complex distributed systems more, I think is really useful. And so with that, I'm happy to answer any questions you might have. Microphone. I said you're mock of MeSauce. How general purpose is it? Are you planning to open source it? Yeah, so it's pretty general purpose. Well, so it's in the JVM. Cook is written in Clojure, so it's in Clojure. So it would be a little bit of work to make it able to use for general JVM users. If you're in Clojure, yeah, it is open source right now. You can take a look at it in the Cook GitHub project. If there's an interest, we can talk about it after. So how did you know when to stop, right? So you have this ability to go quickly change configuration and it seems like you could spend forever trying to find an optimal simulation. So what criteria did you use to say this is good enough? I'm gonna now go to production with the change. Yeah, that's a really good question. So in this particular case, this was the first case in which we use the simulator to find, to change something and then apply it to production. And so I actually tried only a few settings. I already said I had a sense of what type, what direction the changes should go in order to get the results we wanted. So I tried those configuration settings and two others to make sure that my understanding of how changing those values will affect the system in general. But I was really only looking at those. My concern was if I tried a lot of different settings, I might overfit to something that wasn't going to match production well. And so I was still kind of using a mental model that I had. Checked to see that it looked right in simulation and then went right to production because I didn't want to overfit to something that I still wasn't confident in. I think as you build more confidence in your simulation environment, you can start to run more simulations and really hone in on the right settings. And so like that boat paper, they're standing up an actual Cassandra cluster and sending it actual requests. And so they can be more confident in the simulation, in the simulator. And so they can hone in more on the right settings. So it really depends on how much you can trust your simulator, I guess is the answer to your question. Thanks. All right, thank you. Thank you. Thank you.


One of the most famous Shakespeare monologues comes from the play Henry V. A lot of Shakespeare scholars consider the St. Crispin's Day speech in this play to be Shakespeare's absolute greatest monologue, but Henry V has another very famous speech where Henry says, And of course this is a famous speech in which the king is motivating his troops in war and in it, he appeals to the nobility of their English blood, on, on, you noblest English, whose blood is fed from fathers of war-proof, fathers that, like so many Alexander's have in these parts, from mourn to leave and fought and sheathed to their swords for lack of argument. Dishonor not your mothers now attest to that those whom you called fathers did beget you. And sometimes when I'm reading speeches given by military leaders in the 19th century, I wonder if they're trying to copy one of Shakespeare's great war monologues like this one here. And of course a lot of educated men at the time would have been far more well versed in Shakespeare than we are today. When the Confederate Major General Earl Van Dorn took command of the Southern forces west of the Mississippi River, combining troops from Arkansas, Louisiana and Texas, he set out to inspire them with an impassioned speech that sounds like he was trying to take a page from Shakespeare's playbook in a way that suggests that Van Dorn saw himself gloriously playing the role of Henry V. As he turned his troops to face their Union pursuers in what would be the battle of Peeridge, Van Dorn addressed to them, quote, Soldiers, behold your leader, he comes to show you the way to glory and immortal renown. Awake, young men of Arkansas, an arm. Beautiful maidens of Louisiana, smile not on the craven youth who may linger by your hearth when the rude blast of war is sounding in your ears. Texas chivalry to arms, end quote. At least one of those phrases when the rude blast of war is sounding in your ears is almost undeniably plucked out of Shakespeare's monologue and the rest of it seems to be an attempt to imitate the motivational themes of Shakespeare. Even the language that he did not borrow from Shakespeare reads like his attempt at mimicking the writing style. Smile not on the craven youth who may linger by your hearth who actually talked like that even back then. And we can always read things into these vanglorious speeches with some interpretations being more critical and others being more charitable. But to me Van Dorn seems to see himself as the romantically portrayed version of Henry V, gallantly leading his men to victory. He's going to earn eternal glory and the immortal love of his men. He honestly seems to envision himself as a Shakespeare character in a way that might be a bit delusional. I've talked a lot about the romantic version of war that a lot of soldiers carried and how the brutality of war changed to their perception, usually after their first real experience in battle. Here with Van Dorn is an example of a high ranking general's romantic delusion about war. I don't actually know if his romanticism faded. I haven't read an account indicating this like I've cited for some of the soldiers in previous battles, but I suspect it did. After the Confederate defeat at Peeridge, Brigadier General Reigns was asked to buy his men with an earshot of Van Dorn if they lost the battle. Reigns responded, quote, by God nobody was whipped at Peeridge but Van Dorn. Van Dorn had Reigns arrested for the disrespect, but he couldn't prevent the fallout of the line. From that point on, the Confederate participants of the battle maintained Reigns' interpretation. They didn't lose the battle, Van Dorn did. So Van Dorn did not earn his glory and he did not have the love or loyalty of the men he commanded. Going into battle, he may have seen himself as Shakespeare's romantic representation of Henry V gallantly leading his unflinchingly loyal men into battle, but war doesn't play out in reality quite as nicely as it is often portrayed in fictionalized representations. Van Dorn would die during the war after being shot in the head, but even this would be an inglorious end to the general. When he wasn't killed in battle, he was murdered by a physician whose wife Van Dorn was sleeping with. I suppose Van Dorn might look like a character in one of Shakespeare's tragedies, but his life hardly played out as one of the characters he imagined he would be. I'm Chris Calton and this is the Mises Institute podcast, Historical Controversies. In the previous episodes, we looked at the first victories that started Grant's rise to prominence, which is part of the important Western Theater in 1862. Now, when we're talking about theaters of war, we generally divide campaigns between the Western Theater and the Eastern Theater, but of course there are further divisions that can be made. In the Western Theater, you also have the Trans-Mississippi West, which is where the Missouri Border War played out. These conflicts don't fit as easily into the major campaign narratives we have with people like Ulysses S. Grant, so this history is really spread out in my overall narrative. If you want the background leading up to P. Ridge, you might want to listen to the six episodes I did on the Operations to Control Missouri and the three episodes I did on Indians and the Confederacy, as these contain some of the same characters we find in the Battle of P. Ridge, which was part of the efforts on each side to control Missouri. Before we get into the battle itself, we need to cover some of the happenings that took place between Wilson's Creek and P. Ridge, including changes in leadership that took place on both sides. On the Union side, General Halleck was facing a dilemma. Prior to Grant's victories at Fort Henry and Donaldson, Halleck was trying to concentrate forces around Cairo, Illinois to prevent the Confederates from taking control of Kentucky, which President Lincoln considered crucial. But if he moved his forces from Missouri, which had lost to their commander Nathaniel Lyon during the Battle of Wilson's Creek, then Sterling Price, leader of the secessionist Missouri State Guard, would easily reclaim control of the state. On Christmas Day of 1861, Halleck sent Brigadier General Samuel Curtis to take command of the Union forces in Missouri, his objective was straightforward. Go on the offensive, destroy Price's Missouri State Guard, and solidify control of the state for the Union, so that Halleck could more comfortably move Union troops from Missouri to Cairo. Directly under Curtis remained a character we saw at Wilson's Creek, Franz Sigel, who fled the battle without sending word to his superiors. But as a German officer, he maintained the loyalty of the many German immigrants serving in the Union Army in Missouri. Sigel was unhappy that the command wasn't given to him, though his long history of fleeing from battles as he did in Wilson's Creek makes it hard to figure out why he thought he should be given the post at all. His position was almost entirely maintained by the loyalty he enjoyed from the German soldiers rather than battlefield performance. On the Confederate side, Sterling Price was butting heads with the commander of the Arkansas Army, General Ben McCulloch. The two officers had different priorities. Sterling Price was concerned with Missouri, which he believed popularly supported secession and only remained in the Union because of Northern coercion. McCulloch was less concerned with Missouri and more concerned with Arkansas, since it was officially part of the Confederacy, and he was also concerned with the Indian Territory, which I talked about several episodes ago. As I've mentioned before, Missouri had delegates in both the Union and Confederate Congress, with each faction claiming to be the legitimate representation of the state. Missouri's governor, Claiborne Jackson, was an avid secessionist, and he was lobbying Jefferson Davis to appoint Sterling Price as the commander of the Trans-Mississippi West. But Davis wanted somebody who graduated from West Point to hold that position, which disqualified both Price and McCulloch. So he brought in a new figure to take command, and this would be the man I discussed in the opening anecdote, Earl Van Dorn. Van Dorn's solution to the conflict between Sterling Price and Ben McCulloch was to simply take command of both of their armies himself, forming the Confederate Army of the West. He also brought in Albert Pike and his Confederate Indian regiments, which were placed under McCulloch's division. When we ended the Missouri episodes, the Confederacy had won major victories against the Union army, giving them control of Springfield, which was only a small town at the time, but had a lot of strategic value for its connection to St. Louis and its close proximity to Arkansas. Sterling Price was occupying Springfield, but in February 1862, Union troops were in Lebanon, which was only 50 miles from Springfield. While McCulloch was away in Richmond and Van Dorn was paying little attention due to the winter lull, Price was the first Confederate officer to realize that the Union troops were preparing to retake territory in the southwest corner of the state. Price sent word for more troops, saying that if he wasn't reinforced, he would have to retreat, handing Springfield to the Yankees. But he didn't have telegraph wires connecting him to the Arkansas town of Pocahontas, where Van Dorn was at the time. So by the time his message to Van Dorn arrived, the Union was already occupying Springfield and Price was moving southwest. Curtis continued to chase Price. The two armies were marching through the snow, and many of them, including Fran Sigil, suffered frostbite. But Curtis continued to push Price toward Arkansas, and they engaged in a couple of minor skirmishes on the way. The skirmish on February 17th took place in Little Sugar Creek, Arkansas, marking the first Civil War battle fought in that state. The battle was small, but it left a large impact on the soldiers involved, as this was the first combat experience for many of them. Confederate casualties in the battle are unknown, but the Union casualties typically note 13 men killed. However, if you find casualty counts for this battle online, they usually do not include the nearly 50 Union horses whose carcasses littered the field after the battle. But when the Union soldiers were faced with the unpleasant task of burying their dead after the fight, they had to deal with the frozen ground and the carcasses of dozens of men and horses. One rookie soldier, after seeing an ambulance carrying away an injured comrade, wrote about his experience with the grim reality of a war. The heads covered with blood were hanging out at the back end of the ambulance. The tailgate not being closed, jarring and knocking with every jerk of the wagon. A ghastly sight. I followed it, and when it stopped, went up and saw soldiers' burial on the battlefield. Two men were digging a trench about three by seven feet, and into this, side by side, with an army blanket under and one over them. The soldiers were placed, and the dirt, which was being removed for graves for the next dead comrades, covered them from sight forever." McCulloch and his men had joined up with Sterling before the battle of Little Sugar Creek, and the two generals led their men into Fayetteville, Arkansas, which was an important supply depot for the Confederacy. They didn't have enough animals or wagons to evacuate the supplies, so McCulloch ordered the soldiers to help themselves to everything they could. The warehouses were opened, and stocks of flour were literally poured onto the streets of the town. The Confederates at this point were famished, so they were delighted at the opportunity to feast, especially given the large stores of meat held in Fayetteville. What the men didn't eat had to be destroyed, and since it was cold, the food made good fuel for fires, with men even burning giant piles of bacon to warm themselves. McCulloch only gave permission for his men to help themselves to the army supplies, but the soldiers got carried away and started looting the town, making little distinction between military warehouses, private homes, and local businesses. Soldiers took everything from ladies' clothing, baby rattles, and schoolbooks. The entire affair erupted into a full-scale riot, and the officers were unable to restore order. The officer described the scene as, quote, one of the most disgraceful scenes that I ever saw, end quote. As the soldiers left the city, they set fire to all of the warehouses, as well as the local female institute, which had been converted into an ammunition factory to support the war effort. They left the town ransacked. After they made it some distance away, one soldier noted how the smoke over the burning town, quote, rose like a cloud in the heavens, end quote. Coming into Arkansas, Curtis' army was taxed. He had close to a thousand animals, horses, and mules, carrying supplies, cannons, or cavalry, but they were done. Some had died from overexertion, and the rest refused to keep moving. The men were no better off after marching through the snow with little to eat and inadequate attire, many of them suffering from frostbite. Curtis made the decision to stop the chase and hold his ground near Little Sugar Creek. Van Dorn had joined with McCulloch and Price, bringing all of the forces in the Army of the West together with Van Dorn in command. Where Curtis was tired of giving chase, Van Dorn was tired of his army being chased. After the Confederate army recovered from the same issues that the Union was dealing with, Van Dorn wanted to attack. In the last days of February and coming into March, both armies started to concentrate around Bittenville, Arkansas. The Battle of Pea Ridge took place on March 7th and 8th. It would end the day before the Battle of the Ironclads took place. On March 6th, around 5 p.m., the Confederate generals met to discuss what they should do. McCulloch had his cavalry scouting the Union position. In fact, the day before, his immediate subordinate, James McIntosh, had unexpectedly come face to face with Fran Sigil's battery, which confronted them with several rounds of grape shot from only 60 yards away, leaving several dead men and horses before the Confederates got away. So, McCulloch advised Van Dorn to avoid any full-scale battle, as the Union position was too well fortified. Instead, he suggested moving around the Union's right flank, which might pressure Curtis into abandoning his fortifications and force them into a more vulnerable position where the Confederates could attack. Van Dorn loved the idea of moving on the Union right flank, but he had a more ambitious plan in mind. I mentioned in the introduction that he seems to have envisioned himself as one of the glorious military leaders of Shakespeare's histories, and maybe these delusions of grandeur influenced his decisions on March 6th. He wanted to start the flanking movement that night, but only to block the Union from retreating down the road in that direction, while the rest of the Confederate army would attack directly, boxing the Union in, and forcing a surrender. McCulloch thought the idea was terrible. Sterling Price, at this point, blamed McCulloch for the loss of Missouri, and he was refusing to even talk to the other general. But even he agreed that McCulloch's idea was better, and Van Dorn's plan was reckless. It didn't matter. Van Dorn was in charge, and he believed that his plan would lead to a total victory over the Union army and allow them to reclaim St. Louis. Van Dorn would be a hero. And since he was the commanding officer, it was his prerogative to ignore the advice of McCulloch and Price and commence the attack. Van Dorn seems to have been naive about the Union army's willingness to fight at all. This, at least, was the impression he made on many of the men. One Texas cavalry officer, who had known Van Dorn for years and had served with him in the West, wrote of Van Dorn's plan, quote, The truth of the whole matter was General Van Dorn did not believe the Federals would fight him, but rather that they would get away from him, end quote. Truthfully, this wasn't an entirely uncommon Southern view. It's likely just an extension of the commonly held view that one Southerner could whip ten Yankees. But whatever Van Dorn's reasoning and motivation was, he was undeniably underestimating Curtis and his army. He had his plan, gave his orders, and the Southern army started marching that night. The Battle of Peeridge took place in two areas. The attack on the Union right flank, which was led by Van Dorn and Price, concentrated around Elkhorn Tavern and would span both days of the battle. The direct assault was led by McCulloch and it was fought near Leetown. The fighting here would be over on March 7th. The rest of today's episode will focus on the fighting at Leetown and in the next episode we will cover the events at Elkhorn Tavern. At around mid-morning, as McCulloch and his 7,000 men were making their way toward Leetown, they expected to press on toward Elkhorn Tavern where they would rejoin with the rest of the Confederate forces and fight on either side of the Union army. But the Union forces had split as well. Some of the Northerners were kept back at Little Sugar Creek, where they could reinforce either section as needed. But one division moved toward Elkhorn Tavern to meet Van Dorn and Price, while another division, under the command of Colonel Peter Osterhaus, one of Sigil's German officers, moved toward Leetown where he could cut off McCulloch. McCulloch was not expecting to encounter Union forces before he neared Elkhorn Tavern. As he moved down Ford Road, he was distracted by the sounds of battle coming from Elkhorn Tavern. His infantry followed the road, but his cavalry were roaming through the woods looking for berries to forage, since they could more easily keep pace with the infantry. Osterhaus' men jumped out from behind trees in the woods and started firing on the Southerners, completely catching them off guard. At least 10 cavalry were killed and many more wounded in the initial volley. The surprise made it difficult for the Southerners to respond in any concerted fashion. They had six cannons with them, but they realized they were being attacked. They started to unlimber their guns to fight back, but the Southern Artillerists only got off a single shot before their own infantry swarmed around them, blocking their line of sight toward the Union forces. Everything was a mess for the Confederates. The Union had three cannons of their own, which were a particular nuisance for the Confederacy because they were hidden behind two large oak trees, which cast shade over them. From a distance, all the Confederate soldiers could see a pair of trees that looked like they were belching smoke. McCulloch ordered Macintosh to lead his cavalry to take care of the battery. He delegated the task to the officer of the cavalry regiment closest to the battery. After the bugler sounded the charge, 3,000 Southern cavalry rushed toward the Union cannons, waving sabers and pistols and sounding out the Texas version of the rebel yell, which mimicked the Comanche war cry. The Union Artillerists got off maybe one or two shots in their guns, letting the Union cavalry handle the fight. But the Confederate cavalry was roughly six times the size of their Union counterpart. The commander of the Union cavalry, Colonel Cyrus Bussey, later recalled seeing his men disintegrate after engaging the Confederates in hand-to-hand combat. I have read in history of and seen depicted the horrors of battle where foe measured arms with foe in mortal combat, but here my own eyes witnessed them. In every direction I could see my comrades falling. Horses, frenzied and riderless, ran to and fro. Men and horses ran in collision, crushing each other to the ground. Dismounted troopers ran in every direction. Officers tried to rally their men, but order gave way to confusion. The scene baffles description. The charge demonstrated an incredible recovery from the Union ambush and disorganized response, but they didn't bother pursuing the enemy. Inside the Confederate cavalry, wandered around the captured cannons and counted the dead and wounded Yankees lying on the ground around them. Legend has it that the Cherokee-mounted rifles took part in the cavalry charge, but this actually isn't true. They were supposed to, but on Pike's orders, they dismounted. Pike was concerned with the lack of visibility due to the trees, and they did not want to participate in the battery charge. They did, however, route two other companies of Union cavalry coming in to reinforce Bussey on their way to the captured artillery after the charge was completed. When the hundreds of Cherokee joined the rest of the cavalry, they started killing the Union wounded and scalping the dead. There was only one exception to the cavalry's failure to pursue the enemy, and that was the regimental leader of the 6th Texas cavalry, Major Ross. He followed the retreating enemy until the line of Union infantry came in view. When he saw the forces blocking the path to Elkhorn Tavern, several thousand Union troops, he raced back to inform McCulloch of what they were facing. With this information, McCulloch made the decision to stand and fight at Leetown, and significantly, he neglected to send word to Vandorn that he would not be arriving at Elkhorn Tavern as planned. The Union line was formed in a clear stretch of land beyond the woods, called Oberson's Field. They were making their way toward the Confederates when the retreating Union cavalry came rushing toward them, yelling, Turn back! Turn back! They'll give you hell! It was an unnerving sight for the infantry, but the Brigadier leader, Colonel Nicholas Grusel, turned to his men and yelled, Officers and men, you have it in your power to make or prevent another bull-run affair. I want every man to stand to his post. The decision was to maintain the line in Oberson's Field. Osterhaus arrived a few minutes after this, and he was worried. The man was vulnerable. He had no reserves. Both flanks were entirely exposed. The cavalry was scattered, and Osterhaus's aides were currently trying to bring them all back to support the infantry. During the lull and the fighting, the Union cavalry trickled back in a handful at a time, but things didn't look good from the Union officer's perspective. The Union line did have a few pieces of artillery backing it up, and the Howitzers sent shells in the direction of the Confederates, while the rest of the men waited for the attack. The Union artillerists weren't aware that, of course, but their shells were effective. They scattered some of the Confederate cavalry, and the Cherokee fled back into the woods out of range of the explosions, where they would be of little use to the Confederates during the fighting at Lee Town. If you remember from the episode I did on the Trail of Blood on Ice, the Indians had their own strategies for fighting, and waging a direct assault in the face of cannon fire wasn't it. McCulloch was organizing his infantry, roughly 4,000 men, for the assault. When they were ready, he decided that as division commander, he would lead them into battle himself. McCulloch was confident. In one hour, he said, they will be ours. At around 1.30 in the afternoon, McCulloch ordered the charge. The Union line was formed behind a fence, which would offer little protection, but it did allow them to steady their rifles. As the Confederates charged, the Union men were told to hold their fire until the Confederates were maybe 70 yards away, at which point the order to unleash the first volley was given. Both cannons and muskets went off, and a number of Southerners went down. Among them was General McCulloch, who took a bullet to the heart, killing him before he even hit the ground. He didn't survive the first round of gunfire from the Union line. McCulloch and a handful of Arkansas skirmishers were a good distance ahead of the rest of the Confederates. Seeing McCulloch fall, Union soldiers climbed over the fence and rushed toward him, driving back the few skirmishers. The first soldier to reach the body was Private Peter Pelican, and he immediately took McCulloch's gold watch and started tugging off his boots. Other soldiers were close behind and started digging around for anything they could grab as well. But the looting didn't last long before the 16th Arkansas Regiment showed up and drove them off. The vast majority of the Confederates were unaware that their commander had gone down. When the 16th Arkansas showed up to the body, one of them said in astonishment, My God, it's poor old Ben! The regiment's second-in-command took off his coat and covered McCulloch's body, saying, quote, We must not let the men know that General McCulloch is killed, end quote. For the next hour, the death of McCulloch would be kept secret from almost everybody on the field, including nearly all of the officers. This decision would prove to be a grave mistake. McIntosh was now the commanding officer, so he had to be informed, but his next-in-command was left in the dark. Upon receiving word that command had secretly passed to him, he moved forward to take over the advance that McCulloch had prepared. Like McCulloch, McIntosh would lead from the front, continuing the attack as planned. As he moved forward, firing from both lines continued and both Union and Confederates fell. McIntosh was moving through the woods to get to the fighting, bringing the second Arkansas-mounted rifles with him. When they emerged from the woods, the 26th Illinois Infantry Regiment, the same regiment that killed McCulloch, spotted them and turned their weapons toward McIntosh. In an almost perfect repeat of McCulloch's fate, the 26th Illinois let loose a volley that landed a bullet in McIntosh's heart, giving him a practically identical death as McCulloch. It wasn't even 2 p.m. at this point. In less than 30 minutes, the Union line had taken out the two commanding officers of the Confederate division they were facing. With both generals dead, the regimental officers called their men back, halting the advance. The next-in-line for division command was Colonel Lewis Haybear, who was leading a brigade that was positioned to attack the right flank of the Union line while McCulloch was leading the frontal assault. He was unaware of the deaths of his superior officers, and when he heard gunfire, he assumed that it was his signal to advance. While the Confederates in the frontal assault were falling back to wait on Haybear to give new orders, Haybear was leading his men in their own attack. Members of the officer's staff came to find Haybear and tell him that he was in command, but by the time they got to his position, he'd already moved forward. They had no idea where he was. Now, had McCulloch not been killed by the frontal assault being carried out as originally planned, Haybear's attack on the right flank probably would have worked perfectly. The Union forces at Leetown were outnumbered by the 7,000 Confederates, and their line was not fortified at all. Even without the distraction from the front, the Union right flank was so poorly defended that Haybear posed a genuine threat. But Osterhaus had sent a request to General Curtis asking for reinforcements, and those troops, an extra 1,400 men, shut up right around the same time that Haybear was attacking. Osterhaus immediately sent them to strengthen the right flank. Haybear and the Union right flank, along with the reinforcements, fought for a full hour. The combat here was far more intense than the charge led by McCulloch and McIntosh. In fact, despite losing the top two Confederate division officers, the casualties on both sides from the frontal assault were rather light. On the Union right, though, people were falling much quicker. In one gruesome example, a single Confederate cannonball decapitated a Union corporal, flew through the torso of his nearby cousin, and then finally came to a stop in the chest of a private. Three people killed with one solid shot from a cannon. The weak flank held their ground for as long as they could, but eventually they had to retreat. Haybear's attack on the right flank was successful, and McCulloch not being killed, the battle may have been won. But without an attack from the front, tying up the rest of the Union line, Haybear would have no time to enjoy his tactical victory before the rest of the Union troops would start moving in on him. As the men on the right flank retreated, more soldiers swarmed in to replace them. Haybear's men kept fighting, but this time they were overwhelmed. Haybear was pushed south, forcing him to retreat south in the opposite direction from the rest of McCulloch's division. In full retreat, he and his remaining men fled the battlefield. Haybear left the division, never having learned that he was technically in command of it. By 3pm, Haybear was nowhere to be found. He was gone, of course, but the rest of the division's officers didn't know what had happened to him. Next in line for command then was Albert Pike, who commanded the Indian Regiments. But at this point, command had moved so far down the line that it reached an officer who hadn't even been privy to Van Dorn's original attack plan, so even if he successfully turned things around at Leetown, Pike wouldn't know what Van Dorn was expecting him to do in regards to Elkhorn Tavern. Not that this mattered. Pike was already in over his head. The Indians under him were entirely unwilling to take orders from him. He was trying to get them to drag the cannons that had previously been won from the Union out to where they could be useful, but the Confederate Indians had no interest in helping him. Some of the non-Indian Confederates were already fed up with them. The Indians fired their weapons at anybody wearing blue, and since the Confederacy still didn't have consistent uniforms, many of the Confederate soldiers were wearing blue coats, including General Van Dorn, so the Indian allies were proving to be more of a liability than anything else. But around 3 o'clock, Pike formally assumed command of the division, and word was spreading quickly of McCulloch's death, no longer a secret. Pike, at least, sent a courier to inform Van Dorn of the goings on at Leetown. But given the situation, Pike's assessment as division commander was hopeless, and he gave the order to abandon the attack and try to make their way to Van Dorn. He led the way, but only about 2,000 men followed him. Another 1,200 simply took off, abandoning the battle completely. So Peeridge is a good example of how important leadership is on the battlefield, and I don't just mean good leadership. Even an incompetent leader provides some structure, but after McCulloch and McIntosh were taken out, and with the highly competent Hey Bear know where to be found, command was falling apart. Pike's inability to deal with his Indian regiments was enough to compel most soldiers to dismiss him as a leader entirely. So Pike was now out of the battle, giving command of what was left of the division to a fifth person in less than 3 hours. It now went to Colonel Alcana Greer. His first order was to send a staff officer to call back the 1,200 units who left the battle but didn't follow Pike. The records don't indicate what happened to this guy, but we at least know that he was not successful in bringing the troops back. He most likely never found them. Greer was in over his head. He'd never commanded anything more than a single regiment of men he knew personally, as was standard for regiments and their officers at this point in American history. So this regimental officer was suddenly in command of an entire division, not just a brigade but a division. The entire Confederate force had broken down due to the constant loss of its leaders. For the next two hours or so, he tried to gather all the stray troops he could and restore order and organization. But this was to no avail. The men were starving, the animals were exhausted, and morale had entirely evaporated. He didn't even have the spare animal power to carry the three cannons they'd captured, so Greer ordered the burning of their carriages and then gave the order for a retreat. In the time this was taking place, Major Ross, the man Pike sent to give word to Van Dorn about the disaster at Leetown, was already on his way back. He was carrying new orders Van Dorn had written for Pike, but now obviously applied to Greer, instructing him to hold his position no matter the cost. By the time Ross arrived at the orders, Greer and the rest of the division were already gone. The Union now controlled Leetown. Moving around the field, they found the wounded and dead from both sides. With the exception, apparently, of the Native Americans, standard practice was of course to take wounded enemy soldiers as prisoners, but also to provide medical attention. But when one group of Illinois soldiers came across a pair of severely wounded Confederates, the two Southerners started to plead with them. They had been indoctrinated with the idea that the Yankees made a habit of finishing off wounded rebels with their bayonets. Upon hearing this, the Illinois soldiers were appalled that the Southerners believed that any Union soldier would be capable of such a crime. This is a good lesson in the effect of war propaganda, I think. Both sides in the Civil War were guilty of it, of course, spreading false stories of the barbarity of the other side. We can find similar examples of propaganda in other wars, probably any war, I would guess. In World War I, the British spread stories of German soldiers in Belgium tossing infants from bayonet to bayonet as a game. Before Desert Storm, the United States spread propaganda about Saddam Hussein going into hospitals and tossing babies on the floor to kill them, for no reason other than his own sadistic pleasure. In the Civil War, both sides told stories about the horrible behavior of either side, such as the cold-blooded killing of wounded enemy soldiers, even after the battle was over. Stories like these were nonsense then and they're nonsense now, but they are effective. The two Confederates were genuinely surprised that the Yankees had no intention of finishing them off while they were at their mercy. So this is the story of the fighting at Leetown, one part of the battle of Peeridge. While the Confederates were facing disaster here, Van Dorn and Price were fighting their own battle at Elkhorn Tavern, which would continue into the next day. Elkhorn Tavern and the rest of the Peeridge story will be the subject of the next episode. Historical Controversies is a production of the Ludwig von Mises Institute. If you would like to support the show, please subscribe on iTunes, Google Play, or Stitcher and leave a positive review. You can also support the show financially by donating at Mises.org slash supportHC. If you would like to explore the rest of our content, please visit Mises.org. That's M-I-S-E-S dot O-R-G.