Welcome, everyone. Eric Basil is our presenter this morning, and he has a lot of aviation experience, both as an instructor and as a LearJet pilot. And Eric feels very strongly that aviation safety is a matter of how we pilots look and talk and think about aviation safety. So our presentation today will be about surviving inadvertent flight into IMC. Eric, please welcome Eric. Good morning. Thank you very much, Kathleen. Thank you, Kathleen, for the kind introduction. Good morning. Welcome to another beautiful morning at Sun & Fun. Happy to be here. As Kathleen said, I'm from Chicago. So Florida weather is very nice. Change as opposed to the sleet we had a few days ago. So happy to be here. Thank you for your commitment to aviation safety. And my topic is surviving inadvertent IMC. It's a very important topic, and I hope that you find it educational and take away a few things from this. But anybody that's seen one of my seminars knows that I always like to start out with a little piece of videotape. And it sometimes has nothing to do with the topic. But this happened down in Louisiana. And I think it kind of illustrates what ends up happening when you become too committed to a particular decision. So we'll see what happened to this guy. Now, thankfully, he was unhurt. But what happened was this gentleman ran out of fuel in a 210 and did a very nice emergency landing to a road in this town. And once he did that, he started looking for a flatbed truck to haul the airplane to an airport. Well, he wasn't able to find a flatbed truck because nothing was wide enough to carry the wheels of the airplane. So after several hours of searching, they finally decided to put a little fuel in the thing and fly it out from there.  So the point of this is when anything unusual happens to you and you're flying, it becomes very easy to just try and find a solution to it. And you can often talk yourself into a very difficult situation if anything unusual happens. So back to the subject of inadvertent IMC, what we're going to cover this morning. First of all, why this subject is an important topic. Secondly, we're going to talk about something called spatial disorientation. And spatial disorientation is ultimately the reason why inadvertent IMC accidents happen. Next, we're going to discuss the importance of pre-flight and weather planning in your flying. Also, we're going to discuss something called the 5Cs or how to ask for help when you get in an emergency. And lastly, we're going to stress the importance of practicing your skills. Now, I want to start out by posing a little bit of a rhetorical question to you. And this is probably the reason why I started giving this particular seminar. Now, this event happened in Brooklyn, Iowa, just before Christmas in 2003. I lived in Iowa for about five years. And so this wasn't too far away from where I live. This was a gentleman that was flying from Pawakie Airport in Chicago, which is, as some of you know, is in the suburbs of Chicago. And he was headed back to his home in Ankeny, Iowa. Now, a gentleman was flying a Cherokee 6. He had four persons on board and, unfortunately, crashed into a cornfield and everyone was killed. A very tragic accident. And the weather at Newton, Iowa, which was the closest reporting station to where the accident happened, at the time, it happened about 6.30, 7 o'clock at night. So if you want to picture this in your head, it's about 6.30, 7 o'clock at 9 on a December evening. It's dark. And the weather is winds blow 10 knots, visibility of two miles in light snow with an overcast ceiling of 700 feet and a temperature dew point spread of one degree. But most importantly, the fact remains that this flight was operating under VFR and no flight plan was filed. This gentleman was about a 500-hour private pilot with no instrument rating. And he'd been flying for about 15 years. So I mean, he was a fairly experienced general aviation pilot by most standards, but no instrument rating. And for whatever reason, he decided to attempt this flight under VFR because he wasn't qualified to fly under IFR. And unfortunately, this was what the result. The purpose of the flight was the gentleman's daughter had been over in Spain on a foreign exchange program in college. And so she was coming back into the United States and flew into Chicago. So he flew his plane to O'Hare, to Milwaukee, to pick her up from her airline flight into O'Hare. And so it was bringing her home from Christmas. And it was him, the gentleman's brother, his daughter, and a friend of his daughter. And they all passed away. So this affected me kind of significantly. And so I think if we take a look at this as just an example, we can learn some very important things about this subject. We're going to talk statistics a little bit. First of all, VFR into IMC is the absolute number one cause of all weather-related accidents, more so than flying into thunderstorms, more so than any other thing that you could think of related to weather. So it's a very significant accident cause when we're talking about weather-related accidents. Secondly, we're averaging about 36 accidents of this type per year, a little less than one per week. And it's about 14% of all fatal general aviation accidents. More importantly, however, the rate at which these accidents happen and the fact that they're almost always fatal, this fact hasn't changed since the 70s and neither have the factors which cause these accidents. So it's a very pervasive accident cause in general aviation. And so I think it therefore requires as much energy and dedication as we can devote to it to try to reduce these numbers. Now another reason, I'll get to that in a second, but just first of all a question, how many of these pilots do you think have an instrument rating? You think this just happens to VFR only pilots? No, about a little more than a quarter of these pilots have an instrument rating for whatever reason they're not on an IFR flight plan or they weren't current. Secondly, do you think this only happens to inexperienced pilots? No, I mean it really covers quite a gamut. 44% of these pilots have more than 1,000 hours of total time. In fact, the FAs Civil Aeromedical Institute or CAMI estimates that as an average portrait, the average person that's involved in an accident of this nature is about a 30-year-old guy with at least 10 years or 1,500 hours of flying experience. I'm going to expand that statistic a little bit because in my experience I found that these guys have anywhere from about 300 to 1,500 hours of total time. So the answer, I mean basically the point of this is that it doesn't just happen to low-time pilots. It often happens to very experienced pilots. Another reason why it's very important to talk about this subject is because these accidents are almost universally fatal. As we can see from the graph here, only at least 91% of these accidents result in at least one fatality. And so it's a fatal accident almost all the time and that's largely due to the mechanism of the accident where the airplane is basically out of control because the pilot has become disoriented and therefore the airplane impacts the ground at a high vertical speed and high velocity. We're going to break down the statistics here a little bit and take a look at the graph. The first thing I want you to do is look at the left half of the graph here, so everything to the left here. The first column we have is labeled VFR in IMC and so this is what traditionally people think of. In other words, it's a VFR pilot, not instrument rated, flying into IMC weather conditions. And there are two basic columns here. I'll back up for a second there. There are two columns here. The first is labeled day non-rated. So about 40 some percent of these accidents are happening during the day and then the red column night non-rated is about 40%. So there's about an equal split between day and night of when these accidents happen. The second column is labeled VFR in VMC. Well, you might be saying, well, how does that make sense? Well, oftentimes these accidents happen when the weather conditions are not actually IFR. They're marginal VFR weather conditions. So it might be technically VFR but the pilot still becomes disoriented. Probably the best example of an accident in this column would be the very well-publicized JFK Jr. accident. On the night where he crashed, the weather conditions were actually VFR. It was about four to six miles visibility. However, he was flying at night over open water. So you can see that basically there was no visible horizon available to him and that's why the large majority of the VFR accidents happen at night, as we can see from this red column here. The right half of this graph relates to pilots that are instrument rated. And so disorientation accidents under IFR primarily happen during the day and then also IFR flight planes in visual conditions more often happen at night. So at night there's a significant risk of disorientation even if the weather is not necessarily that bad. We're gonna talk about a few common contributing factors to these accidents. In other words, why do these accidents happen? How many people in this room think that they're a pretty conservative safe pilot?  We all think that we're very conservative safe pilots. The problem is, is your mind does funny things to under certain circumstances and it's very easy for you to become pressured into making unsafe decisions even if you're ordinarily a safe pilot. The first thing I wanna talk about is what we call improper judgment of deteriorating weather. The fact is that most pilots don't really have a good idea of how bad the weather is when they're flying around in it. Pilots are not really that good at distinguishing between IMC and marginal VFR weather conditions. To prove this point, the University of Illinois did a study recently where they took student pilots and private pilots and they put them in a flight simulator. They didn't really give them that much information. They just said, we want you to fly from Champaign, Illinois to Terre Haute, Indiana. They gave them all the weather information that they would need to make that flight. So they just got in the plane and they started flying. While unbeknownst to these pilots, the people running the study had programmed the simulator to gradually decrease the weather conditions ultimately to IMC. And what the study was attempting to look at was how long it would take those pilots to realize that they were flying into poor weather conditions and then either make a decision to turn around or divert. And so they were timing to see how long that decision would take. Now, the conclusions from this study were quite interesting. First thing they noticed was that the more experienced pilots flew on into the deteriorating weather much longer. And when they were interviewed afterwards, they often had very high, unreasonably high estimates of what the weather conditions actually were. In other words, they thought the weather was actually better than it was. And secondly, well actually I made both my points there, but it seems a little paradoxical that they would think that. But that was one of the interesting conclusions of the study. And so the other thing is that the inexperienced pilots actually got scared earlier and they turned around earlier. You'd think that they might not know or they might continue to press on, but they diverted instead. Secondly, we have something called underestimating or overconfidence associated with flight in the adverse conditions. And this is simply the fact that pilots may not have a very good idea of what the risks are. You know, we all consider ourselves to be safe pilots and as you build experiences of pilot, you become more confident. And you become more and more confident about your abilities. And the problem is, is that you may not realize what the risks are. The next thing is quite important. And this kind of provides a frame for everything here. And that is social pressures affecting decision making. Compared to all general aviation accidents, VFR and IMC accidents have a very high percentage where passengers are on board. And so you can imagine the fact that the passengers are on board provides some external pressures to the pilot's decision making. So I realize that we all have pressures in our life as much as you do. You're flying a trip, let's say, you go somewhere for the weekend to Lake of the Ozarks or something like that to go canoeing or boating. And then it's time to fly home. Well, you're thinking to yourself, I've got work the next day. I've got work on Monday. My kids have to get back to school. You know, I might have a business deal, something. We've got social commitment or something like that. There's all sorts of things weighing on your mind back home, right? And you want to get home. You don't want to be stuck somewhere and have to pay for an extra night in a hotel, pay for a rental car, have your kids miss school, potentially lose out on some business. You've got all these things going on. And so it's very easy for those things to creep into the forefront of your mind and allow you to start making that decision. Well, you know, maybe it's not that bad. So understand that those social pressures can affect you in your decision making. Now, the next thing I want to talk about is decision framing. Decision framing is how you view a particular decision in terms of whether it is a good decision or a bad decision or whether it's a gain or a loss. And so in this sense, we're looking at the decision to either continue on to the destination or divert. So how many people are familiar with the TV show Deal or No Deal? I mean, pretty much everybody's seen that. Deal or No Deal is, you know, a situation or like let's make a deal where you get some choices and you can either decide what to do. So I'm gonna get, let's just put this in a general context for a second. Let's say that I give you a choice. I can either hand you $750 right now or you could have an 80% chance of winning a thousand. What would you do? You'd take the 750, right? Because that's the sure thing. Who would do that? Pretty much everybody in the room would take the sure thing, right? You don't wanna gamble, you know, and maybe come away with nothing. So in that sense, you're a conservative decision maker. We would call you risk averse. You don't like to take risks, okay? Now let's change the circumstances a little bit and change it from a game to potentially losing something. So I wanna say now, you can either lose $750 right now. I'm gonna just take it out of your wallet or you'd have a 60% chance that you'll lose a thousand. What would you, what would somebody, what would you do in the second row here? You'd take the loss of the 70, what if I change the odds? What if there was only a 50% chance you'd lose a thousand? You're gonna go with the sure loss, really. Who disagrees? Who would take the 50% chance of a loss? Quite a few people would. Most people would take that chance, especially if the odds go down, you know, that you'd lose anything. Because after all, there's a 50% chance you won't lose anything. And then you can take that money and go buy an expensive airplane part here at Sun & Fun. So the fact is, when people stand to gain something, they typically don't take risks. But when they stand to lose something, people might take a risk. So let's apply this back to aviation. If you look at a decision to divert as a loss, in other words, if you look at a decision to divert as, oh, geez, I gotta get a hotel room or get a rental car. We're gonna miss school, miss work. I'm gonna miss out on this business deal. If you look at it as a loss, what are you gonna do? Chances are you're gonna start talking yourself into continuing, right? Whereas, let's change the dynamic of how we think about this. If you look at a decision to turn around or divert as a gain, in other words, by diverting, you land safely and you only stand a chance of making it to your destination. If you look at it that way, you're much more likely to make that decision to divert. And so when you're making this decision, you can't view the decision to divert in terms of anything else that's going on in your life. You gotta just make the decision based on what's safest right now for you in the airplane. But make no illusions about it. This is, of course, a very difficult process and I don't wanna be flippant about it. The next thing we have is something called entrapment bias. Now, entrapment bias is where you have so much invested in a given course of action that to turn around or do something differently would be a huge waste.  A few people, and those of you that don't have your hands raised, I know you're lying, because pretty much everybody's done it. And that's a case of entrapment bias. You're so focused on trying to get to where you're going and you're just hoping and praying that you will have enough gas to make it. Whereas chances are you probably passed the gas station somewhere behind you, but you're so focused on going forward that you don't think about what's behind you. And as human beings, we're all forward-looking people. You know, we don't like to think about what's behind us, we're focused on getting somewhere. And so that's why that decision to turn around or divert is so difficult, because once it's behind us, it's in the past. We don't think about it anymore. And so this becomes more and more difficult, especially as you get towards your destination, right? You know, you've gotten 90% of the way home. What's getting that last 10%, you know? And think of it, how many people have a GPS somewhere in their airplane? Pretty much everybody nowadays. What's the biggest numbers on your GPS? Well, it's either the time or the distance counting down to your destination and big, bold numbers. And as that, you know, now you're 100 miles away from home. Now you're 75 miles away from home. Now you're 50 miles away from home. And it just becomes stronger and stronger pull to make it all the way. And to divert when you're only 50 miles away from home, you know, that can be quite aggravating. But the problem is, what if it's just so bad that you just cannot make it that last 50 miles? You have to be able to put yourself in the position where you can make that decision. So that's entrapment bias. The next we have is inadequate gathering of weather information prior to the flight. Believe it or not, a lot of these accidents happen because pilots just took things for granted or they looked outside, the weather looked fine and they decided to make a trip, especially if it's a trip that's over a familiar ground. Now I'm very guilty of this as much as anybody else. I made a flight one time from Southern Illinois up to Central Illinois. It was a route that I'd traveled many times. I was flying an instrument equipped twin by myself. I was IFR, although I didn't really do a very good job of checking the weather, because it was one of those days where there was absolutely not a cloud in the sky. 72 degrees, brilliant blue sky, clouds nowhere. It was just a great day to fly, right? Well, I take off and I'm flying along and I start to see some puffy little cumulus clouds below me. I'm at 7,000 feet. I'm thinking, oh well, I'll just have to descend down through those and I'll get cleared for a visual approach. Well, I get about 40 miles away from my destination and I hadn't checked the ATIS yet called Approach Control. Approach Control said, Springfield weather is 200 feet overcast, visibility one half mile and fog expect the ILS 3-1. Well, needless to say I was quite surprised at this because it was one of these situations where it was just absolutely beautiful only about 150 miles away. But the thing is, even if it's a trip that you're on a familiar route of flight and even if it looks beautiful, you should still have a general idea of what the weather picture is like. And the problem is people take things for granted and a lot of these accidents happen because they haven't done a very good job of checking the weather. And the last thing, we already talked about the fact that there's an elevated risk at night that we can see from the statistics. How many, what percentage of general aviation flight operations do you think happen at night? 10%. Yeah, 10% of GA flying happens at night, very small percentage. However, it's 30% of the accidents. So I mean, it's a elevated risk compared to how much activity there is. And lastly, eight out of the 10 states that have the most inadvertent IMC accidents are located in designated mountainous terrain. So mountainous terrain is also a factor here. But to sum it all up, I just call this something called the expectation of success. When you become a pilot, you start to put experience into your bag, right? And very few of these accidents happen to student pilots or people that have just gotten a private pilot certificate. And I'd like to think that the reason for that is because flight instructors do a pretty good job of protecting their students when they're coming out of the nest, so to speak. The flight instructor puts a pretty good control on that. But once you get out into the world and you start flying on your own, as you fly more and more, you're building more and more experience and you're becoming more and more confident in your own abilities. And so ultimately, you're gonna start to test those skills by flying in not perfect weather conditions. And so what I think pilots lead to is a situation where they are pressured to continue by their social external pressures. And they also feel a little overconfident in their ability to handle things. And so they expect to be successful, but ultimately, some of these pilots end up having an accident. Now also, we're talking about technology. This is also important to discuss because this is the future for general aviation aircraft. There have been two recent crashes of advanced Cirrus airplanes. And with a plane with a big artificial horizon picture and a big TV screen, you'd wonder, how could you become disoriented in this plane? Well, one of these was operated under VFR conditions and the other was IFR. And spatial disorientation was suspected in both cases. What I wanna ask is, are we perhaps over relying on technology? Is technology developing to the point where pilots are becoming so comfortable with all the safety things in their airplane that they don't do a good job of just exercising good solid pilot judgment decision-making? And are we perhaps failing these pilots by just training them on all the technology and not really doing very much effort into solid decision-making skills, especially for inexperienced pilots? So we've covered a little bit of the psychological things that are going on or why these accidents happen. But let's just get down to the bare bones of it here. These accidents happen, these pilots crash because they become disoriented. So we're talking about spatial disorientation. Now, spatial orientation is your body's ability to detect its position in space, okay? But here's the important point. Your body is a VFR craft, okay? Your body is not genetically built to go flying. And it's only through technology in the past 100 some years that we've been able to take planes into the sky and do all sorts of things that our bodies were never really designed to. Now, am I saying this is unsafe? No, I'm just saying fundamentally, biologically, your body doesn't know what flying is all about. And so your body can go through some confusing feelings.  And the first is the visual component, okay? The second is what we call the vestibular component. And that comes from your inner ear. And we'll talk about that in a second. The last is what we call the proprioceptive input. And that is what is known as the seat of the pants. Okay, it's the sensations from your muscles and your otolith organs and everything that tell where gravity is and accelerations are. And so these things, excuse me, these three things integrated form your sense of whether you're upright or upside down or in a turn. Now, talking about the vestibular sense, this is a close-up diagram of your inner ear. And there's three of what we call the semicircular canals here. Now, this picture does a very good job of illustrating exactly how these canals are oriented so that you determine inputs in pitch, yaw, and roll and how your body, how these organs are oriented. Now, this is an amplification of one of those semicircular canals.  And then you have this structure called the cupula, which has all these very fine hair cells in it. And the way the hair cells are attached, they're free to flow. And it's kind of like seaweed underwater, just flowing back and forth with waves or maybe wind blowing over a wheat field, how the wind just kind of flows like this. It's free to move. So the fluid moves back and forth. And if the hair's tilt, it's converted into a nervous impulse and sent to your brain, which is interpreted as movement or an angular acceleration. So the problem is, is that this structure is not exactly the best way of determining motion because of several factors. First of all, there's what we call an acceleration threshold. So if the movement is about less than two degrees per second of movement, there's inertia in there and the fluid doesn't move. So therefore the hairs don't move and you don't tell that you're turning. So it's very easy, if you were to try this in an airplane, but just by closing your eyes, it's very easy to roll into a very gentle bank, less than two degrees per second of roll, your body thinks it's still upright. How many people have heard of the leans? This is the sensation that comes from that because what happens is, you roll into a turn gently, you don't think you've rolled into a turn. Well, now if you roll out of it quickly, you roll back to level flight. However, the fluid moves and it's interpreted as a bank to the left, even though you're still upright. So now you're gonna be influenced to bank the airplane back to the right now. So there's a disconnect between what your inner ear's telling you and what you see out in front of you. The second problem is, if the acceleration becomes constant, over time, the fluid is no longer accelerated inside the canal, it slows down due to friction and it ultimately stops. So you could be in a prolonged constant rate turn and not think you're turning. Even if you rolled in quickly, initially you'd feel the roll into the turn, now you're just turning, no change to the roll rate, and now your body doesn't think it's turning anymore. So there's some issues here with the semi-circular canals. 90% of your spatial orientation is derived through the visual input, okay? And if you don't believe me, what you could try is, you stand next to a wall, put your hand on the wall, then raise a leg, close your eyes and take your hand down. See how long you can stand on one leg with your eyes closed. Chances are it won't be that long because you've removed 90% of your body's ability to tell whether it's upright and balanced. And that inner ear is not good enough to keep you balanced for a prolonged period of time. And so the other problem is the sensations from your inner ear are so strong that they are hardwired to your brain. And no matter what your conscious mind may be thinking, that inner ear has a direct pipe to your brain through the subconscious and you might consciously know the plane is level, but your body, everything in your body is screaming and turning.  And the reason that happens, even though you know the world is flat, the ground is flat and you're trying to walk upright, your body, the fluid in your inner ear still thinks you're in a turn and so your body is gonna try to turn with it. Even though you're consciously trying to walk straight, no matter how much effort you put into it, your inner ear is so powerful. And so this leads me to my second point. A lot of people think that the whole point of instrument training and everything is to overcome the sensations that they get from their inner ear. This is incorrect. Your body is hardwired from your inner ear to your brain. You can never override those sensations. You will always have those very strong impulses. So the objective of instrument training is not to overcome disorienting sensations, it's to acknowledge that they exist and then try to fight them and use the instruments as your replacement. No amount of training or practice can ever allow you to fly an airplane upright without some sort of visual information. You just cannot do it. So what are our choices here? We know that the inner ear doesn't give us complete information about flying the airplane. We need some sort of visual input. We need some sort of horizon information to tell whether the airplane is upright. So choice one would be, stay out of any weather conditions where you would ever not have a visible horizon. Now how practical a solution is that? Not practical at all.  And so what is the better alternative but to instead learn to use the flight instruments as a substitute for the horizon outside the airplane? You've got a perfectly good attitude indicator right in front of you that gives you all the information you need that you could tell by looking outside the airplane. Now here's why these accidents happen. The pilot, instead of paying attention to the instruments or instead of transitioning to the instruments, what is he doing? Well, he's looking outside the airplane, trying to figure out where the ground is, trying to look outside like this. Even, you know, he's just floating in the cloud, he's searching, trying to descend the airplane, get out of the cloud. Well, he's got his hands locked on the yoke, but he's maneuvering the airplane without any reference to a good visual horizon. And so it can become really easy, especially if you try to make a turn now, to become disoriented and maneuver the plane inappropriately. And so the key is to learn how to use those flight instruments and be comfortable using them. Now here's kind of the most important part of this seminar and we're gonna start talking about stress here in a second. But this is an audio recording of something that happened on April 26, 03. This gentleman was flying a 172. He had between about 350 and 400 hours. And he was flying VFR from West Osh, Wisconsin to St. Joseph, Missouri. Now this gentleman is an engineer, very sharp guy, perfectly normal guy. And I wanna stress this point. He could be sitting in this room right now and you would have no clue because he is just as normal pilot as you or the next guy. And so he's a private pilot, no instrument rating, very good training, and he used the airplane not just for pleasure but he also used it for business. And the purpose of this flight was to go down to St. Joe to go see an auction. So I mean there was an objective to the flight and what happened was he was flying along and he got to about Fairfield, Iowa which is maybe about a little less, a little more than about a third of the way there. And the weather wasn't quite looking so good. Now it wasn't a perfect day to fly. There were some clouds there in the medium altitudes, four or 6,000 feet area, maybe a little lower than that. And he was flying VFR underneath those. Well, he's getting along, the weather's not looking so good ahead of him so he lands at Fairfield, calls flight service, they tell him what's up. They said, if you wait a little bit, chances are things will improve. So he waited a little bit and he thought it looked like it was improving so he started flying along. Well, here's where we pick up the recording. I need one, five, Lema, would I dare you to go ahead? I need one, five, Lema, say last known position. I need one, five, Lema, say altitude. Dodge, release the stick, go forward on the stick and then slowly back again. I need one, five, Lema, remain 7700, you have a transponder. You get level now. I need one, five, Lema, do you have a transponder, sir? I do, one, two, zero, zero. One, five, Lema, please trans out Squawk 7700. Lema, try to remain straight and you are in IFR conditions. Do your best to keep going straight and level and flying straight and level. We're gonna try to get you in contact with Air Traffic Control Center. Thank you, sir, I'm at 3,000. Okay, November, 9 or 815 Lema, are you able to switch frequencies? 9 or 815 Lema, contact Chicago Center, 118.15, 118.15, if you do not establish communications, come back to us immediately on this frequency. 118.15. November, 15, Lema, Roger.  Pretty scary, huh?  The gentleman was able to land safely and he was ultimately vectored to Ames, Iowa and was able to land just fine, probably changed his shorts and he was actually able to finish his trip later that day. Now, there are a few things I want you to get from that. First of all, what could you tell about that pilot in the beginning of the recording? I mean, that is the sound of a pilot that is in the absolute throes of panic and that is the sound of a pilot who, if he didn't pull it together, was gonna die. I mean, I would never play a recording of somebody dying in a situation like this but it had a good outcome but he very well could have lost it. Now, what did you notice about the last half of the recording about his demeanor and his tone? He calmed down, right? What do you think it was that allowed him to calm down, right? And what do you think allowed him to do that? I'm sorry? Well, on an internal level, do you think it was his training, perhaps, that allowed him to recover the airplane? The solution to this is training. The way to get through stressful situations is training. So this is gonna be the focus of the next half of the presentation. But the thing I wanna stress to you is, like I said, this guy is a perfectly normal pilot. He's since gone on to get his instrument and commercial and multi-engine ratings and he's still flying but this guy is a perfectly normal pilot. He's as normal as you or I and so if this could happen to this guy, it could happen to you too. No matter how safe a pilot we think we are, as we've seen from the psychological things, we can always talk ourself into making a bad decision. And so when we're flight planning, here's our opportunities to counteract these situations. Now, of course, this probably comes as no surprise but weather accidents almost always occur on cross-country trips. And more often than not, they occur on the last portion of a cross-country trip where someone's trying to get home. You know, we'd talk about get-home-itis or get-there-itis. Now in many cases, such as the one that we just heard, the pilot stops, in fact, stops short of his destination to check weather or something like that. And then when he's on the ground, he starts checking the weather and then he thinks about all the stuff that's waiting for him as his destination and then he leads himself to make that decision to continue on. I can't tell you how many times this happens.  You know, we're happy to lend you the crew car. You can go to spend the night at a hotel. And then the pilot's response will be, oh, I don't think it's that bad.  So a lot of times there's that missed opportunity to intervene in the accident chain. And another thing is, how many pilots get in their planes thinking that they're not gonna make it to their destination, right? Nobody gets in their plane and thinks, oh, there's a good chance I might not make it. You know, pretty much everybody's focused. Like I said, we're all forward-thinking people. We're always wanting to get to our destination, right? Now, when you do your private pilot training and you plot that course line out and everything, what's one of the required tasks in the private pilot PTS, a diversion, right? Well, the FAA did not just throw that in there to be spiteful. They threw the diversion task in there because they thought that it would be good, you know, good to actually teach pilots why this is necessary. The problem is, is probably not all instructors are doing a very good job of explaining why this is such a useful skill. And, you know, so you're flying along and you're so focused on your destination and you're direct to your destination, you may not even really be paying attention to where you are or where you're flying. You're just focused on getting there.  And so the idea is, how much effort does it take to figure out a plan B before you take off? It's really not that hard. I don't think that there's no copies out on the back there, but not this month's issue of the FAA aviation news, but the last issue, there was a very good article called The Tale of Two Diversions. And it was describing a very hypothetical situation. And it was an amusing story because you had two pilots, identical planes, man and woman, they both have to divert. Well, the woman diverts to a fairly large metropolitan airport, rental car, that kind of stuff stays in a nice hotel, everything works out fine. Bob, our second guy, diverts to a very small, rural airport and there's the only rental car available is from like rent a wreck and it's radiators broken and there's no heat and it's like minus 20 out and he can only place he can find to stay is some flea bag motel with an all night bowling alley downstairs and there's no, doesn't get any sleep and he's all tired. So the lesson of the article, however, was that if you put some thought into diversions, you can actually make your trip go a lot smoother if you have to divert. Now, for instance, if you're playing a very long trip, how might you plan your route? Rather than just drawing that direct line from departure to destination, why don't you draw that line to include some larger municipal airports along the way where you know there's gonna be rental car available if you need one or a hotel or something like that or you could even plan your route to go by a major city so that if you had to divert, you could hop on the airlines if you really had to get to your destination right away. You know, there's that old saw that says time to spare, go by air, you know and general aviation flying for good or bad, we cannot rely on always getting to our destination when we expect to get to our destination. And so the idea is you need to be able to plan to have something available to you and think about something if you can't make it to your destination. When you're flying, constantly evaluate your conditions and also develop a set of personal minimums for yourself. This is a term that's popular now, personal minimums. FAA and AOPA have some good guidelines on how to establish personal minimums but the idea is personal minimums are what is the lowest ceiling visibility that I feel safe flying in? And then if the conditions don't match your personal minimums, you don't fly. The idea is is you set your personal minimums and then you don't bust them and that's what's gonna keep, one of the things that's gonna keep you safe. Also keep in mind the fact that making a no-go decision is a lot easier when you're on the ground than it is once you're already in the air and on your way to your destination. And another thing about, and I understand this completely, you're getting the air and maybe you didn't think about where to divert. Okay, the weather's bad. Well now you're diverting to Boonville Municipal Airport and there actually is a town called Boonville. It's on I-70, East of Kansas City, I've been there and it's one of these towns where I spent the night in a hotel and I called for a pizza and a pizza delivery lady came, I was on the third floor and she comes to the door of the hotel room, she says, that's the first time I've been in an elevator my whole life. So I'm thinking to myself, okay, I guess I've walked onto the set of deliverance or something, but I mean it was, you know, I understand, you don't wanna be stuck in Boonville, Missouri, you wanna be home, right? And so I understand you've got these pressures there, but you understand you're so locked in on your destination that everything else becomes so easy to trivialize all those other safety issues. And lastly, because we know that we don't do a very good job of estimating weather conditions, how do you know when the conditions are so bad that you've gotta turn around? This is a question, sort of a hypothetical question you have to figure out the answer to for yourself. Now, here again is an example of where committing to something and seeing it through leads to further complications. As you can see down here, there's some poor guy, I don't think this is the guy, but some poor guy drove his car off of the cliff there and into the water. And okay, they've got a crane here, no problem. All right, so they got the car in the sling, they're lifting the car up, everything looks good, and then the crane flips over. Okay, small problem becomes a larger problem. So what's the solution? Get a bigger crane. Okay, so here comes the car now, and now we're gonna lift the crane out. So, now we're gonna talk about importance of knowing your terrain. This is the famous last words here we've seen this cartoon quite a million times. But we're gonna take a look here at this sectional. How many people know what this element is here? Yeah, typically, I hear the same answers from everybody, but the technical term for it is the maximum elevation figure. And this is determined for the quadrangle, that on the sectional map, and the quadrangle is formed by the four lines of latitude and longitude with the tick marks here. Now you can see I can tell you that the highest obstacle on this sectional here is 1997 MSL, this TV tower just south of the Illinois River here. 1368 ADL. Now what they do is they take this figure, they put a little fudge factor on top of it, and then that's how we arrive at 2,100 feet. Now let's say you were flying through this area. Now how many of us know our exact position on the ground within a nautical mile? It's very difficult to do, because you might have drawn a line from the center of your departure airport to the center of your destination airport, right? Well you're not actually flying on that line, you take off, you fly a nice wide turn out from the pattern, you get established on course, then you hit direct enter on your GPS, right? Well you're not flying on that line from one airport to the other, you're flying on some practically new line here. So the thing is you may not know exactly where you are on the ground within a mile or so of what you thought you were gonna be on. So if you're flying along on some just random GPS line here, do you really know whether you're gonna fly next to this tower, over the tower, a couple miles away from the tower? Not really, unless you have a nice fancy GPS like a Garmin 396 with the obstacle database and all that kind of stuff. I mean that's nice to have in everything, but the thing is if your GPS directs somewhere, you just really don't know whether you're gonna hit this tower or not. So if the clouds are 2,500 feet and you're starting to dip down closer to 2,100 feet, couldn't this be a good idea of when it might be a good idea to turn around? So the importance of knowing terrain is very important. Now we're gonna talk about stress here. When you become in a stressful situation, you go through four phases. First is denial. Weather can't be this bad. Next you go through a period of confusion or inactivity. Okay, there's no emergency checklist for flight in IMC is there. I mean, so you're thinking, okay, what do I do now? Where am I gonna go? I didn't really have a decision to divert. I'm not really sure what to do now. Next you're gonna go through a phase of anger. Anger maybe at yourself. Anger at your passengers for pressuring you to get home. Anger at flight service for not telling you though the weather was gonna be this bad. And then lastly, you go through challenge. And challenge is, okay, I got myself into the situation. Now I gotta get myself out of it. And the trick is to get yourself from denial to challenge right away. There are pilots that hit the ground while they're still in denial. And so challenge is what's gonna let you get through the situation. It's what's gonna get you, what this pilot that we heard got himself out. Now talking about stress and performance, everybody performs better under a small amount of stress.  And whenever the light bulb turns on, you hit the button. Well, if that goes on for a couple hours, you're gonna start to get bored, maybe daydream, maybe fall asleep and you're not gonna do it. So if I introduce a little stress by calling my friend Guido to come and slap you on the face every time you don't hit the button, you're gonna start to hit the button when the light comes on, right? Well, if I up the stress some more, now I just tell Guido to just slap you all the time regardless of whether you hit the button or not, now your performance is obviously gonna deteriorate. So this is what we call being task saturated. Now we're gonna compare simple and complex task, a complex task or an emergency. You can see with the increasing amount of stress, it becomes, it takes a lot less time to become task saturated or overwhelmed than it does with a simple or routine task. So this doesn't have to be a lot of stress for you to become saturated. So when you're handling stress, this is the Air Force's philosophy and I'll just explain it briefly. When the Air Force does pilot training, they have a thing called standup, okay? They get all the cadets in the room, they get all the instructors, everybody in the room, and they pick a cadet at random. So it's a very stressful situation because he's gotta stand up in front of all of his peers and they give him an emergency situation and he's gotta recite the bold face emergency procedures to deal with it. But before he can even say anything, he's gotta say, I will maintain aircraft control, analyze the situation, take appropriate action and land as soon as conditions permit, right? Put this another way, you get aviate, navigate, communicate, right? So the point is maintaining control in an emergency instrument condition is as much a matter of your mind as it is of your skill. We saw from that guy that if you panic and you get overstressed, you won't be able to recover the airplane no matter how good of a pilot you are. I don't know if you heard that, they said it could be miles off course and he said that's impossible, they're on instruments. This is not instrument flying, folks. Just briefly about basic instrument skills, the U of I did a study very long time ago back in the 40s or 50s where they took pilots that had no instrument training at all, just brand new student pilots, they put them in a simulator and they said, you've gotta fly the airplane, no visual input. So in other words, without that 90% of our visual input, how long can you fly an airplane upright? Well, they figured that the mean was 178 seconds or just about three minutes. That's how long you can fly a plane in the clouds with no training. And so the minimum was like 20 seconds. I don't know what happened to that guy, he must have just like completely flipped inverted as soon as he got in the thing and crashed. And the max was something like 600 seconds. But it's not a long period of time. The first thing that you have to do is learn to let go of the controls. Avoid that grab and grip reflex. When you get in a stressful situation, what's your first instinct?   You get a shot of adrenaline, you know, and you get stressed out. Well, as any flight instructor will tell you, what happens when you white-knuckle the yoke? You can no longer feel pressures on the palm of your hand anymore when you're white-knuckled on the yoke. And the key to instrument flying is not to manhandle the yoke around, but it's flying, I like to say instrument flying is about pressures, it's not about inputs. And the general order that you can fly an airplane, the less risk there is of you making control input that allows you to become disoriented and lose control. So the first thing you have to do is avoid that grab and grip reflection. You just gotta get your hands off the yoke. Because if you're flying the airplane looking out like this and you're all stressed out and you don't know what's going on and you gotta find a way to get down, it's not a good idea for you to be flying the plane like this. So if you take your hands off the yoke, is the plane gonna flip inverted out of control? No. I mean, airplanes have inherent stability. So if you take your hands off the yoke for a few seconds to get a hold of yourself, nothing bad is gonna happen. And then you can re-approach the yoke, calmed down a little bit and fly the airplane the way it needs to be flown. Make your attitude changes smooth and small and then relax and don't try to do too much. You just gotta get yourself slowly out of the situation. We're gonna talk about the five C's. The five C's are climb, communicate, confess, comply, and conserve. These apply to any emergency. It doesn't have to just be an inadvertent IMC. Now, climb, that might seem a little paradoxical. Why would you wanna climb if maybe descending would put you out of the clouds? Well, no one has ever collided with the sky. But a lot of people have collided with trees and buildings and things that live on the ground, right? So the higher you can get, the less chance you are of hitting something, right? What else does climbing get you? Well, if you're scud running at 1,500 feet and you try to call for help, what kind of range do you have on your radio? Not very far.  So the idea of climb helps. You can save fuel, you can pull your power back, confess your situation to the ATC, comply with what they tell you and also lean your mixture if you need to to conserve fuel. Now, kind of a final point. Under periods of high stress, we do not rise to the occasion. We sink to the level of our proficiency, okay?   And so the importance of training cannot be overemphasized. And in fact, we can see that over time, you become a lot more confident about your abilities than your abilities actually are. So if you get in a plane and this happens, chances are additional training is probably warranted. Now, I'm gonna just briefly make this point about hazardous training. I know I have to wrap up, but when you do your training, you have to train to avoid instrument conditions not to handle instrument conditions. There is a distinction here. NASA did a study with new private pilots where they asked the private pilots, they said, what did you get from your hood training with your instructor? And the problem is 21% of the pilots said that now that they've had hood training, they'd feel more confident in their abilities to handle the plane in the clouds and they'd feel more comfortable about flying in the clouds. This is the exact wrong thing that we wanted them to learn. Instead, well, 68% had the correct response and they said that they would feel more confident in their abilities, but not more comfortable about flying in the clouds. So that's the key of training. We, you don't wanna train people to be scared straight about flying in the clouds. It's not the objective. We want them to be able to rescue themselves. Now, what can you do starting today? Participate in the wings program, of course. If you follow the wings advisory circular right now, I know it's gonna be revised, but it recommends an hour of hood training. Seek out some periodic hood training or training in marginal weather conditions. Pursue an instrument rating if you like. And if you're instrument rated, keep current. That's probably the most important thing. Now, final thoughts. Obtain weather from all available sources prior to the flight, even if it's a trip that you're very familiar with. Know the tools that you can use to update your weather information in flight once you're in the air. Establish some solid personal minimums for yourself and don't bust them. Learn to interpret in-flight weather cues and deteriorating visibility and lastly, separate the mission from the goal. If the mission is flying safely, the goal is getting to your destination. So, if you want more information on the subject, there's some very good publications. IFR for VFR pilots. FAA safety publications. There's also some publications from Cami. And I'd like to thank you very much for coming today and really appreciate your commitment to aviation safety. Thank you.


Welcome to another edition of RCE. I'm your host Brock Palin. I have again Jeff Squires from Cisco Systems and Open MPI Jeff. Thanks again Good afternoon Brock. What do we got today? We have with us two people who work on the rocks cluster distribution. At least that's what I think it is I've actually I'll admit I'm familiar with it, but I've never used it myself. So I'm flying blind again They are Greg Bruno and Mason Katz and they're both at UCSD. So guys, welcome to the show Hi, I'm Mason Katz. I'm work at UCSD. I've been here for about 10 years working on the rocks cluster distribution project Prior to that I worked at University of Arizona doing security writing network operating systems It was a lot of fun and prior to that I was a 8-bit embedded developer So I'm a big fan of the the of the accumulator Yeah, good old Z80 My name is my name is Greg Bruno. I have a very similar background to Mason here for the last 10 years I hired on one month before he so I always remind him about how I outrank him in seniority And but prior to that I was working at NCR where they build and can still continue to build the world's largest databases They're using teradata technology Okay, guys, thanks a lot for taking some time Can you give us a quick rundown on what rocks is meant to accomplish and therefore explain what it is when you say a cluster distribution? Expand on that So that's that's a good question because we've had this question internally for for many years. We originally our first release in November 2000 we called ourselves the Rocks clustering toolkit and that was that was a name that we put together the toolkit part was added by Greg Gregum And my own boss Phil Poppettopoulos Name should be last name should be familiar And at the time we're a toolkit because we're really a layer of tools that went on top of an existing machine And then allow you to deploy a cluster and over the years We've become more and more automated more and more turnkey and now we're really more of a distribution We're more of a cluster on a DVD so you can pop it into one node turn it into a master for your system and Just one by one bring up your compute nodes with zero human interaction But we'll get we'll get into the details of that more Okay, so what's the history where did rocks come from how to get started? Actually Mason and I met Phil Poppettopoulos prior to working on rocks in another lab That was here at UCSD So all three of us were hired into to work for a PI there where he was investigating Windows clusters So this was in 98 99 and 98 99 when he was investigating Windows clusters and so we all came on board and we're all UNIX heads So this was we were really out of our element and so we were trying to use a lot of the windowing tools in order to You know basically put a windows distribution down onto all the compute nodes reliably They had all these these programs it did like multicast it None of them worked reliably and Phil one day said you know what I'm just going to make a partition on every node And it's going to run Linux and at that time it was Mandrake and so anytime we wanted to Re-image the entire cluster Phil ran a program out there that told the nodes go back into this one Linux partition and then basically just do a DD in order to you know, Spot all the new bits down So that was really kind of the prototype of if you want to reliably build a you know back in cluster nodes at any sort of scale Linux is your horse So what exactly is is rocks is it is it middleware? Is it an operating system? Is it a distribution? You know, how exactly would you classify it? It's it's all of those rocks includes All of the software you need to go from bare metal to fully functional application ready computational appliance So rocks for example includes all the bits from smell all the relevant bits from centos It includes middleware for cluster modern such as ganglia It includes middleware for grid such as globus or did it one time? So it's it's it's everything we actually have referred to it many times It's kind of the the hypodermic is this is the hypodermic you use to inject Software onto your bare metal. So it's it's just It's the methodology and it's the technology to wrap all that up OS to application treat everything the same applications are treated. No differently from the OS It's just how do you bundle everything and get it deployed so Check technology wise is really a distribution, but includes components from operating system all the way to application including the middleware It's a fuzzy answer So run and give us give us a sample scenario, right?  They talk to their network admin get an IP address Simple stuff like that. They go to our what where our website at dub dub dub dot rocks clusters org They pop in the DVD turn the machine. They answer a few questions. It's just like installing red hat It's actually a little easier because we remove a lot of the installation screens And they just put in their IP information time zone password Hour later the front-end comes up They will run a very simple command on the front-end when they first log in called insert ethers and they turn on these slave nodes They're all on a private non-internet routed network and they just automatically come up and it takes about 10 minutes a node you do them a Few seconds apart from each other. So really in a matter of two hours you can go from a room full of 32 64 nodes and boxes to a fully functioning MPI machine and that's that's always been our target That's where we started we started funded as NSF funded and Paki program To build to make it easy for researchers to build their own build and manage their own MPI machines And we move past that but that's where we started and that's really what we're best at still Okay, and so when you say it's about 10 minutes per machine. I assume this means a disk full load So you're you're black testing in OS and some level of middleware and software and other things like that and That must include you know loading it on to the machine itself or or what? Yeah, that's right. I mean we do all all of our installations are from bare metal and so There's a there's a lot of benefits there One of them is is that when you start from bare metal or you you're actually starting from the empty set And so when you start from the empty set, that's that's known, right? You know what the machine was going to do in the when the you know the amount of software in the set is empty And then as you start to build up you're going from known state to known state to known state to known state And so with at the end the configuration and Characteristic of that machine is fully known because you started from a known state and you transitioned all the way through known states Another benefit of having the disk out there on the node is I mean simply just to cache the operating system And also to cache applications. So if you're running at scale Having that disk drive there in order to help the nodes You know relieve the pressure of the network just to go out and grab some files or some common files Or some read-only files is we found that to be a huge benefit So in the religious debate of disk list versus disc full it sounds like you guys are coming down on the side of of disc full Well, I don't I don't think it's a religious debate. It's it's it's what do people prefer We've chosen as a group just not to worry about the discless problem We we understand that people care about it and we've had we've collaborated with one group Kessler start University Good friends of ours in Bangkok who created a discless role. So they did add discless support to rocks The project there that they were with is no longer funded. So that's that's a support That's some support that went away, but for us. It's not a religious argument. We're very open to discless It's just not something that we've focused on Okay, fair enough You mentioned the word role in there. What exactly is a role? so role a Rolls a meta package really. It's it's the easiest way to think about it. It's it's a meta package, but it's not just Configuration and bits for one machine. It's a meta package for your entire infrastructure for your cluster for your data center even because in that role you have Individual packages and in the in the context of Linux. There are PMs. There's a little bit of XML in there That's actually maps directly to Whatever the automatic installer languages of the OS is in the case of Linux.  I'm sorry just to follow up a little bit there with basin. He mentioned part of a role is there are a Collection of XML files now most of these XML files if you if you looked at them It's it basically has one nugget of functionality and an example. We always like to use SSH So we have an SSH dot XML file inside inside the base role and then there you'll see the configuration of SSH on you know on every node because that's a common Function that we have on every single node and then there's another file That is it's it's a graph Specification and so all of rocks is the configuration of the node is specified As a graph and we have several or I should say the root nodes of the graph or what we call appliances So one root node would be your front end another root node would be compute another root node would be this tile and when a node asks for a kickstart file we Send it over to a CGI and that CGI looks at that node and says oh this is noted IP address Whatever we look up in the database to see what What appliance it wants to be we go to that root node of that appliance and then we do a graph traversal of And this graph traversal is accumulating all of the XML files across that graph and so Getting back to what is a role a role is this thing that when you add it to the front end it actually splices itself into a Machine so if you start off as an MPI machine and you bring the vis roll on earlier And you also want to create vis tiles from that same rocks machine What happens is that that graph XML goes and it splices itself into the current base role Okay, so everything lives in a role the OS management tools the rocks meta tools themselves Everything is belongs to a role That's right. Everything. Everything's a role Okay, so everything's a role so Rolls are made up of these XML files and RPMs or tar files. I mean do they have to be a certain type of package format? Yeah, so even expand on that Do you so do you focus on Linux or do you do other operating systems as well? So like would RPMs be the Linux way and and have different kinds of packages for for other operating systems? Yes, we focus primarily on Linux, but we've added support for Solaris we had a support for Solaris one because We were funded by it We ended up hiring a new developer to our team a few years ago His name is a new bridge Indra who's done an excellent job at taking rocks which started out as a very not just Linux, but very red hat dependent anaconda dependent system and Mapping kickstart Ideas into jumpstart which actually isn't that bad because most ideas and kickstart came from jumpstart So in many ways everything that rocks was doing was being mapped back to where the ideas actually came from from underline OS um, so we To try to focus on the OS OS part of the questions. Yeah, right now we do Linux and Solaris We view red hat is kind of the dominant definition of Linux. That's not to diminish the significance of other distributions just to kind of Focus on the reality of we've us being a very small group being being government funded And we can't really justify spending our time to support rocks to yet another Linux distribution When it doesn't really functionally do very much for the users. That's not not to say it's not a great idea But it's not something we'll do in Linux. Yes, everything maps to a package an RPM and solaris everything maps to a pkg Just a package everything my maps to jumpstart jumpstart files layers everything maps to kickstart file and Linux rolls themselves are Packages from whatever the native OS is and then the XML is Really our own invention, but it really looks a lot like kickstart language and then that can be interpreted that can be parsed and Converted into either kickstart language or jumpstart language. So we don't do we don't do tar balls Because we really want to take advantage of the native packaging system and being able to do package dependencies So we do we have very easy easy tools I mean one command we could say rocks create package and it will take a standard GNU configure make install package And it'll build an RPM or even a macOS package or solaris package for you So the fact that we're focusing on RPMs and PKGs isn't really a constraint. It's just the technology We use we use RPM more of as more of a Software distribution methodology, we don't use a lot of post sections and packages or anything We're just using it to wrap up the bits. It's a it's a it's a bit transport as far as we're concerned So one thing you threw in there you said macOS package after you said you supported only Linux and solaris So what gives is that is that a hint of things to come or? It could be and we were again. We did solaris specifically Well one we're interested in because solaris brought to the table a kernel with true threads So from a technology, there's some real real Interest that we have from from HBC and even non-HBC context Mac OS we did well, I mean we're sitting here on Macbooks So it's kind of nice to every now and then build little bits of rocks on the Mac So there is there we have we have the support to build anything and rocks can be built into a macOS package And just to follow up on that on the Mac story there was a a guy pretty high up and Apple who was here at UCSD and he was making the rounds and talking to a bunch of different groups, and he said hey How about rocks on Mac OS X we said it sounds great, and he says okay And we said all what all it's gonna need is a funded developer for two to three years, and we haven't heard back from him So in what you describe I'm gonna jump back a little bit in the conversation what you described for roles It sounded like there was at least some level of overlap for package dependency resolution because that's always a sticky problem And it's and it's such a sticky problem that people have invented multiple systems to do such things like with yum And yast and and various others. How how do roles compare to that? Is it is it a different thing or do you use those underneath or or what do you do? Yeah, we use those tools underneath when we write RPMs Most of the most of the things that we author for rocks actually don't have these very strong dependencies But rocks is I don't know probably 99% third-party software that we're integrating and configuring for people So for those things that we have to compile Sometimes we specify the dependencies sometimes we don't But usually we rely on in Linux now and anaconda now it is all young so young is doing the dependency resolution But that's just the resolution on what the bits are that's only half of the story The other half of the story is what is the configuration and that's where rocks comes into play It's very easy to say I need x y and z on all the nodes in my cluster That's really easy to do you almost fantastic at that where young Falls down and it was never designed to do this is I need package x y z on my cluster And this is my IP address and I need this service on from at this machine I need that service off of that machine. It's the configuration that rocks focuses on so that XML those Kickstarter files it's really All of the post configuration that everything you would have to do after you run yum to get things working and It does things like look up variables in the cluster database to figure out what's going on the cluster and what things should be doing and Just pre-configures Everything for you and their dependencies that we care about in the XML and the pre-configuration You have to append to some files before you pin to others. You have to start some services before you start others So that's where we do our own dependency tracking. We don't really we don't address the package dependency problem itself Because the underlying technologies do that for us and we're big big proponents I'm not reinventing the wheel and if I if I could come into work and not have to write code That's a really good day so when you say Not reinventing the wheel for the standard cluster tools people and end-users used to working with MPI libraries batch systems Compilers you're not making your own. You're just kind of packaging up things like Torque SGE GNU PGI Just kind of handover work other people have done in a roll form But let's go and open MPI of course, right? Yeah, that one specifically. Yeah, sorry I had I had to is just that mandatory. Yeah, absolutely. Oh, so all those things are rolls We are completely agnostic on things like that we rolls themselves actually came about because of a conflict no not a conflict but Differences of opinions sometimes a conflict in terms of what local workload manager do people want to use We we started out in rock supporting Was it called PBS then or open PBS or Whatever name that they had I don't know. It was pre-pre-torque obviously. I guess they were just PBS So we started off on PBS and Sun Grid Engine was started to come out and we ourselves became big fans of it because we found it easier to run and We started to have a distribution in rocks where we would give people the bits and they would have PBS and Sun Grid Engine And then they'd have to run some crazy command to shut one off it was completely ridiculous and We finally figured out how to Turn rocks into a component model and that's that's really what roles are roles are the components in the component model of rocks And then we made PBS of a role. We made SG a role We're now now torque is a role. There's commercial roles that do PGI and Moab and stuff like that So it's it's yeah pick your favorite middleware and somewhere there probably is a role for it Do only you guys create roles or do other people create roles? So we create, you know, kind of the base system and some of these these ancillary Middleware things that Mason was just talking about but then we also have a guy long-time developer Who's been with us up? He's actually north of the Arctic Circle in Norway And I've been there in June when the Sun never went down and it completely freaked me out but that's a side story and So Roy Trump Roy Trump so Roy drags it up in Trump So produces the PBS role because as Mason was talking about there was a time where we had PBS and SGE and We decided to drop PBS for SGE and Roy said hey, I really like PBS So he's been the maintainer of that role for several years now there's some other Projects here at UCSD which they make their own role. So after we we come out with our distribution They bundle up their own tools and put it on. There's you know, some of those guys are doing Quantum mechanics codes and other guys are doing visualization codes with that and then there's also commercial roles out there so Mira com again, we initially did the mirror net role and We conned them into taking it back and they've they've done a great job with it They've they've been pumping that roll out for you know, the last couple years as well And there's other commercial entities out there that are also making roles cluster corp is Another one that that I know of that are making roles out there and there's going to be several other institutions and other Commercial companies that we have no idea that they're actually making them and using them day to day So actually you touched on a couple of things there too So this is obviously a wide-reaching project, which is a really really good thing So you've got some open-source guys. You've got some industry players and so on how how do you function as as a project? Do you have core developers? You know only there in your group or do you have people who submit patches to you? I mean do you consider yourself a true open-source project with developers and random places around the world or or does it mostly focus on? External people making roles or or how does that go? So you touched on a lot of a lot of things there and actually this is one of my favorite things to kind of talk about about rocks So rocks itself is is Is open source? I mean it's it means the basic definition We release all resource code. You could watch our development day to day We don't necessarily we usually say we're open source But we're not strictly open development And what we mean by that is if you go around and look at kind of the key open-source projects that have truly made an impact And in some cases even change of the world They are actually open development There's a very small group of core developers that run the project and as it scales out You have a very small group of lieutenants that are in charge of accepting patches from the community and Usually rewriting them or deciding what to do. So it's it's it's a tough line to walk It's how do you how do you go through and? We have a vision of where a rock should be and we want everybody to be very welcome to the development community So how do you stay on that vision while at the same time pulling in developers from the outside to help? You march towards that and it's something that we struggle with every day And it's something that we we really want to do much much better at but we are in terms of the core of rocks we know exactly where we're going and we're quite opinionated about it one of the reasons that we created roles to decompose all these things was to open up our development community and the the success that we had of having Roy Draxith volunteer or be drafted to become our first external role developer was huge That was a huge one and since then I think we've we've come to a point where kind of roles are now 50-50 UCSD external commercial and industry. I mean commercial and non-commercial So We're like to kind of go back. We are open source. We publish everything we do on the curio We're looking at other tools for this to kind of use better tools We're all CVS guys internally So we're trying to move trying to Change for the times and move to better systems to get get people to be all contribute better. We do accept patches We have contributed patches. We have put patches back into our code We probably have a 50% hit rate when people submit patches on whether or not the right thing to do We've had we've had good success and bad success. We had a group in Singapore who were the first guys these were actually the guys that introduced us to sun grid engine one of the guys name was on a G named Naba and He he did this is pre-rolls He put SG into rocks and converted us to it and those guys were were just an absolute blast to work with for a very long time So and then we've had we've had guys that have tried to convert patches that would They were great because the patches worked for them, but they didn't work for community as a whole It's very easy to raise solution that works for your cluster It's really hard to raise solution that works for every cluster out there including clusters. You've never imagined and that's that's what we're trying to do So that's actually sounds very similar to a lot of the things that we wrestle with in the open MPI project as Well, you know getting patches that are are good for somebody and not for everybody and so on and telling the line of Getting new people in but you know staying on vision It was you you've put to words very well what I have not been able to express well, so thanks I hope you don't mind. I'm probably gonna end up borrowing those words from you Yeah, what's it's it's really hard open source is difficult a lot of people there was so much height between around kind of cathedral and bizarre release early release often and I'm frankly all this is nonsense. It's is great for a application We don't ship an application and open MPI's on applications middleware something critical to the system If you're shipping one little web service. Yeah release early release often we're shipping a system We can't do that. We have we have to have a formal release process Where we start a beta we go in for a month of beta We do code freeze before that for a month and we're we're a for lucky a two-release a year group We're struggling we're struggling with the question and we want we're struggling with technology to figure out what technologies do you Do you use to attract more developers because we our intention is not to be closed development It's just the way things are turning out. We're trying to expand Yeah, we wrestle with the same thing. I mean open MPI I assume very much like your code base It's it's complex and there are subtle reasons why things are the way they are and it's just something you have to work in the code base for a year or three before you can Get the scope of what is happening and I assume this is similar is very true for you And so even though you have very well intentioned people submitting patches. You're like, hey, this is great I appreciate the spirit. Thank you But you know that that kind of thing and how to not turn away people and get them frustrated because of those things now I I do actually want to touch on one thing that you said in there That was very interesting to me. You said that you're using mercurial, but internally you're all CVS, I wonder if you could explain that and your rationale for that a little bit Let me let me just start by by saying Our boss philpapodopoulos has a great line And he says I've been using CVS for 20 years and CVS has never lost a file for me And so that one really resonates with me because I mean I've been a long long long time CVS user And again since we're so small That we don't run into the big difficulties of you know, the branching and then trying to to deal with the Conflict conflict resolution or conflict resolution in the code not with the people and so if I mean all you have to do is walk across To the next guy's office and say hey, you know, what are you working on to to try and manage those types of things? I mean if we were split Geographically CVS may pose a lot more problems than it does today But in terms of technology, we Well, I'll back up there was Many years ago When I worked in industry there I had I worked with an absolutely fantastic source control system by sun microsystems It was part of what they called spark works. I think it was right about the time they were switching from bsd to system five And this was a distributed source control system built on top of SCCS so obviously commercial But it had this we had we had some consultants across the country and we had the ability to Take a repository pull off an entire clone that was still hooked up for diffs And they could operate on that remote repository commit back there And then once a month they'd come over to um, this is when I lived in Tucson, Arizona they come over to Tucson and We'd all sit down in a room and move their Their diffs back into the tree and it was all the software took care of all this So I've I've been a huge fan of distributed source control for 20 years And I don't know what sun did because the software was fantastic and it disappeared from the face of the earth and mercurial get Bizarre these guys are just now starting to come back to Well tools we had 20 years ago. So we looked at all of them. Uh, we looked at bazaar for a while I really liked it. I loved that it was made only mainly python Because that's what we do most of our stuff in And it turned out to be just it was just slow. It was just unusable Get very very interested when we first looked, uh, it was just too hard. It's matured a lot We're looking at it again now Starting to do a a git role for building git appliances So kind of a hint that we may be moving towards that Uh, and we looked at looked at mercurial and mercurial That looks pretty good. It's it's well documented. It's easy to use But what we're doing right now is we're committing everything to cbs internally And once every 20 minutes, I believe we're doing a cbs checkout into a mercurial Repository sandbox clone I'm not sure what the terminology is and then that's doing it at hg add L I believe the command is to kind of update that mercurial mercurial repository and then people can check out from there So there's a 20 minute latency from from our hg repository from uh from our cbs repository And we're using the now for remote developers We're drag set used to have cbs access and we kind of pushed them to mercurial Again, we drafted them the first time So we figured we draft them as our first guinea pig on a distributed source control and it's worked pretty well And a new presenter is a huge fan of it. So he's been doing a lot of his Solaris work in mercurial. So hopefully the Hopefully the young guys can teach us. Um, how do you develop it the right way? Okay, moving on from that What is some of the um Upcoming features you would like to add to rocks that aren't there now or maybe are in, you know, your beta branch or But what would you like to see rocks do? Oh, wow, that's uh, that's tough. Um I mean where where we're focused right now is in the virtualization space So for the past two years we've been developing tools to To take the tools that people have come to to know and understand and rocks on building physical machines physical clusters To have that same tool set support virtual clusters and to have That tool chain be almost indistinguishable between the two. I mean, obviously, there's a little bit of this, you know differences and being able to specify what a vm is but after that We want we wanted all of our kickstart stuff to work exactly the same and we also wanted, you know, how we deal with distributions how we deal with roles and so, um We're We're actually funded by the nsf To do that. That's what our proposal was focused on was virtualization So our first release of that was just being able to handle virtual compute nodes Then the following release we had virtual clusters So you could deploy a virtual front end which installed your virtual clusters or your virtual compute nodes And then you could have multiple of those things all overlaid on the same physical machine And also the kind of the last part of our proposal And we've done some prototyping with is how do we bring up compute nodes in ec2 and make that part of your cluster and Phil Poppidopoulos is the guy who's leading the charge on that one And so we see that as As a very exciting opportunity where people would have their physical cluster in their lab And either using the physical cluster or they overlay a virtual cluster on top of their own physical cluster And when they for those times and they need a little bit extra cpu power That and if they have an account on ec2 and if they have a credit card associated with it That we have the toolchain that allows them to easily spin up some compute cycles out there When they need it for those peak periods So that's That's I mean, that's the that's the big area that that we're looking at these days So how much of that work is available right now? The ec2 stuff is kind of proof of the concept that's stuff that Some of that's you know, so I got a quote line from from phil. I'm going to quote a line from our boss Uh, it's if it's not a cvs. It doesn't exist A lot of that stuff is not in our source control So it doesn't yeah, it probably is somewhere Yeah, it doesn't really exist though. Uh, we yeah, we're the ec2 stuff is far out there. We're probably Every release we try and say we got every developer gets one feature to add to the release Um, I already know one feature that's going into the next release. Uh, I don't think the ec2 one's going to make it So I think we're talking two releases out, which means we're a year away Year away Not too bad. I think a lot of people have been you know, work on that problem And I haven't seen many solutions that I've liked yet So, uh, it'd be interesting to see a couple other approaches people use From there, what is the strangest use of rocks? I've noticed some of the roles you've mentioned have been kind of Not what I would normally consider a traditional hpc kind of cluster node or head or anything like that What's what's one of the oddest roles or uses you've seen Well, just to kind of tack on to that I'm glad that when you went to our site and looked at the roles some of the roles surprised you Because I think probably the conception and it's probably the majority of the clusters which are spun up with rocks. They are hpc clusters but We're actually more than that where we can't support visualization display walls We know people who are using rocks to just manage the desktops in their lab um And so it's it really it really is more than hpc. We were I mean we we do what we can to make it more than hpc But to back up to your question of what are some of the strangest uses of rocks where One message that came through on our discussion list was um, this guy said Hey, I downloaded your code I installed a four node cluster in my garage. Now, what do I do? And so we looked at this and just said I mean it was funny that the guy took some took our code with no real purpose He just wanted to get the code and he wanted to learn about clustering and he just he spun it up in his in his In his garage and he just sat there and said Okay, now. What do I do? And he was like he was twiddling his thumbs there So I that was that was kind of a strange comical use of it but kind of a really cool use of it is um a guy who I talked to and I Escapes me where he's from but what he's using the cluster for Is to do sound processing in a room? And the processing that they're doing is they want to recreate the sound of these great Hauls Where you know these these these fabulous, you know live performances have been done So say, you know, hey, I want to be at arena x y z That they they they take all the sound and we they pump it through The cluster and then it comes back out and it replicates what the sound would be like at that venue And I just I just thought that was just a a really neat a unique use of clustering technology Well, and my current favorite today is we have a Probably a 40 tile visual being built in Saudi Arabia right now Not strange but kind of interesting And that's actually going on right now What's the largest rocks install you've seen most of rock systems I'm familiar with are medium sized What was the largest one out there? Yeah, so that's a tough question to answer because the largest cluster Um, we're not we're actually not quite sure because we have a Volunteer registration page that we ask people from time to time if you're using our software Please go to this page and register your cluster there because we use that data to go back to the national science foundation To say, you know, basically to beg and say hey look Look at look at these people that are using our software that you're funding us It's it appears to be, you know, fairly widespread and it's being used You know for all all sorts of different reasons. So please keep giving us money so we can continue to pay our mortgages So we On that page. I believe there is a cluster in Germany that has about 8 000 cpus So you can imagine it's at least it's at least a thousand nodes. Maybe 2 000 nodes um, we know of a cluster out in uh, the university of Nebraska at Omaha. That's all 1200 nodes so um, oh and there's also one Roy draxeth from university of trumpsa who we've mentioned a couple times already He's got a pretty large cluster. I think about a thousand node cluster as well up in uh, norway So you guys have mentioned a couple of times, uh throughout that, uh, you know, you're you're trying to your roots are definitely hpc But you're trying to grow beyond that and you've given at least one example of you know virtualization But you've mentioned a variety of other appliances as well. Um, so what what are you doing? You know to what else do you want rocks to do? What else are people? You know commonly and maybe uncommonly using rocks for besides hpc Again hpc is our core. It's it's where we came from. It's where most of our users came from and still are Our ambitions are for rocks to be a a general purpose Uh data center controller and that's really what it is a cluster just happens to be A subset of what's in a data center, but it rocks again rocks is the hypodermic for putting bits and configuration into your system So we have the The easy the easy path away from hpc Which several of our users have taken is how do I take rocks and just run a computer lab? Because it's still a cluster. It still has a private network and we have users that will set up a nice computer lab from the undergrads Lab lab a comes in at 10 a.m. They pound away on the systems for a while 1150 hit 1050 or 1150 hits students leave And they re-image all the machines to get rid of all the stuff that students do to machines that you don't want on your network And then it's ready for the next class So that that's kind of a simple example to to expand beyond hpc, but we're trying to figure out how to generalize more I We we handle This is just one example where we've we've moved away if this cluster is An open gl machine as opposed to an open mpi machine other than that. It's so the same A database clusters or just clusters of postgres servers or minusql servers We build clusters a cluster doesn't mean hpc Okay, so Okay, and we get some contact information mailing list website irc channel Where are you guys available? Our website is www rocks clusters.org And all the information is there we have to we have a mailing list that you can see from our docs We have every week. We run something called office hours We're one of us and in so far. I think it's been 90 greg bruno Maybe 99 greg bruno Is on for a week on skype for to answer your users questions. Our main list is very active. We've got 2000 3000 2000 2000 users on our main list right now So that's Signal to noise is pretty good And just one more thing there is yearly in about mid may we put on Rocks of palooza And so that's our all hands Meeting we've we've had a couple of them here One up in amd at sunnyvale one in singapore And I think that's all and we've had four right we've had four and We haven't planned where the fifth one's going to be it probably be at ucsd But so every year we have a meeting that lasts about a day and a half or two and a half days And we try and bring some developers together and also some new users we have We generally have like a Basic track that kind of gives an overview of clusters and rocks in general And then we have more of an advanced track where people can Really understand the role structure and how to build their own roles And that that event is always a free event for academics And we usually charge commercial users a little bit so we can pay for the food and take care of everybody


The ancient Romans called Rome the eternal city as they believed it would go on forever and indeed this seems to beat the case. Even if you have never visited Rome before you have probably seen the city in films and are familiar with some of its monuments and landmarks such as the Colosseum, the Roman Forum, the Pantheon, the Trevi Fountain, the Spanish Steps, and of course the Vatican City. In fact, the entire center of Rome is a UNESCO World Heritage Site. Rome hosts many festivals as well. Exhibitions, concerts, expo and other shows and events take place throughout the year and themes include everything from technology, fashion and modern art to theater, music and dance from all over the world. The city is brimming with history, arts and culture. From the opera to pop culture, from pizzerias to nightclubs, Rome has it all. Once was the heart of Western civilization, Rome is now the soul of the European Union. Above and beyond all the sightseeing you can do, you can also enjoy a leisurely stroll down the narrow cobalt streets of Trastivir or Monti, enjoy delicious pizza, pasta and ice cream, and wash it down with some top quality wine. You can escape city life in one of Rome's large green parks, relax by the Tevier River or hop on a train and go to the coast in 30 minutes. Best of all, you can become more than a tourist and get to know the vibrant Italian lifestyle and friendly locals. You can experience Rome. The Rome TEFL Tesol certificate course is demanding, but if you are going to study somewhere, you may as well to do so in pleasant surroundings surrounded by culture and amazing views. Another good reason to consider Rome for your TEFL Tesol course is that Italy has a thriving English teacher job market, from the moment you arrive you can start building your future. All roads lead to Rome after all.


I'm Dr. John Lieberman at Mid-Michigan Medical Center, Midland. I am a gynecologic surgeon here, part of our robotics surgery team, along with many other skilled surgeons and multiple specialties. I wanted to share with you some of the equipment that we use to complete our surgeries. This is one of the trocars that's placed through a skin incision. It's only eight millimeters in size, so it allows for us to complete our surgeries through very, very small incision sites. Once this is in place, then instruments are inserted. We have a variety of different instruments that have different capabilities and different purposes. They simply are inserted into the trocar and into the patient's abdomen. These work in concert with each other and a camera, a high-definition three-dimensional camera, to complete the surgery. This type of surgical approach provides a multitude of benefits to the patient, both in terms of surgery or both in terms of safety during the procedure, decreased blood loss, less post-operative pain, less manipulation of the tissues and the abdominal wall. All of those things result in a better experience for the patient. There's a quicker return to their usual activities and work, less hospital stay or shorter hospital stay. All of those things combined really give our patients outstanding outcomes and satisfaction with their experience in surgery. Here at Mid-Michigan, we're very fortunate to have two DaVinci robots. We have one of the original DaVinci's, the SI. More recently, we've added the XI version, which has multiple added benefits to it. I feel very privileged, very fortunate to be a part of an outstanding robotics program here and be able to offer my patients the latest in technology to have minimally invasive surgery.