My name is Brandon Bailey. I work within the cybersecurity subdivision at the Airspace Corporation, and we're excited to be here today in conjunction with the Space ISAC and the Airspace Village to be discussing the topic on some spacecraft research that we've been working on, specifically related to exploitation of some of the implementations upon the spacecraft. So we'll jump right into it. I would like to refer people to the paper linked in the front for more information about how you can apply cybersecurity controls for spacecraft. So from an introduction perspective, when you look at vulnerabilities in information systems, particular satellites, they're often overlooked in the wider discussions when you talk about critical infrastructure, but that's changing quite rapidly currently where there's a lot more people discussing spacecraft, vulnerabilities, ground system vulnerabilities, and how it relates to our critical infrastructure as a country. So there's been a lot of misunderstanding and somewhat complacency in this regard because there's been a lack of real live events that's known in the public space of attacks against space vehicles. But the space systems are similar to other systems. One that what you would kind of know about is the industrial control systems or operational technology field. So you see some good parallels that can be drawn between the spacecraft and the space systems arena with that. So malicious insiders, external threats, splotching threats, those are all various threat factors that people can get into effect spacecraft from the sensors to the data being uplinked or downlinked. So there's a lot of different areas where you could have impact that could disrupt military government or even commercial activity for our day to day activities. So the threats are changing quite rapidly in this arena. So there's a lot more knowledge about space and things of that nature. So, and then when they would have happened there'd be an absence of warning. So there's not real time, consistent communication between ground and space a lot of times. So there could be times where you're not aware that there's a tax ongoing and the attribution is difficult in the IT world and it's even more difficult in the space world. And there's definitely needs to be more research in this area. And that's one of the reasons why we're presenting this material is, and this is I'm assuming one of the reasons why the aerospace village exists is because there's a lot more collaboration and research that needs to happen in this arena. And the CubeSat arena is just exploding as we all know and the barrier to entry into space is being reduced quite extensively. So the good thing is, in recent years there's been some open source technology that's been put out there that can help people learn more about space, space communications, space protocols, things like that. So Cosmos, which is linked here. NASA's core flight executive or core flight software open SAT kit was put out by NASA to help people get started with CubeSat's NOS 3 was put out by NASA's independent verification and validation program to help people build simulators for space. So there's a lot of good open source material out there now so people can actually start researching this from the cells, which is great. But I will caveat that with one thing is please change your default configs if you're pulling the core flight executive or core flight software from GitHub and running it accordingly. You wanna make sure that you can change those default configs. So let's move on to some basics. So for people who aren't necessarily living or breathing space communications and commanding and telemetry in that terminology, here's just an introduction slide to kind of explain what we're talking about. So command and data handling or refer to as CDH is essentially how data is relayed to from the spacecraft. So that's where the encoding and decoding happens for the commands and the telemetry. There's kind of generically four pieces of a space system. You have the user terminal where you do the commanding and the telemetry and process telemetry that runs the ground software. There's some sort of front end processor that does the conversions of the information from the ground system to get it connected to a modem or an antenna that you can send the information to the spacecraft and get it back down. So from a cyber perspective, the attack surface is quite large because there's a lot of standard IT equipment in there, you know, like with the ground terminals, the front end processors, there's a lot of operational technology, industrial control systems that operate within these facilities that run these systems. So our control the antennas. And of course the RF link is obvious one. So something to note on this diagram is the bulk encryption component is kind of left out for people that are more accustomed to satellite communication where you have cryptos on each end. So I left that out just for simplicity, but sometimes that you do have that bulk encryption there in the middle. And this is just a quick animation of kind of what this looks like where the user would say turn this heater on and then the software translates that to some hex numbers and then it gets converted by the front end processor to get framed appropriately then sent to the modem to get modulated up to the burden back down. So that's just a simple animation to kind of articulate what we're talking about here. And slide five here is something, paper that was put out by NACIC, the National Air and Space Intelligence Center, which is a pretty good resource if you're wanting to understand the high level architecture of space systems and where cyber threats could have an impact and the types of cyber threats there are. So in this diagram, it's basically just outlining that you have the user segment where you have the ground software terminals and then the interacts with the ground segment where you have the antennas at times or in the link segment with the RF and the space segment for space. So vulnerabilities exist in all these segments and there's cyber threats in all these segments. Obviously the user segment is the highest likelihood for cyber attack given that it may have some level of internet connection where it's operated by people. And we understand as being a security researchers or testers or red teamers that the people are generally the weakest link in the cyber paradigm. So the user segment is one of the easier areas to get in. But we're gonna focus more on the space segment and the link segment specifically because that's where there's less research and less publicized information. So we're gonna focus on three types of attacks today with some demonstration screenshots and to discuss how they work, what they are and how you can mitigate these things. So we're gonna talk about a replay attack which is where you record the command signal and replay it to the vehicle and see if it processes it. So we'll do that and we'll talk about command link intrusion where you basically simulate modulating your own signal to a satellite and seeing if it processes those commands. And then we'll do a denial of service where you could do some GPS jamming to maybe cause the denial of service condition on a satellite depending on the implementation for the satellite. So we'll jump in to the command replay first and discuss that attack vector. So in the real world which is not what we're operating with this demonstration, we're operating with in simulation but in the real world you have your ground terminals and an antenna that sends a RF signal to a satellite. So in the real world, if you were to do a command replay attack you would need some sort of sniffing device to sniff the RF signal from the ground antenna while it's transmitting the uplink to the satellite. Well, in our simulated world cause we're using some high fidelity simulation capabilities that have been built that we can use simply UDP gets translated from the antenna to the satellite. So we're not sending RF signals from a ground station to a simulated satellite. Although you could do that but for our purposes we're gonna use a simulator that just essentially takes the TCP IP traffic from the front end processor and then it translates it to UDP and then sends it to a spacecraft simulator that's running. So if we're doing that with UDP then we can obviously something as simple as TCP nut for our research. So that's kind of how we're set up. So this is the simulator that we're using and it's got some real components to it that's runs in operations today for various missions that are flying. So there's a ground software component there's a front end processor there's the ground station SIM that is used to translate the information to UDP and then that sends it to the spacecraft that has a dynamic simulator with it. So we're using real ground software in this simulation we're using real front end processor software. None of these have been modified they're using the same software baselines as operational missions are using it. It uses the same communication protocols CCSDS TC and AOS. So you can Google those if you wanna learn more about it we'll get a little bit more into that detail later but we aren't using any type of encryption so there's no point to point encryption or bulk encryption anywhere in this and we'll explain why because that's the point of the demonstration is to show when you don't have encryption what can happen. We have some real flight software that runs on several operational missions currently and it's but it's cross compiled for Linux. So it's not running on an embedded processor or a PowerPC or an ARM or some FPGA it's running just in Linux for our purposes because we're just focused on the software testing at this point and not being too high fidelity but there are technologies that exist where you can get super high fidelity and we've used those in the past for several things like things like CIMU or TSM these are different technologies that you can use to have higher fidelity simulations to run software like compiled for target so those can be integrated in these simulators as well. So the ground station CIM is kind of what mimics the RF transmission for us but for us it just translates to UDP as I said earlier. So there's a simulation that's out there called 42 that's put out by NASA that helps do the attitude, the orbit dynamics and the environmental model. So that helps us understand the orbit and how we're progressing there. So here we go. So the, and you can build your own one of these with some sources from GitHub. So these are linked to your OpenSatKit and NOS3 you have some great components that you can construct something similar for yourself. Okay, so for our specific demonstration for command replay, our satellite in this scenario orbits the Earth and is scheduled to up and down link when it's over top of three different ground sites one in California, one in Spain, one in Australia. So one would be Australia's in Canberra, Spain, Madrid, California, out near LA. So the point of our demonstration is we're gonna capture the UDP traffic with TCP dump as it's leaving the ground station during the uplink pass. So this mimics an RF sniffing capture and then we're gonna replay that with TCP replay. So we did a quick Google search on the internet looking for what type of hardware and technologies needed to reproduce this type of an attack in real life. And basically is around $7,000. You could get some equipment that could record capture and transmit signal to a lower orbit. So it's definitely achievable for, you know, people who have a little bit of money, but then with the advent of some of these new technologies that are coming out today, like AWS ground station, the barrier could be much lower. So you may not need so much equipment. You may need sniffing capabilities, but we're a little unsure on what practices are gonna be in place for AWS in the future. So to prevent these type of things from happening, but with that being said, it's just trying to articulate that the barrier for doing communications to a vehicle is getting less and less. Okay, so setting up the attack. So in our simulation, the satellite is visible to the ground station over the Madrid ground station. So it's there. And then in the bottom middle here, you see a signal lock in that window. It says, okay, it's locked, it's acquired. So it's sending the connect message and then it's uplinking the command. So you see several commands being transmitted. You know, that's a bunch of numbers there. But what that does is you look in the far right, you see the flight software that's running. That's kind of like the UART window of a flight software where you have the flight software responding. You see it processes these no-op commands. So in this example, we just hit some simple no-op commands, which is kind of like the ping in our terminology. So just to see if the application running on board is alive. So upon receiving that command, the flight software indeed does process that command and downloads the event messages and the command counters increment. So you see here in the ground, you see that the event messages are processed. It says, hey, there was a no-op command and it's downlinked to the ground. And then you see the command counters increment accordingly. So you see that the CMDPC, which is short for command process, for each of those applications that were sent commands process to increment to one, it was zero. So it goes to one. Now, the satellite in our scenario is in transit between Madrid and Canberra. And you see in the ground station simulation that it's lost signal. So it's no longer in view. So now here's where we could actually do our replay attack. So in a real-world scenario, you would need equipment, obviously in the Madrid area to capture the RF traffic. And then somewhere over India, in this example, you would need transmit capability to be able to send, transmit the information to the satellite. So in our scenario, it's a lot easier. Obviously, because we're using just the UTC-free replay and here's just a simple command. It replays the PCAP file that was captured. And then you see the flight software respond accordingly with the same event messages within the UART. But what you don't see is any event messages on the ground station because you were in between passes at that point. And the command counters, you notice when you establish the downlink again over Canberra that the command counters went up. So the flight software is reporting, hey, I received another command and in between ground passes, which would be odd. So from a defensive cyber operations perspective on the ground or a C&D perspective, if you're not monitoring certain key telemetry points on your vehicle, you may not be aware that someone has tried to make a connection or send commands. So it's very important to understand what the telemetry values are that you could be monitoring that could provide you insight into potential attack vectors. So there's a lot of different telemetry points out there that could help with that. So that wraps up the command replay. So essentially what we did was capture the command link up and then we replayed it and we saw that it processes. So we'll move on to command link intrusion, which is a little bit different. It's a slight different take on a command link attack. So since there's no encryption in our environment or simulated environment here and there's no authentication, theoretically the thought was, well, if you just inspect the traffic and understand the packet structure and understand what information is in there, theoretically you could craft your own packet that matches that structure and just send it up and see if it processes the command. So you're not really replaying anything. You're just constructing your own raw command packet and then uplinking it and seeing what happens. So let's try that out and see how it works. So once you've inspected the traffic and understand there's no encryption, you see in one of the telemetry values and you could obviously do this with RF as well. You could record the RF on the downlink and record it as well. But you see once you have the information and you see here in the middle, you see CFE, TBL, NOOP, CFE versions <IP_ADDRESS>. So you see that information there and that implies that they're using core flight executive or core flight software. And so then you can say, well, that's available on GitHub, I believe. So this is where that comment about default configs really comes into play.  So then you get into this research study on command packet structures, which is a fun study would not recommend it for people if they're not really interested in it because it's a little bit of a read. But so we're looking, we start to break down how these systems work from a protocol perspective. So we have space packets and then we're going to get into some really space buzzword bingo here. So I apologize for that. But for the people who know what this is, you'll get it. But for the people who don't know, forgive me for kind of speaking in this lingo, but the space packets get wrapped in these transfer frames. And then the transfer frames are wrapped in what's called these command link transmission unit CLTUs. So that's kind of how this structure works is CLTU transfer frame headers, space packet and then the trailers. So then we start looking in the CCSDS protocol and we understand, okay, this is the space packet. These are the octets that are there. And then we start really diving into some of these documents and we say, okay, well, this is what the primary header is. This is what a transfer frame looks like. This is what the transfer frame primary header looks like. This is what the CLTU looks like. And then so we're kind of in this overload of terminology, protocol breakdown, things like that. So it gets a little cumbersome and confusing, but it's important to understand all these things. And it's also important to realize that these protocols are open source protocols. So a lot of missions, spacecraft that are out there today leverage CCSDS, because that is an international standard that the CCSDS group is a consortium of many space agencies from across the world who get together and standardize on protocols and things like that. So you can just Google the standards and find them and they'll have these diagrams in there and you can break down the bit patterns and things. So from our transmission, since there's no encryption on that, you just basically have the raw number structure. And to the kind of uninformed, this looks just like a bunch of gibberish, but what we're gonna do is take the knowledge we gain from reading the hundreds and hundreds of pages of those CCSDS protocols and see what we can find out. So let's look here, we see the fives in our scenario is the acquisition sequence. And then we've determined where the CLTU is. So now we've got the CLTU, so let's break that down and see what we can get. So then we take off the headers and trailers in this example and then now we have this subset and then we have this code blocks and these check bytes and we take those out. So now we've identified the actual TC transfer frame which is where the command information is stored on the uplink. So now we've got that. So let's take out the header on that and now we have the command packet. So this is the one that we were looking for. So with all those numbers, this is really what we were looking for. We're looking for this command packet because it's gonna tell us hopefully what kind of command will send up on that uplink. So from the telemetry stream we've kind of discussed we see this NOUP command that came down. So we look on the GitHub repo and we've realized that in our example that the spacecraft that we're looking at seems to be using these default command and telemetry IDs. Because if you look at the header file for the message IDs for the CFE table command message ID, you see 1804. So that would indicate like the high likelihood of actually the software is using these default message ID types. So what that does is allow you to basically transmit information to the, you're gonna try to transmit information using this knowledge of gain to the simulated spacecraft to see kind of what will happen. So let's see if we can reconstruct the packet with this knowledge and see what we can do with it. So with this knowledge, let's play around a little bit and see what we can do. So we've got our knowledge of the command structure from GitHub and if we can't construct a new command maybe something else to see what happens just to see if the command, the spacecraft will actually process it. So you have to do a little digging and understand the functionality of this software but luckily all that information is open source. So once you dig in a little bit you see, you notice there's this functionality within the core flight software called ES, it's ES application where you can potentially have control over the applications on board. So we're gonna see if we can't reconstruct the packet to do a reboot of a particular application. So in the real world, you would have to reverse engineer the packet structure of the RF signal. But in our scenario, we're gonna transmit the UDP packet with following information and see if they can get a reset of one of the applications. So we use that 1806 because that's the default message ID or for the ES application. So that tells the flight software, hey, this command is meant for the ES application to process once it gets on the satellite. And then the rest of the numbers is just understanding the translating the command and telemetry database into this format, which I won't bore you with all the details that had to happen for that to occur, but you can get there essentially. It just takes a while to figure out what structure is and what numbers need to go where and what those values need to be. So let's send this raw packet up, see what we can do. So we send that raw packet up to the satellite and look at that. So we see in the UART of the actual flight software that we have some sort of response. So you see at the top of the window, you see CFE ES restart app initiated for HK. So it looks like it was successful. So we essentially have reconstructed our own packet based on information that we've gained from GitHub and then transmitted it to our spacecraft simulator and it indeed did process it. So that's an example of a command link intrusion where you've basically built your own command structure and send it up and got it to process. So that's kind of the response. So the question is like, okay, why does the replay attack work? Why does this command link intrusion attack work? So part of it's the open sourceness of the information's out there so you can pull that data from GitHub and do your own research, but it really gets down to the protocols and some of the protection. So in this example, there's a communications operational procedure that's typically runs on a satellite, at least ones that operate CCSDS called COP-1 and that helps with sequencing to make sure that the sequence of the commands and the packets received are proper. On our example, COP-1 wasn't turned on by design in our example, but there's a lot of actual spacecraft who don't leverage COP-1 for various reasons. So COP-1 would have provided some level of protection, but that can be fudged and gotten around if you know what you're doing. But so in example, we did the same example for replay with COP-1 enabled and you see here that the commands not processed because they were out of sequence. So I could have really fudged the data around with sequencing and probably got the process, but we didn't really want to get into that. But the point was just to articulate that there are sequencing protections that you can put in to help with the replay. But the real control that you want is encryption and authentication. So you really need to encrypt and authenticate the command link. So it's not enough to just do encryption because you theoretically could just replay an encrypted link and it would process accordingly. So you need the authentication on the command link as well. So that's important. So you need both. So to properly protect from a replay or an intrusion attack, but here's some just guidelines for pseudo requirements, simple requirements about cryptography and authentication. So you need to authenticate and you need to do cryptography to ensure that these attacks won't work. Okay, so now we're gonna jump into denial of service via GPS jamming. So the first two were particularly to the command link. So those were, you know, intrusion and replay. This one's a little bit different and it's gonna use a different simulator, but we won't get into the specifics on the SIM, but it's a high fidelity simulator that has a lot of dynamics and a lot of commanding data handling and GNC capabilities. So what we're gonna do is get the vehicle in an operational state, where it's doing emission operations. And then we're going to denial service, the simulate the denial service with GPS signal, which is basically cut off the GPS. And then we're gonna see how the system responds. So jamming for those who aren't necessarily aware is, you know, the intentional or sometimes unintentional interference in the signal that prevents it from being received. So it's relatively simple to do. It's easy to overpower a close range and jammers can be on the ground and it can be in the ocean, airspace. Jammers can be about anywhere. So in our scenario, the spacecraft simulator is set up to be in what's called mission science mode or operational mode. And once it reaches this mode, we're gonna simulate the GPS signal by stopping the flow on the space wire bus, which is where the GPS sends its data to the Singapore computer, the flight software. So here is just a graphical representation of some of the telemetry and dynamics data for this particular SIM as it's operating. And we are, you know, it shows, you see the attitude control mode previous to attitude control mode current, which is MSM, which is the mission science, which is the operations mode. So what we're gonna do here is we're gonna block that data from getting to the Singapore computer, the flight software from the GPS sensors and we're gonna see what happens. So in the middle here, you see the GPS data packet counts increasing. So you see it goes from 920 to 944. So everything's optimal here. We're operating, we're flying in orbit, GPS signal is being processed and everything is fine. So here, now we're going to, in our simulation, we're gonna basically leverage a capability that was built in the simulator to block data across the space wire bus based on whichever values we want. So we can block about anything that's flowing across the space wire bus or 50-53 bus for that matter and see what happens. So once you initiate this block sequence, you see the ground telemetry value is free. So you see, which indicates the gray background there. So at this, the ground station is reporting, hey, I'm not receiving GPS data anymore for some reason, it's not being downlinked and that's because it's not being actually provided to the single board computer for downlink. So you see that freeze. So for spacecraft, one thing to understand is, you can't go up there, you can't like a, obviously a real computer go hit the reset button. There's a lot of autonomy built in. There's a lot of fault management and a lot of testing goes around fault management response. So what you should know is like typically when you have a satellite in orbit and some anomalous thing happens, the last of Jeffert to save it basically is to go into what's called a some pointer safe mode. So it basically takes the solar rays and points it to the sun and says, I'm just gonna hang out here until I hear from the people who are controlling me at the ground station. So what we're gonna do here is see what the autonomous flight response is once the GPS signal is blocked. So safe mode is an operational mode where a lot of non-essential systems are shut down. It's very similar to Windows. When Windows goes into safe mode, it shuts off a lot of the features of Windows from that perspective. So preservation of the spacecraft is the highest priority. So here you see in the downlink telemetry, the flight software starting to do its autonomous fault management, it's running what are called RTSs, so relative time sequences. These are just automated flight responses that are there. And it's starting to do various things. It's doing movements, it's stopping antennas, it's transitioning to some point mode from an attitude control perspective. So it's starting to save itself. And in the telemetry you see from the attitude control mode perspective, it's reached its desired some point state at this point. So why would you perform a denial service like GPS jamming on something? So maybe you're just wanting to have a little fun, maybe you're just trying to disruptive degrade some sort of mission operations for a specific time. Maybe you're wanting a mission not to achieve an objectives, but what if you're a little bit smarter than that and you understand autonomous fault response for satellites and you understand the design and fault management a little bit more. So a more strategic attacker understands that there's a lot of autonomous fault response for mission to put itself in a really safe state. So if you just think about the safe mode for Windows, a lot of the security features get disabled while you're in safe mode. And it does similar things do happen for spacecraft. So forcing a spacecraft into safe mode could potentially open up additional vectors that aren't there. So think about a maybe if you've jammed the GPS signal, maybe your fault management responses to safe yourself, turn off cryptography, and open up and you're commanding for any signal that you can process. So just depends on the fault management response. It's a, that's a design decision, but just think about that as you're doing your design that you may not think through it's like, hey, maybe adversaries may know that I can be more vulnerable, get put in a more vulnerable state. So maybe I need to design in something, you know, the terminology that we put in the, that paper that was linked in the front defending the spacecraft in the cyber domain paper. We talk about the cyber safe mode. So that's, you know, that's a slight transition from the traditional safe mode terminology, but it's basically it's a safe mode that's cyber hardened to a degree. And you have a high integrity mode that you're in that you know you're in a good state and everything's valid. And you still have authentication and encryption going on. So that's a concept, that's a kind of a newer concept that's being discussed in the space and the space sectors, but it should be there because knowing, you know, more and more people that understand space and understand fault management response may use that against you. So you may wanna think about that a little bit more as you do your design. So here's some recommendations for just generalized protection. This isn't specific to GPS jamming because there's a whole, there's papers upon papers out there that talk about how to protect against jamming, but this is more in line with protecting against, you know, leveraging fault management against you. So you really need to protect that documentation because that's just basically providing an adversary a pathway to get you in a more vulnerable state if your design is built that way. And it's other design considerations about always encrypting information, especially on the downlink. So some people think that the downlink on the telemetry side should not necessarily be encrypted because it's not controlling the vehicle, but there could be a lot of good information that is ascertained from telemetry. So it could be, you could have adversaries monitoring your telemetry stream looking for various, you know, indicators that you are in a less secure state. And these are other some design considerations that are here. I won't read them all to you, but basically getting the cyber safe mode, getting only report error messages when needed and encrypting it accordingly and getting, make sure you have the ability to recover and reconstitute to a known state. So these are just some considerations to think about as you're building your design for a spacecraft. So with that being said, those are just an example of doing some protection. So let's talk about what it takes to, let's transition the discussion a little bit and talk about what it takes to just protect the overall ground system. So we've, at this point, we've talked about, you know, the commanding intrusion, we've talked about command replays and denial service, but let's talk more generically about space systems in general. So I like to use the fifth and depth as a presentation material and discuss and a principle for design as many people do. I mean, I think anyone in the security community really understands the fifth and depth, but with the fifth and depth, I like to illustrate basically using an onion. So in the typical, you know, IT sense, you know, the onion diagram, you have, you know, the hardened server configuration wrapped with intrusion prevention, firewall type stuff, but in the space domain, it's a little bit different, but it's somewhat similar. But so here's some graphics that wanted to provide that give you a representation of the fifth and depth for both ground and space. So this first graphic here is kind of the ground system. So it starts with the data that runs, you know, on the endpoints wrapped by software supported by endpoints connected by networks, protected by intrusion detection and perimeter controls and physical control. So there's a little, you know, this isn't all inclusive, not every subcategory, you know, within each block is listed due to space, but it provides a good list of things that you would expect to be at each layer of the onion. So from a data perspective, you may have, you know, data at rest encryption, you may have tempest that you need to worry about. From a ground software perspective, you know, SDLC, secure coding, common weakness, numeration, prevention, things like that, binary analysis, dynamic analysis, static code analysis, stuff like that.  So we won't dive into every one of these, but it's kind of important to understand that you can pick and choose to apply controls at each layer and it's important to apply controls at each layer. So you don't want to just put all your eggs in the perimeter basket and say, I'm going to stand up a good DLP, a good firewall and security zones on the perimeter. And then hope that gets me there because you have insiders, you have other avenues where you need monitoring capability, you need protections. So that's pretty common. You know, that's nothing groundbreaking from a ground perspective, but similarly you can do the kind of the exact same approach for the vehicle itself. So this is a onion representation for the vehicle. So you have, you know, similarly the data that's on board the vehicle wrapped and processed by some sort of software on board that has a single board computer or, you know, a processor and a bus, hopefully protected by some sort of on-board intrusion detection prevention. So that's kind of a new area. What's being researched today is kind of onboard cyber monitoring and intrusion detection prevention. That's not typical on a lot of missions right now. What you see currently, a lot of the focus on protecting space vehicles is really geared around the crypto comms area. So, you know, the authentication encryption and the comms link, the transect, things like that. So, but there are controls that you can apply and there are threat vectors that exist for basically every one of these blocks on the vehicle as well as on the ground. So, since there's threats and vulnerabilities that can happen on kind of each block, it's important to put controls at each block honestly so that you can have a better chance of protecting yourself. So, in conclusion, we've talked, you kind of three attack vectors, but those are just three examples and there's many more that can be discussed. But it's kind of convenient, you know, to ignore security honestly because there's not been this, you know, windfall of events that have happened that are cyber related on the vehicle. But it's not really an option moving forward. Satellites are too critical to our everyday life or operations of our nation and the world honestly. And it's kind of important to understand that the barrier to entry into space has been dramatically reduced in the last, you know, five, 10, 15 years. I mean, with small sats and launch services and AWS ground stations, it's getting real easy to get a lot cheaper to get information. It's a lot cheaper to get assets in the space. So, and the fifth step needs to be a big part of the solution. So, we need to design security at the beginning, which is kind of cliche, but obvious. And it's going to be more and more important as we start to deploy all these small sats and a proliferated constellation moving forward.  So, I mean, there's a little bit of an interesting parallel you can draw, you know, with space systems and the industrial control system arena.  For years and years, people have kind of ignored security in the industrial control systems, OT, IoT arena. And it's not just been in the last few years, people have really started understanding that threat factor. And the main let's do to, there's been documented attacks and a lot of bad things have happened as a result of that. So, let's try to learn from that lesson and not wait till something really bad happens to make corrections. Because we know there are vulnerabilities. We know there are corrections. We know there are things we can do. So, we should be doing that. And the small set marketplace, you know, and the embedded security community is evolving. There's a lot of commercial things that are being put out. There's some open source solutions that are out there. And we need, you know, accountability. We need people who are researching this to understand the security ramifications of these things and need to be somewhat agile to get these things verified and validated and accepted and put out in the community as opposed to some niche areas, niche markets, people doing things. So, it's important for us to get, you know, leverage the explosion of this market. But it's also, we need the security researchers and, you know, ethical hackers and things like that to really, you know, hold people accountable for the things they're putting out. Because, you know, the path of least resistance is often used. So, if no one's looking at your work and understanding the security ramifications of your design and what you're doing, you know, people kind of get lazy with their implementation at the time. So, it's important that we all get out there and research this stuff and look into it to understand the ramifications, some of these design choices that are being made. So, we need to really, you know, the government industry, I think we're coming to a very good, you know, agreement that cybersecurity is super important for space programs. And there's a lot of investment that needs in education, training and talent development retention. So, I feel like there's a huge step forward this year at DEF CON with the hack-a-sat. I mean, that is a fantastic event that, you know, that's ongoing within the aerospace village to hack into representative satellite systems and, you know, there's these open source technologies that are put out that you can build your own ground of space simulation and you can start learning. You know, five years ago, there was not that much out there to learn. So, the activation energy and the level of access to information wasn't there for the security community to get into it. So, I think it behooves us, you know, within the government, within the industry to start getting this information out there and making this stuff available for the researchers so they can, you know, you can find some of these vulnerabilities and get them fixed. And technology has evolved from a simulation perspective so much in the last 10 years with the advent of, you know, digital twin type technology for space. The embedded arena is getting there with open source, things like Kimu, with commercial things like CIMEX or TCEN. You can get really high-fidelity simulators running software and that's, you know, built for Target and running a lot of tests. You don't have to buy these million-dollar space platforms. And also, these, you know, these CubeSats that you can buy and these embedded processors, you can buy some of these things, kind of conversely, have the chefs start doing your own research if you have some of the budget. But the last point is, you know, the reason I wanted to do this presentation and be a part of the aerospace village and work with the Space ISAC is information sharing is a must. I mean, coming from the government arena, you really get into this departmentalization of information and the need to know and things like that. So, but we really need to have information sharing. We need to do research. We need to have events like this, the aerospace village. We need to, you know, come together as a community and start doing this research together. And we're not trying to point fingers at people and, you know, say that all your implementation is bad. It's just we need to learn together and make things better and share what we learn and do presentations like this. So with that being said, that's about all I have to present. But I do have here on slide 29 a link to some resources, the Space ISAC there for information on it, the Hackasat link. Something that's going to be really cool is they, the Hackasat was, you know, they had qualification rounds and then they had the finals at DEF CON this year and they're going to open source a lot of the CTF's qualifications and things. So keep an eye on the Hackasat URL to, you know, maybe pull down some of those CTF's once they get published and give them a try yourself and learn. This is a really exciting exercises that were put together. If you want to build some of your own simulators and these links are there with Rope and Sack kit, Noskewed 42, things like that, open source flights offer CPCFS. And there's some info here on jamming if you want to learn a bit more about that. And the paper that was published in November 2019 is also linked here for your reference. So with that being said, that is all I have to present. And thank you so much. And I'll take questions in the offline session. Appreciate it.


Good morning, welcome to our presentation this morning on open stack divination to delivery We're going to talk a little bit about how to start an operations team and some of the challenges and difficulties that may come from trying to build the operations team We're going to talk a little bit about How Operations teams kind of get started such as You find yourself with a running production environment and then now you need to build the team And then we'll talk about some of the things that you need to do to keep in mind as you are building the team Then we'll cover some challenges and opportunities to take advantage of Some insight that other people have run into as they're building their teams and some of the experiences that we've had So that if you're trying to build your own operation team that maybe you can take advantage of some of that knowledge and Then we'll summarize and and get a questions and answers So I introduce myself. I'm Heather Maniscalco. I've been a Walmart now for since November My focus has always been on for like the last 15 years My focus has been on technology modernization enterprise processes and application implementation So I don't really come from the low-level engineering world. I'm more at the service management level So for example, I've been a people soft architects Oracle solutions engineer. So I've been a DBA and an engineer just more at the enterprise application level so that's where I've kind of the space that I've played in and at Walmart I've kind of taken on the role of working with our service management infrastructure Technologies and that layer of stuff that lays between, you know, the the phone call coming in from your help desk to Actually, then going to your orchestration layer or your top tier in your support operations office So that's kind of my focus right now So as you might imagine and I might be the only one I really appreciated the Gartner keynote yesterday The bimodal IT I'm sure I heard a lot of crone sitting around around me But I really understood what she was saying and I think it's very important and we'll kind of get into a little bit of that later on just how important understanding that both modes are very important to your to your stack and your operations and sometimes you need to Really codify and protect that mode one layer and I'll get into a little bit of that when I talk about ITS and solutioning later So I did want to take just a moment to discuss who we are We are from Walmart and I know that probably everyone in this room knows what Walmart is and you've probably been inside of Walmart at least once but what you may not know is just how large our Technology initiative is right now. It's in the billions of dollars of trying to actually Elevate and move forward swiftly into a really modern digitized world So we have 2.2 million employees globally. So we consider actually the largest employer on the planet We actually are considered that we are the largest retailer with the largest revenue We have 11,500 stores in 28 countries And we had over 480 billion dollars worth of sales last year Why am I saying that it's not to brag about Walmart? It's to just tell you just how large of an initiative this is and what we're going to talk about We're not doing perfectly because we we are pretty massively scaled So I'm going to turn over Scott, but I just kind of want to give you an idea of just how big this operation is and How complex Well that we're dealing in at this layer Yeah, and my name is Scott at Kins I came up through the CIS admin career as most of you probably have as well I ended up at Comcast and spent several years with the Comcast team building up the OpenStack team and being part of the Integral building of the operations team at Comcast. I now work at Walmart Haven't been at Walmart for very long. None of us really have and so our story At Walmart so far is that we're we're helping with the the operations and and Taking basically OpenStack to another level from where Where we've talked about in the past at summits with just building e-commerce so everything starts in the beginning with a POC generally most companies start that way We come from a background of bare metal VMware became pretty integral as part of the virtualization efforts of Hardware and then the cloud came along VMware became very involved in the cloud OpenStack came along matured and became really a force In cloud technology So a lot of companies embrace OpenStack, but there's a lot more companies out there that have yet to Take a look at OpenStack and give it a try. So what you end up doing is setting up a POC and Proving that the OpenStack does work for your company and that there are real use cases that the company can take advantage of It's usually set up on on leftover hardware or lab hardware stuff. That's not actually being used and And then you find out that it actually works at that point You you talk to management you talk to Sales you do whatever is necessary to kind of take it to the next level and you get get it approved Somewhere along the process you Involve other users at the company other teams who are interested in cloud technology They bring their apps onto the platform just to see if there's any issues with it if they can develop for the cloud platform if this is something that's going to be really good for the company and Guess what now you have production Operations is usually kind of an afterthought at that point because a whole effort is nothing more than just trying to get it up off the ground and putting some steam into it to try to Kind of get it to the point where you do want to take it to the next level So what happens after it's approved? Well now you buy hardware You take the OpenStack stuff that you installed and package it up you finalize the configurations you look at Stuff like Ansible and chef puppet whatever your favorite method of deploying configurations You really start to kind of pull the whole thing together The teams are usually pretty small at that point. It's an engineering team They had maybe engineering DevOps combined operations is kind of done as part of that effort because well you build it you run it type philosophy, but At this point you need to really start looking at operations as something that needs to be dealt with seriously and Think about all the things that you need to do to run a successful environment without it falling flat and becoming A real problem for the company So what does an operations team do? Well the first and foremost thing is keep the lights on you got to keep the environment running In a healthy way and also be very performant Operations teams do a whole bunch of other stuff all the traditional stuff that a typical sys admin Would do in an environment you deploy new hardware It's often the operations that does that you perform upgrades you do maintenance is You provide users support not only for Running their apps in the cloud, but also just to get into the cloud especially for a company who's just getting started in OpenStack There's a lot of well There's a lack of knowledge on what it takes to run an application in a cloudy way At scale Operations inevitably has to communicate with these users and say that you know you got to run your apps in this way Don't use static addresses. Yeah All the kind of things that are typically part of like legacy applications Operations also Does a lot of monitoring and reporting capacity reporting is usually in there if you get off the ground and and Start getting some steam under running the environment Naturally Upper management is going to come back and say I'd like to see some dashboards and reports How's the capacity? Do we have room to grow? So that's that's a very important aspect of a operations team Not to mention that things always break That's almost always something as sys admins doing is fixing things same thing with an OpenStack engineer OpenStack tends to run better these days with the older versions There was a lot more issues In the end issues change over time So something gets fixed in one release and you get a new problem in another release Being an OpenStack world. That's just kind of the nature of the game So how do you build a team up? Well, one thing that you got to keep in mind is that You can build it just about any way that you feel like it needs to suit your company's needs It needs to support how your team structures are at that company one size model does not fit all Often teams start off as a single unified team of DevOps engineers operations all rolled into one You might split it out so operations is on one side and DevOps and engineering's on another And if the team grows to a certain size, you may actually have them as three separate teams When we were running an operations team Previously our team started off as one large team. It wasn't large at the time. It was One single team, but as the OpenStack environment grew our team had to grow to compensate for that the number of users that come onto the cloud and use it also continue to grow in size and so we found that it was difficult for a single team to manage the building and Mapping out new features that we're going to be in in the next release and also running the cloud and supporting all the users So we broke it into two pieces which was OpenStack and the engineering Roles can be intermingled obviously if you split the teams apart you're more naturally going to split the roles between the teams But that doesn't necessarily mean that you have to have the roles very stringently Defined as if it's this it only can be done by this team if it's this it can only be done by this other team If you split a team into two and have operations engineering expect a lot of commingling of responsibility Which is really healthy for an OpenStack team as a whole And expect that it's going to change over time the bigger the environment gets the needs are going to change And your team structures and the wet what what they do and how they do it will also change What about hiring This is one of the difficult things in the OpenStack community is actually hiring OpenStack engineers One thing that you need to keep in mind is that you don't need to hire only experts If you've got a couple good people on your team that already knows OpenStack You know hire some juniors hire some regular sys admins that are pretty competent at what they do and train them up on OpenStack Kind of build the grassroots from within I'd also say that Hiring someone at least a few people depending on the size of your shop that have an understanding of applications Not only that's what I did But you know having someone understand how applications actually work and live and how people use them is actually a Really useful thing to have on your operations team because you can have engineers and developers and even Linux system administrators Etc. But at the end of the day you also need someone who can bridge that gap and understand how the applications are actually supposed to play and fail and recover and actually You know so the security needs and all that stuff of the application So I think that's very useful to have all your offers to you In our experience we found that hiring OpenStack engineers directly for operations was very difficult One there aren't really a lot of OpenStack engineers that are looking for jobs So that's the first challenge the second challenge is a lot of the OpenStack engineers who are looking for jobs Are kind of more interesting in the engineering side and not necessarily the operations Quite frankly a lot of people find operations a little on the boring side and rather not go that route And if you do find them you may find out that OpenStack engineers are kind of expensive It's hard to acquire them. It's hard to keep them in this kind of environment It is a very competitive environment as we've seen here at the summit There's a lot of interest in hiring OpenStack engineers. It just doesn't come cheaply And the little bull at the very bottom I could have probably moved it up, but it's an important point Hiring and training are expensive. Hiring is definitely expensive So you need to invest your dollars wisely and choose how you want to spend that money It would make sense if you could hire a couple engineers to do it that way that already knows OpenStack But it's a lot cheaper to Hire people that don't know that much about OpenStack or maybe just used OpenStack from a user standpoint Who's built applications either in OpenStack or maybe AWS and then just train them up on what you need from the other side On the training aspect, there's a number of different ways that you could approach it You I've already spoken about how you train people from internal You can kind of mentor mentee type relationship with the more experienced engineers Training up the younger engineers But you also can leverage online training. I know Linux Academy offers OpenStack training which is pretty good and OpenStack now is offering the administrator Certification which is also really good You can go the formal training route as well Mirantis or some other third-party vendor or vendor to either come in or you ship your people to Them to actually be formally trained. That's probably the most expensive route to go but the most successful way to train people up is to use a combination of the both and of all of them and Again, it comes down to money. So use your dollars wisely figure out what works best for you and go that route But be open-minded to try all the different ways so You you find yourself having to do operations. What is it that you need to do right now in order to be successful? Well, very obviously everybody already knows this you need to do monitoring first and foremost and metrics first and foremost without that You're pretty much running blind and if you have an issue you you won't know that it's actually occurring until it's too late Or maybe even how much time you spend trying to figure it out You also need to immediately Create kind of a tiered level support 24-7 coverage Which encompasses an on-call person somebody rotating in someone's always on call and if there's an issue They're the ones that respond and try to figure out what it is We also need to define your documentation as clearly as possible that defines what the escalation process is When something occurs, this is exactly what you do. These are the people you call This is the ticketing system that you open tickets in everything necessary to Define what that process is I think these are probably the most important things that you need just to Get out of the gate with an operations team Looking a little bit further ahead What are the more long-term planning goals that operations team needs to do? Procedures and processes and procedures are probably the weakest part of any operations team. Some companies do it better than others Definitely look at what your company does in other areas for operations because inevitably there's non-cloud operations at a company, but All the different things that need to be considered is you know event management. How do you handle things when they occur? incident management Handling things when they break and how to fix them problem management dealing with things that break Repeatedly or some serious issue Basically root cause analysis and how to try how not to let it happen again change management is very important process as you probably know Most things that break are related to some kind of change specifically Changes that people weren't aware occurring Change management puts more rigorous controls over Over that process so that everybody is aware that those changes are occurring. What is occurring? What is being affected and how to roll back? Maintenance procedures are another important thing for an operations team and also I think probably one of the weaker areas of operations You need to document what your maintenance procedures are if you're automating it It's tends to be more self documentation But anything and everything that you typically do on the command line to as part of maintenance needs to be very well documented You need to have SOPS associated with it and all of that feeds into the change management as kind of it's Point of record for what is going to happen Capacity planning also another very weak area of operations and it's also pretty difficult to do You need to get a handle of not only what is currently in use, but what is expected to be used down the road As part of growing a cloud You kind of need to be in tune with the users if you've got big data users coming onto the cloud you need to Understand how much CPU they're going to use how much storage they're going to use and how that affects Your ability to not only intake them into your cloud today But what is it going to mean six months to a year from now?  Accommodate them down the road and know that they're not the only Tenant that's going to be coming on the cloud with big needs patching an upgrade strategy Upgrades kind of come naturally It's one of those things that you can't stay on a particular version of open stack forever and With each release of open stack it has gotten easier Now we can do upgrades in place with as little disruption as possible patching is another story and I think every company fights patching In different ways with different sets of policies some if it's not broken don't fix it others may patch more aggressively I think that I'm personally at the view that you need to patch more often than the knot I see a lot of Companies don't patch at all when it comes to open stack they wait for the next release but in this day and age with zero day vulnerabilities and and Important bug fixes that kind of stabilize some of the features that are broken in open stack You need to kind of build in the process to Patch and know that that's going to come and not wait for the zero data occur and go I gotta figure out how to do it now And and end up disrupting All the applications that are in the environment It's also very important to really document patching So notifications you want everyone in the war in your world to know about that you're actually doing the patching You need to have a sign-off strategy So other folks need to know that you're doing the patching so that they're aware and you need to have release notes for your patching I think that kind of documentation is incredibly helpful You also really need to understand your shop's critical times of need So for example at Walmart as you can imagine Black Friday is hyper critical to us and we touch no systems Surrounding that time at all so and have an understanding that calendar and that schedule and actually the prior organization that I they came from Before Walmart also had a very aggressive calendaring system to understand when patchings could occur when they couldn't and the final note on patching is Understand if you have or you want to have a roll-up strategy It depends on how much patching you're actually doing doing a lot of one-off patching Maybe really cumbersome So if you can work with your company's schedule on calendar and you can work on a method to actually roll up Cerebral patches into one release to your system that can also be very useful and and prevent a lot of headache and Automation can't underscore how important automation is From an operation standpoint It touches all kinds of things, but one of the things that I think doesn't get touched enough is automating things like repetitive tasks that like, you know Something's broken and you restart a service or you have to clean up some log files or You know, you need some important information or pull something out of logs From an operation standpoint what I see a lot of teams doing is that they're content with just fixing it by hand and being done with it and moving on Those are areas that can easily be automated and not just that but also pushed back through the The lifecycle of OpenStack and maybe find some permanent fixes or permanent ways to deal with it So that you don't even have the need to repeat or automate those repetitious tasks Self-healing is a One of those things that you could say is kind of one of the repeated repeatable tasks that get done Service breaks your alarm goes off you someone goes in they restart the service Maybe clean up a couple things and they're back running again Self-healing is kind of a technique where a monitoring triggers Some event to occur some script to run that might go restart the service on its own I've seen some products out there that do it monitor stack storm Things like that that you can integrate in to kind of help you along the process of Doing that but again, I will underscore that just because your self-healing doesn't mean that you don't want to push it back through You need to push those things back through the lifecycle of OpenStack because if if you're having to do that There's still a fundamental problem with why it's breaking to begin with so Monitoring is is kind of hard You get some basic monitoring out of the box Most companies already have some kind of monitoring solution in place for other things whether it's Nagas or Sinsu or something like that and generally you'll try to integrate your OpenStack environment into Whatever existing environment there is sometimes you'll stand up your own One of the challenges in OpenStack is actually monitoring everything that needs to be monitored and knowing what what it is That needs monitored and with each version of OpenStack. There's Change there's things that are different than the previous version your stuff that worked before doesn't work now New features that have been implemented that you haven't had time to create new checks on It's it's a real challenge to try to stay on top of it leverage the OpenStack community I provided a wiki link for some of the communities Thoughts on on what to monitor how to monitor and with what? Don't rely on simple basic checks like service monitoring Use functional checking if if the users are doing something and it's something breaks for them And you don't see it. It doesn't help. I mean, yeah, sure Their services are up, but your functionality isn't there So do your own functional checking you can leverage tempest you can leverage rally There are other tools out there. You can write your own home screen scripts Write a script that launches the VM make sure that comes up all the way Creates a volume mounts the volume and then tears the whole thing down Also log logging Logging is another kind of tough area OpenStack needs a little bit of help I've heard that on a number of occasions here OpenStack needs a little bit of help on logging Centralized logging is key whether using Splunk or using log stash or some other method aggregate your logs build tools around those logs to Correlate events and and understand if something's broken that you can detect it Take advantage of the fact that a lot of information is there and can be leveraged for your monitoring Coming back to your relationships with the help desk or your on-call People document the checks very very thoroughly if you've got a check that's in Nagios or in Sinsi Document what it is document what it's monitoring document why it's needed document what it means when it actually alarms Document how to verify that it's not a false alarm so that someone on-call person is engaged they go in and say oh Yeah, this is really a problem and then document what it is you need to do to fix it If all of those pieces are documented on each and every single check that's in your monitoring system It's going to enable on-call to be able to do their work better It's going to enable your new employees and new hires that you just brought on to spin up faster because they can read this It's kind of self documentation and You can get them into the on-call rotation faster From a help desk from a knock standpoint If you've got a knock that wants to be a 1.5 knock or a 2.0 knock That information is going to be useful to them as well because they can actually use the information to maybe do the fixes themselves If they had that kind of access Alarms can also be very noisy How many of you have received alarms in your mailbox and there were so many of them that you just kind of got Lost in the noise a lot of it may not be That important, you know, they self-clear Fine tune your monitoring system so that when an alarm is sent out that it's actually Paid attention to that it's not ignored that it's important that it needs to be dealt with and taken care of Nobody wants to be inundated by a thousand alarms a day Most of it is just noise talking about capacity a little bit this is a Found to be one of the hardest things to do from an operation standpoint There's a number of different things that you can consider capacity looking at it from OpenStack standpoint You've got flavors, you know, you got a flavor that says it's a 4 gig VM and maybe two virtual CPUs fine OpenStack Allows you to launch your VMs up to a certain point But once you reach a certain allocated maximum that won't let you spin up any more VMs So allocation in OpenStack is very important over commit values tend to scale how far you can do allocations the default Over-commit settings in OpenStack is a little on the aggressive side. I highly suggest that you tune them back I I know in older versions. I'm not sure in the newest versions. That was something like 150% for Ram and 16 16 times over commit for a CPU. I think both of those are unrealistic But Other things that you need to track as far as capacity is not just allocated, but the quotas When users are given capacity in a cloud they tend not to use all that capacity right away It may take them a while to actually use all that capacity. It's kind of a ticking time bomb with regards to capacity The allocation is kind of what's in use right now as far as what OpenStack is concerned quota is kind of what could be used in the future With regards to actual use, you know If you got a VM that's launched with four gig of RAM and two virtual CPUs and they're only using one gig of it actual use may look really small But in when it comes to capacity planning, it's actual use is really kind of a meaningless value. You're more concerned about Whether users can spin up VMs and if if you let things go on for a while If you even have the space to spin up all the VMs that users are allowed to Typically operations teams will close an environment at a certain threshold. So, you know, we'll say 80% full We're not going to take on any more new tenant requests. We're not going to accept quota increases We're just putting a stop to it, but the thing is in in OpenStack That's typically done from the allocation standpoint Users that still have a lot of unused quota is going to continue to spin up more machines And you may still find out that even if you closed an environment at 80% that you still run out of space So do you plan your capacity based on quotas? Do you plan it on allocation? Do you do a mix? It's a really tough thing for operations to kind of decide on and then on top of that Not everything has quotas So ephemeral disk is a good example you launch a VM on ephemeral disk. There's no quotas associated with it So it's very easy for disk space to run out unless you kind of build it into the way that you do flavors at the company Swift is another Object storage is another tough thing. There's no native quota support in OpenStack You have to kind of do the quotas on the swift side with Additional ACLs and kind of build the tooling around that to make that work So all these can make capacity planning more difficult One of the things I think is really kind of It's my personal preference it doesn't mean it's anybody else's but I think there's a lot of value of having a separate team do capacity planning user support kind of Take that load off operations. It's so it's it's a difficult thing for operations to do when they've got all kinds of other stuff but when it comes to Thinking about users coming onto the cloud You really want some dedicated team that can can really communicate to the users find out what the users are doing What if their needs are what is it that they've got coming down the pike? What are what are their expectations for the next six months year two years? How much stores CPU ram they're gonna need? talk to them about what kind of Applications that they're gonna run and how they can run them in a cloudy way so that they're successful in the cloud And then if they're the same team that actually manages capacity and not operations managing capacity It can be all rolled into one so they know how much capacity there is they can see the applications coming down the pike They can determine if you have enough capacity to meet everybody's needs. They could engage infrastructure and Management and purchasing more hardware and growing the environment before you run out of space Give this team also some teeth to audit the the stuff that's running in the cloud Give them the teeth to enforce policy give them the teeth to Come back and say you know you're eating way too many resources you asked for this much and you're using this much Why can't you scale that we're gonna take away some of your quota? Give them some ability to help manage capacity after the fact not just before the fact So I will say we only have a couple minutes left So what I'm about to talk to talk through I'm gonna rip through these slides as fast as I can But to that end I want to let you know that I will be if anyone is interested in what I'm gonna do in the next couple minutes Please find me afterwards or if anyone's really interested I'm perfectly happy to do a little bit of a round table with a group of people as well So I think what I want to talk about next is fairly important because as he was talking about future stake capacity planning Asset management people teams communicating the question is how are you gonna do that right? So it's not a matter of you know, that's nuts and bolts of what you have to do How are you gonna do it? So the first thing that I would say is you really have to take your help desks very seriously I think a lot of times folks in the organization in this role are kind of I don't want to say They're not thought of as being part of operations and integral But you have to realize these are the folks that are interfacing with your users and your customers and your clients on a daily basis They have a very hard job and they're very prescriptive So when they come to you and they say I need something thoroughly documented They do their bonuses their their merit pays their their promotions a lot of that stuff is very intimately tied to How much they adhere to their SLAs and how much client satisfaction they have? So just give them some a lot of credit and work with them You'd be amazed at the amount of knowledge that they actually have about your organization and how it operates And especially how it interfaces with its customers So the next thing that I want to talk about is process of workflow design So when you do go through and you're starting up an operations team or if you already have an operations team You really need to understand your processes and workflows and it's not just a matter of saying oh, okay I need a provision environment. That's a workflow done You need to really understand why you've been doing something if you've been in a traditional enterprise model Your workflows and processes have probably been done very differently if you haven't had automation before if you haven't had an ITSM solution So you really have to be able to break down those older processes and workflows into why were you doing it? What function does it serve? What's the fit gap? Between your modern way that you're now moving toward versus what you used to do So I kind of put in here what I when I go through this process I really look for I identify the process understand it thoroughly plan for it design development and implement and Plan for a phased approach. You're not going to get a big bang You're not going to be able to eliminate all the spreadsheets in one day So you're gonna have to really phase this in and give it a lot of thought You really also want to identify the key components for your For your processes every process will need to have a metric. It's got to be measured the success of the process You need knowledge. It's always got to be documented. You've got to know What to do when something goes wrong or just how who does what and why and then what the outcome is supposed to be You really want to understand That part of your process So one of the key things that I think and this is kind of the meat of what I want to talk about One of the key things that I think is is a big problem in a lot of shops is tool duplicity and the implementation of tools What you'll find especially if you've got an older shop and that you're moving toward a whole new world you're gonna find that you have five thousand tools that do the same thing and You're going to need to call that herd to something that's That's reasonable that doesn't mean that you have to make every team look like every other team But even especially within teams or divisions really take a look at your tool set understand what you're using them for Understand if this was just someone's pet project and there's really something much better out there Just really do a tool audit That's a very key thing to do when you're starting to set up a new operations or just in general to run your IT shop Again do a fit gap analysis. What does the tool do? What it doesn't do a key way of knowing what doesn't do is how many spreadsheets emails and post-it notes are sitting on someone's desk Playing for change and you want to modernize and digitize you want to move to tools that are really going to serve your needs And you don't want to reinvent the wheel So, you know if there's a especially if it's open source if there's a tool out there that you've been leveraging You don't need to go build it. You don't necessarily need to buy a new tool Really understand what you have and don't reinvent those wheels And by doing this you're going to reduce your cost and increase efficiency typically so again IT service management, so what is that IT service management? I consider to be both an infrastructure a framework And I also considered to be a set of tools generally, so It is typically mode one. That's where I would put it into kind of tying it back to our first keynote It's really a system of record and when you think about if you're going to adopt an ITSM solution What it should have you really need to think about system audits company audits data metrics You need to think about incident problem change CMDB knowledge onboarding all of these are part of ITSM and all of this an IT service management solution will give you what he's been talking about But you really need to go give a lot of consideration and understand that it is not a trivial thing to implement I think the ratio of time here on this section versus the other section is very indicative of how it is in most shops actually you get like It's it's a very complex thing to implement so make sure that you dedicate enough time and effort and resources to do this and understand that implementing even if it is a more classic mode You got to understand that this is engineering. It is architecture. There's a lot to these big applications and having them work Well, so don't trivialize your people who are working on that side of the house either And you really want to plan for also implementing these solutions takes a lot of training Stakeholder occlusion is key. It might be the most important thing I can think of Because if you don't have stakeholders and people on board, you'll never be able to implement these solutions So another big thing Near and dear to my heart is integration So the other thing is a lot of people don't understand that once you have an ITSM solution And then you might have your gear on the back end or your remedy or whatever the heck it is you have or a separate knowledge management system You need to integrate these things and you and I'm not talking about message bush integration I'm not talking about low-level send a message here to there get some data and done I'm talking about a real orchestration a real transformative layer that can take your data from one system remain In point agnostic and be able to transform that data add business logic and rich the data Make sure it gets to where it needs to go and then once it's there you can actually leverage that data to make good business decisions So I highly recommend that that's another piece of the puzzle is that you consider integration to be a hypercritical layer And don't assume that that's just something you can shove it afterwards and please please please do not adopt the mentality of doing Point-to-point solutions with everything. It does not work. It does not scale. So I know that from painful experience A knowledge management is another key success factor. It's a very critical to running an IT operation shop Build versus buy decisions. I know that's something especially here very near and dear a lot of people are moving to the open source mentality Which is great? Buy building is also a big thing especially in top-notch engineering shops Well as to build it but you really have to think about what your business does if your business is not providing an IT IT SM solution or an integration solution to the world if your business is not selling that kind of stuff I would highly recommend that you consider not building That and really looking at what the tool does and being able to work with a partner who can really help do this because they're spending They're dedicating their world to R&D on those particular solutions So certainly do that and also expect that you're gonna have to be a champion if you are the one that's supposed to put in The solution you're gonna have to evangelize you're gonna have to be a champion And you're gonna have to do it from the ground up and the top down and you're gonna send emails to everyone Talking about the solution that you're implementing. So that was fast. I will definitely be available for afterwards if anyone is interested next So key takeaways, we're just gonna jump right to the end team structure Build your team the way you need it hiring. You don't need to hire all experts You can train within use a mixed set of training Techniques if necessary Build a relationship with your knock leverage them let them help you But of course as an operations team your biggest thing you need to do is keep the lights on that's the most important job Challenges and opportunities monitoring is hard leverage the open stack community and and let Let them help guide you in setting up a proper monitoring solution capacity planning is also difficult You measure everything collect metrics on everything Again build the relationships with your knock in your user community. It makes things better in the end Tooling try to reduce the number of tools that you have reduced Duplicate tools but recognize that you may have to have some because of the way things work within the company And then finally automate automate automate that is a thing. I think everybody can benefit from So since we're out of time Since we're out of time We will be outside for a little bit to answer any questions if you have any We would be happy to help answer any questions. Thank you for coming


Sorry to disturb you. These officers wanted to talk the names Yabu. I'm with the Odonari police department Can I have a minute of your time? The police Huh see here Kuno say son, or is it Hitoshi? It's Totono sir Kuno Totono What kind of name is that hey quit it so Kuno son? Where were you at 10 p.m. Last night, huh? At home why I'm actually in the middle of making curry, so it smelled wonderful alone That's right. Hmm. I did notice the ruckus this morning Did something happen in the neighborhood a body was discovered at the park nearby Sagai Ken? You know him, right? Sagai he goes to my university. You don't seem too surprised by his fate I'd like to ask you some questions Will you come with me to the station why I barely even know him and I don't know anything about what happened to him Just come with me. Will you okay? Just a sec. I have to turn off the stove How disappointing One of the fan favorite writers whose shoujo and Jose work include renowned manga such as there goes Tomoi Basara Chicago Seven Seeds and many more Yumi Tamura has won multiple shogakugan manga awards throughout the years her library is Diverse and rich in character studies mystery action adventure romance the list goes on But for this video we will shift our focus to one of Tamura's recent creations Don't call it mystery in it We follow Totono a young man who stands out partly for his bushy hair partly for his finely honed abilities of Observation and deduction when Totono is accused of murder He puts his skills to work delving into the lives of the cops Investigating him and uses his insights to find the real murderer. So I got to be honest here I went in completely blind when it comes to this book. I had heard the hype. I believed it I know of Yumi Tamura's pedigree as a creator However, what I didn't know was how much I ended up loving these first four chapters I can't wait to get volume two so I can continue the story I really took a liking to the characters and was charmed by the Simplistic yet pretty artwork presented throughout because at times it can be highly detailed and other times Slightly comedic with some funny faces and expressions Which obviously accentuate the mood and the jokes that are being told or at least the funny Scenarios that are being presented I should say so yeah Don't call it mystery like I mentioned at the beginning of the video stars the character of Totono Kuno Just a college kid that loves curry and just somehow is thrown into these weird Circumstances where detective work and his keen observation skills are pretty integral to solving cases He's not a detective. He's not a police officer He's just a college kid and I find that to be one of the most charming aspects of the book that he's just a regular guy He happens to be super skilled in this particular area, but it's not that he chose that route Things just sort of fell on your lap and I guess that happens to a lot of us in life things just sort of happen when you least expect it and His inquisitive nature leads him to solve different scenarios in the case for this first chapter here It's actually going to help him clear his name after being accused of murder But before I dive deeper into the story aspect a lot of people warned me that this was going to be a heavy dense read and I was a little bit put off by it at first because I didn't know what I was getting myself into but as soon As I started reading everything just flows nicely together. Temuta has a way with dialogue and characters Yes, it is dense, but it flows naturally and you pick up a pace as you're reading and everything just sort of Speeds up and once you know it you're finishing up the first chapter. So in that aspect I really enjoyed don't call it mystery the first story like I mentioned in the description and just now involves the main character getting accused of murder because he is somewhat related to the victim he is asked to go to police headquarters and The detectives and officers there interview him and grill him non-stop wanting to get that confession because they are 100% Deadset on Incarcerating Totono for something that he may have not committed obviously we kind of know going in that he's going to free himself I mean we see that in the description of the book one of the key aspects for this character of Totono Kuno is his observation skills like I've mentioned and how he is often questioning things taken for granted in society You first see this in this opening chapter But that is an integral part of who he is and the stories that soon follow chapter 2 It's the longest story in this book and it deals with a bus hijacking and Totono is part of it by accident Then in chapter 3 we get the resolution for that and chapter 4 leads us into a new story arc And how it is involved with what came before and what's going to happen after which I really liked I thought going in this was just going to be a Drama of like solving cases and every chapter was going to be a different story Similar vibes to and I know this is an unfair comparison But let's think to detective Conan or case closed where yeah, you have these characters But the majority the bulk of the story is just solving cases I thought this was going to be like that However, I was pleasantly surprised to see that the stories or the narrative weaves through each chapter And there are hints of bigger things at play and different characters do keep showing up in different aspects of the story I really enjoyed that it made it feel more fleshed out and lively and not just characters on a desolate backdrop The art for don't call it mystery is fantastic I really enjoyed when it goes into like full detailed mode and you get a nice view of each character And they're defining traits and facial features and all that stuff But it's not afraid to cut loose when it's making some silly jokes and having some fun with that You get some chibi-esque expressions, which were a little bit weird at first Especially with the tongue flicking on some characters. I thought that was a little bizarre But it brings its charm and it makes it stand out. So yeah, I really enjoyed that book artistically I love the characters their narratives I love that even if you're not gonna see this character again There's so much dialogue happening because again, this is not an action-heavy series You really learn about them and they make lasting impressions from the bad guys to good guys to neutral individuals They all have a story to tell and I think Temura has a natural way of Making that dialogue happen and be as smooth as possible I enjoy a book that can go the slice of life route while Interjecting mystery drama crime all that fun stuff. I really do enjoy it Okay, so I didn't really want to spoil things I just gave you sort of my first impressions and my thoughts and mini review on this part one of Don't call it mystery a really lovely book I can't wait to read volume two and volume three that is also scheduled for the end of this year Super excited about all of that. Thank you everybody for tuning in. I really do appreciate it This video was a little bit different than the norm tried to mix up a couple things from voiceover work to actual reviews to first impressions All that fun stuff, so it can be a little bit nerve-wracking. This took me a little bit to construct and put together Hopefully you guys enjoyed it. If you did, let me know in the comment section. Give it a thumbs up I'd appreciate that subscribe if you haven't to the channel. That'd be even greater Thank you everybody for tuning in. God bless. Stay safe out there. I will catch all over you on our next video