Hello, everyone. My name is Kobayashi Daisuke. I belong to Houditsu as a software engineer. I work as a developer in the OSS community. I've been involved in Kubernetes development for three years, and I acquired CKA. In this session, I will talk about introducing the latest Kubernetes features for troubleshooting. I will share three of the latest troubleshooting features in Kubernetes that are currently being discussed and developed in the community. The first is structured logging. A quick difference is the standard structure for logs and standardized references to Kubernetes objects to make the most common logs easier to query. The second is future code contextual logging. The key idea is that the library is passed a logger instance by the caller and uses it for logging instead of accessing the global logger. The cell is a function called distributed tracing. This is a feature to enable distributed tracing across Kubernetes. Most logging features can improve existing log formats and facilitate log analysis, aggregation and development development and troubleshooting. Distributed tracing is a feature that makes it easier to debug requests that cross-service boundaries by introducing open telemetry into Kubernetes core components. As more components introduce distributed tracing, Kubernetes will become easier to monitor, match and troubleshoot. In this presentation, I will explain the specific log format and usage, including an actual example of drug tracing. This will give Kubernetes system administrators and developers some insight into how these newest features and great and hard-rogan traces in the future. The theme is structured logging. I will talk about an overview of structured logging. This is a future proposed in CAP 16.02 by Malek. This CAP proposes to define standard structure for Kubernetes log messages. Add a method to K-log to enforce this structure as ability to configure Kubernetes component to produce log-in JSON format and initiate migration to structured logging. Conventional Kubernetes logs do not have the concept of key value and the values corresponding to keys are not unified, so there are issues during log analysis, processing, storage, query, etc. It is difficult to search if the naming format differs depending on the log, so by making the key a uniform naming format, it allows users to search with a uniform filtering pattern when analyzing logs. This image shows what the logs will look like when they are migrated. These are the advantages. The advantages of this format include this point. For users, by having log messages in structured formats, it is possible to standardize how to refer to Kubernetes objects, for example parts, nodes, etc., during log analysis. These make it easier to search many Kubernetes logs. By using the JSON format, we can lessen the following benefit. It works with many logging backends, so, for example, L6 search, start driver, a weak query, and sprunk. And it is easy to convert and pass, so analysis to write JQ commands are also available. And for DELPA, new K-log methods are allowed to output all structured log-in. Some methods introduced in K-log, and the methods introduced by CAP include these methods. The K-log library has methods called info-s and lrs. And this is a method used where previously info-f and lr-f were used. With these methods, the output of the log methods will be the message and some key value pairs. The example below shows the usage of the methods and the resulting log output. I will talk about the log-message structure. This CAP would write for the Kubernetes community to set around one preferred log-message structure that will be enforced by new K-log methods. Proposed structure should separate log-message from its argument and to read log and argument as key value pairs and be easily to pass and query and have a specific guidance on log-message and its argument. For this purpose, CAP suggests to use the following log-message structure. Structure is a message, key1 equals value1, and key2 equals value2, and so on. And so the message is formative using %q format logic, and keys are formative using %s format logic, and values are formative using %q format logic, except for number and vm. And keys are separated from values with an equal sign. And messages are key values are separated with a single space. The values to be put in the key value pairs part can be literally determined by the developer or code. So there is a possibility of variation in naming. For example, when representing a bot, use only the port name, or the name space is included, or use uuid. To achieve a uniform representation of a Kubernetes object, structure logging created a helper function to format it in a specific way. These functions are k object and krf. This shows one free file format for responding reference in logs, and build this format. For example, if we create the following pot objects, specify the pot object itself with the krf function. Although need to be, if we specify the name space and name of the pot object with the krf function, logs output in a uniform format of pot, equal, name, space, slash, name, as shown below. Pass some object left to info s and lrs function, and it will output in one prefaced format. For node names placed object, object left with name space part empty is recommended. Now let's talk about the flags that will be introduced. Structure logging arrows used to set the output log format from text format and JSON format. For that to a flag logging format is introduced. This is implemented as an option in component base. Some advantages of using JSON are broadly adapted by the logging library with value-efficient implementation, the adopt and the allow. And out of the box supported by many logging background is easily possible and transformable. And are existing tools for ad hoc analysis, so JQ. And some JSON keys are reserved words. So tlc's time span are the next time, and v is our velocity, and ar is our string and mfc's message. At the end of this chapter, I will introduce the development plan and progress of the future. This future is already in upper-end base development. Calog has migrated to v2 and supports logging instruction format, along with that the logging format flag has been implemented, and it is also possible to output the JSON format. Let's talk about migration. 21 logs of free account to the majority all equivalents of all logout groups have been migrated to structural format. Among the components, key bread and key scheduler have been migrated. Our performance benchmark is also being done with this transition. The next contextual log-in. Contextual log-in is future Proposing Caps 1377 by Patrick Owe. The proposal is to extend the scope of the ongoing convention of logging calls to structural logging by removing the dependency on the global care log logger. Like the conversion to structured logging, this activity can convert code incrementally over as many Kubernetes releases as necessary without affecting the usage of Kubernetes in the meantime. Contextual log-in replaces the global log by passing a logr.logger instance into functions where the context.context or an expressive parameter. It enables the caller to attach key and value pairs that get including all log messages. It enables the caller to add names and describe the component or operation triggered by a log message. It enables the caller to reduce the amount of log messages emitted by call-v by challenging the diversity. This works without having to pass any additional information into the call-v because the additional information was having stored in the method logger instance. That is passed to it. So the party components that use Kubernetes packets like client-go and node logger forced to use KLO they can do an arbitrary implementation of logr to logger and configure it as if they did. As contextual log-in does, it is possible to add information about the function, the caller's and relationship to the log and making it possible to trace the flow processing. For example, give scheduler automatically adds the following information to the log in the plugin to having filter processing by the volume by new plugin who supports etc. As a guess, we can adjust the amount of logging by challenging the diversity of the logger. When give scheduler performs common processing for parts, I was put to detailed laws for implement port and rest for rest important ports. Overview of log-out. Log-out define common logging API and interface for the purpose of not locking go-run-gauge programs to a specific implementation. Log-out itself does not actually have a logging implementation, but a log-out defines what method should be implemented as a logging library. The actual implementation depends on the backend logging library that supports the log-out interface. Log-out consists of a logger type and a log-sync interface. These definitions are shown in the great boxes. A log-out type is structured with log-sync interface and log-out type have some methods of log-out type. It can be passed as an instance to a function argument. A log-sync interface is an interface for the main method of log-out and its actual implementation depends on the logging library. Here are the methods that log-out defines. Info is a standard log-out width for log-out. And error is error log output. And three is velocity specification of log. The higher the number, the less log-output. There is default. Return the new logger instance after setting the velocity. Note that the specified numbers are added instead of overwritten. With name function as the name of the logger instance, return the new logger instance after name selfing. We can pass this logger instance to a function. If we use with name function inside a function, the name will be appended and this name will also be appended to the output logs. With values, with values as a key value pair values to log instance, the value of this key value is also added to the output log. I will introduce one use case of contextual logging. Contextual logging has the ability to trace function calls and relationships. Give scheduler developers want to know which port associated with which operation they are doing those. When queue scheduler starts processing port, use logger.with-value function to create a new logger and pass it to subsequent processes and functions. Then logger is used here after port name is automatically assigned. Then execute logger.with-name function for this logger when calling the volume binding plugin for a specific operation. From now on, logs using this new logger will automatically be given the information of port name and filter-slash-volume binding. Second use case is data and bubbles. In unit tests, developers can provide their own logger and distinguish output for each test case without challenging the logarithic code. We can change the implementation of functions such as info for each logger. The value set with values and with name can be applied to the logger. So we don't have to pass the content to be displayed individually. It can basically work independently of KL log output. For example, we can have our own logger. We can completely replace our binary logging backend and see that our KL log base Kubernetes code picks it up and use it. And second guess, if we use client-go in our application, we can reduce the diversity of the log messages from client-go by creating a logger.v1 and passing it to your client-go code. Logger v3.info called in client-go is the same as logger v4.info in application, not visible with v equal 3. Since the free value is associated with the logger, we can collectivity hide the logs output by components that you are not interested in. I will explain how to use each features. If we want to assign a value to the entire log, so use with values and with name method, we can add messages to embedded in all logs that inherit that logger using with values and with name. This also makes it possible to follow function calls. As a case, if we want to specify a logger, so set logger when calling when it is library. How to set the logger is to attach the logger to the context and pass it. Some functions to support context were added in the catalog library. klog.newcontext makes new context that has the specified logger and klog.fromContext returns the logger that is included in the specified context. The features can be used just by importing klog and context library, which values and with name functions as an example are here, like these yellow boxes. First logger has only pot key value, so the output is added, the pot is called ns-potname. Second logger, the logger has name filter and volume binding, so the output shows the filter-trash volume binding and the pot is called ns-potname. Third logger, the logger has another name, filter and node resources. So the output shows the filter and the pot is called ns-potname. Next, I will explain how to use the logger. As I said earlier, the logger can be attached to a context and propagated between cards. In this example, we passed a logger with the name example and the value if we use the logger to output the log, the value so example and the value will be added in addition to the message. To follow an example, attach a logger to a context and propagates it. Here, we use a function klog.newContext to create a context with the logger attached. We pass that context to a function and within that function we deduise the logger by using klog.fromContext. After that, we pass new information to this logger in the function and add to the log. As a result, all rows in this function will have the prefix, example, slash, something and the values are full e-golper and the root is called ns-name. Using loggers in this way makes it easy to trace correlations. At the end of this chapter, I will introduce the development plan and progress of the future. This future is already fully developed up to alpha. For contextual logging, klog can log out implementation based on the log-out API. The key idea is that libraries are faster logger instance by their caller and I use that for logging instead of accessing a global logger. The library decides about the logging implementation, not the libraries. The initial version of this cap describe a plan for moving or code for logging into kubanet.io slash klogr repository. Transition to ZAP would have removed all legacy code from kubanetis. However, ZAP transition would have been complicated to force all consumer of kubanetis code to adjust their code. So, the scope of the KP was reduced from removal dependency on klog to removal dependency on global logger in klog. Next step is deprecated klog-specific flag in kubanet components. This future is proposed in cap 2845. Klog has a lot of flags which might not be meaningful nowadays. This cap proposes to deprecate in the future the removed subset of kubanet's command line flag from kubanet components with goal of making logging of kubanet's core components simpler easier to maintain and extend by community. Luck of investment and growing number of klog futures impacted project quality. Klog has multiple programs. For example, addison format. It doesn't support throughput to fulfill kubanet's scalability requirements. It's complexity and confusion caused by mounting backward compatibility for real CG log future flags. In this future, what kind of flag are left? All klog-specific future flag except blue and green module are removed from kubanet's components flags. The components with log-flag frequency flag is also kept. Let's see what we change. This table lists deprecated flags. These flags include futures such as log file lighting or load load editing. We are using those features. We have suggested some migration options. If we want to output logs to the file, we can use kubanet's log-runner solution as an alternative. There are solutions like load-stache etc. or load-dated. We introduced a development plan and progress of this future. This future is always being offered in a better environment. Among the kubanet's components flag, the target flags are already deprecated. Remaining the kubanet's components flag for klog are blue, green module and log-flash frequency. Flag deprecation should comply with standard frequency and required three lists before removal, so these flags will be removed eventually in the 1.26. The next theme is APS avatar tracing. The next feature to introduce is APS avatar tracing. This feature was proposed in CAP 647 by David Ashpole. This CAP proposes enhancing the APS server to allow tracing requests. For this, it proposes using OpenTelemetry library and exposed in the OpenTelemetry format. So first, what is distributed tracing just briefly? If a user makes a request to something, it often passes through a variety of services along its path. We'd like to know what path a particular request took and it would be great if we could get a graph or something to show it visually. So OpenTelemetry is a better neutral open source of the ability framework for instrumenting, generating and collecting and exporting telemetry data such as traces, metrics, rows. As an industry standard, it is natively supported by a number of vendors. So why instrument the APS server? The Kubernetes APS server is a great candidate for tracing for a few reasons. It follows the standard RPC model and serve a request by making requests to downstream components. Which makes it easy to instrument. Users are right in the center. If a request takes more than 10 seconds to complete, many clients will time out. It has complex service topology. A single request could require consulting a dozen webhook or involve multiple requests to EDCD. This feature was proposed as there are many benefits to distributed tracing in APS server for this reason. My use case is to allow control plane administrator to easily find the APS server's issues. In distributed system it can be hard to figure out where problems are. It also has to find out what route a request took. If we can trace it, we can see which component was included in the path of the request. In addition, traces are easy to visualize so we can understand the seconds of events how long they took and easily spot potential problems area without being familiar with the internal of the infrastructure. In APS server, normally corrects traces at low sampling rate. Sampling only some requests have less impact of system performance. Even for issues that don't happen every time, a low sampling rate is generally still enough to suppress a representative trace over time. Traces can be also corrected on demand when problems occur. If we know the request we want to correct so we can correct the trace of that request by setting trace context and sampling the request. When tracing a request that reads nodes, we can press the request by using the command like below. I will introduce the future to be added. Tracing of APS server requests this can be left APS servers, HTTP server and HTTP client with other HTTP to get spans for incoming and outgoing HTTP request. This generate span for all samples incoming request and propagates context with all client request. Second, this cat proposes the use of actually tracing framework to create and export spans to configure backends. Thus, as context propagation to APS server, context propagation is implemented across APS server to propagates spans. A process provides function related to tracing utility libraries that wrap all of them. Having control over how the client requests are used in this can enable maintenance to enforce policy and make road improvements to the quality of the term. How to use this future? Let's turn on future get and list of APS server. First, enable the APS server tracing feature get then start the configuration for tracing by pointing the tracing config file flag on the client's APS server and our config file. Of it contains APS server kinds and sampling rate per million. In addition, ADCD can tracing is enabled by setting the following three flags in ADCD and then open them to the corrector to correct exported spans. Service export spans in ODLP format and correct corrects them. It is the load loop backend to visualize and correct its information. We quickly chose the backend. This image is visualizing traces with data and we can easily see how long the process is taking. At the end of this chapter, I will introduce the development plan and progress of this feature. This feature is already fully developed up to alpha. Tracing of incoming and outgoing HTTP and the RPC request is implemented in QAPS server. This makes it possible to correct traces of APS server requests using an OD corrector. In the data version, they will implement tracing 100% of request does not break scalability test and publish example of how to use OD corrector with Kubernetes and party with all text-based traces. In the future, they plan to solicit feedback and make improvements and migrate text-based tracing and more. I will also introduce keyword tracing as a related cap. This cap is proposed in QAPS 2831. This cap proposes to enhance the keyword to our tracing gRPC and HTTP API request. Keylet traces can also be visualized with Yega like this feature. It is easy to see how long the process will take. I will introduce the development plan and progress of the feature. In QAPS 2831, they will route up to alpha. They implement tracing of incoming and outgoing CRPC and HTTP request in the keylet. They plan, publish example of how to use the open temporary corrector with Kubernetes and all time to feedback and implement a revisit format Finally, here is the URL for QAP, which is a function introduced in this time. If you are interested in, so please refer to the detailed information on the content. Okay, that's all. Thank you for listening my presentation.


All right, so I'll just get straight to the point here. This is really just an announcement video to say that I've actually decided to split this channel up into two separate channels. So if you're interested in sourdough and bread baking content, then you're in the right place because that's what I'm gonna continue to post on this channel. And the good news is that I do plan to get back to posting that type of content a lot more regularly. But if you're interested in my other cooking content including my New York style pizza series that's actually gonna be moving over to my new channel which just goes by my name, Charlie Anderson. So I'm still in the process of uploading all of my older videos over there and I'll still be doing so over the next few weeks. But after that point, you'll start to see some new videos pop up that you haven't seen yet. And if you're wondering why I'm actually doing this, it's really just because I found that on this channel, most people either seem to be interested in the bread baking content or the cooking content but not necessarily both. So I think it just makes the most sense for all of us to split them up. That way you won't have to see videos that you're not interested in popping up on your feed. And for me, I can be sure that my videos are actually reaching the proper audience that wants to see them. So again, if you're not interested in sourdough and bread baking content, you can safely unsubscribe from this channel, The Regular Chef, and you won't be missing out on anything. And if you wanna continue to see my other cooking content, you can subscribe to my new channel right here. And if you're no longer interested in either one, that's no big deal either. I just wanted to make this announcement video so that you aren't confused as to why a lot of my videos have disappeared and why I've stopped posting my cooking videos here. So anyways, thanks a lot for watching and I'll see you over there.


Okay, so this talk is called a deep dive into the world of DOS viruses. And if you happen to be at the 8C3, that is 27 years ago, you would have seen a very young and awkward, even more awkward than I am at the moment, version of myself speaking on basically the same subject. The stage, of course, was a lot smaller than this. This would have really intimidated me back then. But I was talking about a university project that we had run for about three years at that point. And our possibilities are very limited. Meanwhile, 27 years later, our speaker, in between fighting battleships over the public BGP network and trying to encode data in dubstep music, was able to actually do all of the stuff that we were trying to do with a lot of effort, basically in, I guess, four hours of CPU time or something like that. Please help me in welcoming Ben to our stage to talk about a bygone error. Thank you. Hi, I'm Ben Gorick-Ox, as the slide suggests. So I have an admission to make. So this is a thing to be aware of.  Anyway, so what is DOS to get straight into it? You can do it in a bullet points way. DOS is an upgrade from CPM, another very old legacy system. But the third thing to be aware of is that DOS covers a wide range of vendors. It might not just be like those old IBM PCs. Some of the DOSes had compatibility with each other, meaning that some of the DOSes had shared malware with each other. But to be honest, most people know DOS as these lovely old beige boxes. The same era gave us our loved model M keyboard, hated by some, loved by others for the sound. But, most people's knowledge of DOS came from a user interface that looked like this. Pretty basic. There we go. Okay. So this is WordStar. Some of you may not know that Game of Thrones was written on WordStar. George R.R. Martin is apparently not a big fan of modern word processing. He admitted he had some issue with disliking how spell checking worked. So just uses, and I also guess it's a good security point of view. You can't get hacked if it literally has no internet access. So, also, for a lot of people, this is also their first experience into programming for the some of the older crowd. This is also the invention of QBasic, which gave a very basic language to program creatively in DOS. For some people, this was the gateway drug into programming and perhaps the gateway drug into what they started as a career. For other people, though, the experience of DOS was not so great. For example, let's just say you were doing some work in an infinite loop. And at some point, stuff like this happens. Unfortunately, I don't have sound for this one. But you can just, in your head, imagine your PC speakers playing some small techno music on only one frequency at a time. This might get especially embarrassing if you're in an office environment. Just slowly beeping away. You can't exit this. It has to finish fully. And if you touch the keyboard, it reminds you not to touch the keyboard and continues playing its music. So, you know, this would be fun, but this wouldn't be fun, especially if you're in an office environment. But, you know, ultimately, it's not malicious. And that trend continues. This is another good example of a DOS virus. This is ambulance. For when you run it, an ambulance just drives past. And then your normal program just continues running. I think this is amazing. It's an interesting era of viruses. It was all, the history of it was collected very well by a website called VxHeavens, which sort of still lives. But, unfortunately, at one point was rated by the Ukrainian police for what is the fantastic wording they used. Basically, that someone told them that they were distributing malware. Unfortunately, not malware that operates in this century. But I guess that's good enough for a raid. But, luckily, for the archivists, there are archivists of archivists. And so, we have a saved capture of VxHeavens. This is actually an old snapshot. There are way more modern snapshots. But, thankfully, the MS-DOS virus era doesn't move very quickly. But the interesting thing here is, like, there's 66,000 items in this table, and it's 6.6 gigabytes of code. And these viruses are, like, super dense. There's not much to them. Like, they are just blobs of machine code. They're not, like, your electron out these days that ships an entire Chrome browser. And normally, an out-to-date Chrome browser.  The normal distribution also changes with it as well. For example, the normal life cycle of an MS-DOS virus is, you know, you download or, for some other reason, run an infected program. That presumably does nothing. To you, it looks like it does nothing. So, you know, it remains roughly undetected. That then, you go and run more files. The DOS virus infects more files. And at some point, you're probably going to give one of those computers to some other computer or some other person. Whether it was by giving someone or copying a floppy disk of some software, maybe some expensive software so they didn't have to pay for it. Or uploading it to a BBS where it could be downloaded by many people. So, the distribution mechanism is a far cry from the eternal blues of this era. Where, you know, we can have a piece of strain of malware spread across the world very brutally, very quickly. So, most DOS viruses are pretty simple. They start, they say, have my payload conditions been met? If not, then they'll go and display, if they are met, they'll go and display the payload. And the payloads are definitely more, I don't know, nice. You know, you have stuff like this, which is pretty. And it uses VGA colors and all sorts of really nice stuff. You get also some very demo scene vibes from this. Another good example is this like VGA, like super trippy thing, which is really impressive because this is really small. This is less than one kilobyte of code. It's, in fact, way less than one kilobyte, like 64K. Or you just get interesting screen effects as well. For example, it's quick, but you can just watch the entire computer just dissolve away, which also might be quite worrying if you weren't expecting that. Alternatively, if the payload conditions are not met, then, you know, you hook sys calls and you, or alternatively, if you want to be way more aggressive as a malware author, you scan for files on the system to infect proactively. And the way you infect DOS programs is pretty simple. Imagining you have, like, one giant tape of all the code you have for the target program. Most of them work like this in that they replace the first three bytes of the program with a x86 jump. They append their malware onto the end of the executable. And so the first thing that you do when you run the executable is it jumps to the end of the file effectively, runs the malware chunk, and then it optionally will return control back to the original program. But there's also a thing about hooking sys calls, right? So, you know, MS-DOS is an operating system. It does have sys calls. The programs can reach out to MS-DOS to do things like file access and stuff. So as you expect, you run a software interrupt to get there. Thankfully, though, MS-DOS does also allow you to extend MS-DOS by adding handlers itself, or even overwriting existing handlers, which is very convenient if you're trying to write drivers, but it's also incredibly convenient if you're trying to write malware. For some of the examples of the sys calls, most of them are relevant towards DOS virus making. Here is a decent example of the things that DOS will provide you. A lot of them are just very useful in general for producing functional executables that users want to use. This is what the average program looks like. This is almost the shortest hello world you can make, minus the actual hello world string. In fact, the hello world string might be the largest part of this binary. It's a pretty simple binary here. We're moving a pointer to the message we just set. We then set the AH register to 9, or hex 9. That's the sys call for printing a string. And then we run a software interrupt, 21H, which is short for 21 hex. And we continue on. AH, we then set AH again to 4C, which is return with an exit with a return code. And the program will return. So in the meantime, this is roughly the loop that just happened. You have your program code that calls an interrupt. And that gets passed over to the interrupt handler. In the process of doing this, the CPU has quickly looked at the first 100 bytes of memory in the interrupt vector table, IVT, as it's abbreviated. And then it's effectively a router. If anyone has written like a small piece of code to root HTTP requests or anything, it's basically like that, but in the 80s with sys calls. So it just basically is saying compare this, compare that, jump that. Then the thing gets passed to the call handler. It goes and does the sys call, the thing that was required. Normally it will leave some registers behind as state or results of actions it has performed. And it returns control back to the program. So theoretically speaking, if we wanted to go and look at what a program actually does, we need to set a break point here. Because this is the only place that we can be sure the location exists. Because this is way before the era of ASLR, address space randomization. And this is way before the era of kernel space randomization. In fact, MS-DOS has almost no memory protection whatsoever. Once you run a program, you are basically putting full control of the system to that program, which means you can happily also boot things like Linux directly from a com file, which is handy if you want to upgrade. So if we look at certain files, we can go and see what they do. So in this case, here is one example. This is a goat file. A goat file is like a sacrificial goat. It is a file that is purely designed to be infected. So what you do is you bring a virus into memory in the system and then you run a goat file in the vague hope that the virus will infect it. And then you have a nice clean sample of just that virus and not another program inside the virus, which makes it way easier to test and reverse engineer. So we can see things that are happening here, for example. We can see it opening a file, moving like where it is looking towards a file, reading some data from the file, just two bytes though, and it closes a file. We see the same sort of thing repeat itself except at one point it reads a large amount of data, moves the file pointer, writes another large amount of data, does some more stuff. And this, we parse some file names, we display a string which is almost definitely the goat file message. And yeah, we pretty much exit after that. So there were a few syscalls here that we would really like to know more about. So for that, it is the open files. We would really like to know what files were being opened. We would also want to know what data was being written to the file rather than having to fish it out of the virtual machine later. And we would also, just out of curiosity, really want to know what file names it was asking MSDOS to parse. The display string is also a nice test to know whether your code is working. So to do this, you're going to have to look a little bit deeper into how the MSDOS runtime and by proxy how the x86 in 16-bit mode works, or legacy mode, I guess. This is basically all the registers you have in 16-bit mode, and some nice computations at the bottom to make it easier to read. So as we mentioned, AH is the one that you use to specify which disk call you want, and you'll notice it's not there. AH is actually the upper half of AX. AH is an 8-bit register, because sometimes people really just want only 8 bits. It's a very obscure that we were saving that much space. And so this is what a, this is the definition of the sys call of a print string. So you have AH needs to be set to 9. This is once you, in order to call the sys call for printing string, you just set AH to 9, and then you need to set DS and DX to a pointer to a string that ends in a dollar. And that doesn't make a lot of sense, or it didn't make a lot of sense to me when I first read that. And so to do this, we need to learn a little bit more about how memory works on these old CPUs, or the CPUs that are probably in your laptops, but running in an older mode. So for this is effectively what it looks like. They have a 16-bit CPU, 2 to the 16 is 64 kilobytes, and we have a 20-bit memory address in space. 2 to the 20 is 1 megabyte. So if you ever see an MS-DOS machine limiting at 1 megabyte or some old operating system saying like the maximum memory it can have is 1 megabyte, it's because it's running in 16-bit mode. And the maximum it can physically see is 20 bits. So the question is, how do we address anything above 64K if the CPU can only fundamentally see 16 bits? So this is where segment registers come in. We have four segment registers. Actually, we might have more, but they're the ones you need to care about. There's the code segment, the data segment, the stack segment, and the extra segment for in case you need just another one. So anyway, with that in mind, let's have a quick crash course on segment registers. So imagine if you have a very long piece of memory, and we can only see 16 bits at the time. So, however, we can move the sliding window around in the memory to go and see, like, to move our view of where it is. So we can do this and put data around the system, and we can use the final pointer to specify how far in to the memory segment we should go. So the DS and DX really just means a multiplier. So where the data segment is 100, you need to just move 100 times 16 to get to the correct place in memory. And then DX is the offset. This continues on. So where we have 16-bit CPU, we have a bunch of general use registers or general purpose registers. They're quite useful for ensuring you don't need to touch RAM too often. X86 actually has a fairly small amount of general purpose registers. Some architectures have way more. I think more modern chips like GPUs have hundreds, well, hundreds, maybe thousands. However, this doesn't really change over time in X86 because we have to force backwards compatibility. So really what actually ends up happening when we move up the bitage is that the same registers just get wider. And we add some more ones for the programmers that want them. And the exact same thing happened to 64-bit, the registers got wider. So thinking about it, we have a lot of malware now. What if we want to know everything that's happened in this entire archive? So we kind of want to trace all of these automatically, but we might not know what we're looking for. So let's go through the checklist of what we need to do to trace all of this malware. We need a breakpoint on the syscall handler. When we get that breakpoint, we need to save all the registers so we know which syscall was run and potentially what data is being given to the syscall. Ideally, we're going to save 100 bytes from that data pointer, especially because we need it, but it's quite handy in a lot of registers, in a lot of syscalls. It's, for example, what you use to get the open file path when you're opening files. We should also probably record the screen for quick analysis rather than just staring at HTML tables. And so we can do that. We burn a lot of CPU time and probably call some minor amount of environmental damage. And we get nothing.  At best, they return a Goat file string. They just do nothing. So if we look deeper into the reason why, it's sort of a smoking gun here. So we can see the syscalls that run on this file that does nothing. And the smoking gun here is the date. So it's asking for the date from the system. And this sort of flags out the first issue is that a lot of MS-DOS viruses don't really have a lot to go on because they have no internet connection. And there's not really any other state they can decide to activate on. So the date syscall is pretty simple. The get, date, and gate time just return all of their values as registers, and some using the 8-bit halves to save space. So a naive way of doing this is what we do is we would run the sample. We'd wait for the syscall for date or time. We would just fit all the values because in this case, we're using a debugger. So we can automatically change what the state of the registers are. And we can then observe to see if any of the syscalls that the program ran changed, which is a pretty good indication that you've hit some behavior that is different. And then we can say, hooray, we found a new test case. The downside is running every one of these samples takes 15 seconds of CPU time because MS-DOS, or 15 seconds of wall time, which when you're emulating MS-DOS is 15 seconds of CPU time because of the fact that MS-DOS doesn't have power saving mode. So when it's not doing anything, it just goes into a busy loop, which makes it very hard to optimize. Or we could take a cleverer look. So when we think about it, we are in the interrupt handler. All we ever see is the insides of the interrupt handler because we don't know where the program code is. The interrupt handler is the only place that we know is consistent because MS-DOS could potentially load the code for the malware or the program anywhere. But we want to know where the code is. It would be really handy to know what the code is that would be about to run. So for this, we need to look towards the stack. Just like the DS and DX registers, the stacks are located on a stack segment and a stack pointer. Luckily, the first two values is the interrupt pointer in the stack segment. So we can use that to grab exactly what the code will be around afterwards. So we just need to add a few things to our checklist. We're using to grab four bytes from the stack pointer. And then using that, we can calculate the destination that the Cisco will return to. And if we look at some of them, we can look at an example here. This is what one of the calls return as. So we see we're running a compare on DL against the hex of 0x1e. And then if that comparison is equal, it will jump to one memory address. And if not, it will jump to another. So if we look back at our definition of those Cisco's, we can see that DL is the day. So with this, we can conclude that if 0x1e is 30, DL is the day, this malware effectively is saying if the day of month is 30, we need to go down a different path. If we run these all over time across the whole dataset, what we see is roughly this as a poorly drawn bar chart. We see out of the 17,500 samples we have, around 4,700 of them, check for the date and time. And these are the ones that are really tricky because they're really hard to activate. They're also the most interesting, though, because those are the ones trying to hide. So with that in mind, we have the code segment that we're about to run when we return. And we can't really brute force because it takes a lot of CPU time. But we can't brute force it inside a real or emulated machine. But we can brute force it in a significantly more interesting way. We need to build something. We need to build the world's worst X86 emulator. So dubbed Ben X86, it's 16-bit only. Any attempt to access memory effectively ends the simulation. It's got a fake stack. If you try and push something onto the stack, it says sure, fine. If you try and pop it, it's like, oh, actually, I never held any of that data anyway. Sorry we're ending the simulation. 80 op codes, most of them are jumps because that's the primary purposes comparing in jumps. The difference is it logs every op code, every address that it went through. And it can be run with just a small X86 code segment and a register snapshot. This means we can test all days from 1980 to 2005 in roughly about 100 milliseconds. And most programs ended up having just three different code paths on average. So that yields us with 17,000 virus samples and about 10,000 samples that had date variations as in once you explode the complexity. So I'm going to now use my final remaining time to go through some of my favorite. So this is an example of a virus that just doesn't do anything on the first of 1980. However, if you were to happen to be running this on New Year's Day, New Year's Day, you would get this. No matter what you do, every program, you can't exit out of this. Your machine is hung. This might be great, right? You might be like, oh, cool, I don't need to do work anymore because my computer will literally not let me. This also might be terrible because you might need to do some work on New Year's Day. Here's another example. This does nothing as well. Just another innocent.com file. Of course, reminding that these pieces of malware will be wrapped around something else. So, you know, almost anything could be infected in here. In this case, though, these binaries are nice and shaved down. However, instead, we get this, which I think is super interesting. And it's basically the author is aware. They're telling you. They're actually self-disclosing. They're saying, the previous year, I've infected your computer. And for some reason, it's they're being nice. They're just saying, yeah, actually, you have been infected. And as a, I guess, a pity, I'm just going to remove myself now. I don't really, for some reason, it's also encouraging you to buy McAfee. This was back in the day when John McAfee himself actually wrote McAfee. Interesting times. Definitely interesting times. Here is another example. This one I found particularly obscure. On the 8th of November in 1980, or any year, I think, actually, it turns all zeros on the system into tiny little glyphs that say hate. If anyone understands this, I'd really like to know. I've been thinking about this a lot. What does it mean? Is it an artistic statement? Is it, I wish I knew. There could be a CCC variant that says Marte. Another good one in that it's the last thing I ever want to see any program tell me is this one here where you run it and it says, error, eating drive C. I never ever want to error in any program that unexpectedly just says, sorry, I failed to remove your root file system. Don't know why. Could you change your settings so I can remove it? Cheers. And finally, this is one of my absolute favorites in that it's just brilliant in that it also stops you from running the program you want to run. It exits prematurely. This is the virus version of the Navy SEAL copypasta. It says, I am an assassin.  I also hate Aladdin. And I also will kill it. I will eliminate you and we know where this is going. It says it's fear the virus. It is more powerful than God. It only activates on one day though, so it's fine. Thank you for your time. I know it's late. And I will happily take any questions or corrections if you know this topic better than me. This totally brings tears to my eyes with nostalgia. So if there's any questions, we have microphones to threw it around the room. There's like one, two, three, four and one in the back. We also have questions perhaps from the internet. If you want to ask a question, come up to the microphone, ask a question just as a reminder. A question is one or two sentences with a question mark behind it and not a life story attached. So let's see what we have. I'm going to start with microphone number one just because I can see it easiest. Let's go for it. Hi Ben. Thanks for the talk. Really interesting. My question would be did you do any analysis on what ratio of the viruses was more artistic and which one actually did damage? So most of them surprisingly don't do damage. I actually really struggled to find a date varying sample that specifically activated on a certain day and decided to delete every file. There are some very good ones. Some of them are like virus scanning utilities that just don't do anything on certain dates.  So that's particularly cruel. But it's actually surprisingly hard to find a virus sample that actually was brutally malicious. There were some that were just in fact binaries, but it's very hard to find one that I think was brutally malicious. Which is a far cry from the days well from the days that we live in right now where we're taking down hospitals with Windows bugs. As everybody's leaving the room, please do it quietly. I see a question at three on that side. Yes, since a lot of industrial control systems still run DOS, what's the threat from DOS malware that might be written today? It's probably unlikely. The industrial control system that's running DOS would come into contact with DOS malware. The only way I can think is if one vendor was, or if a factory or supply or whatever was basically downloading or basically wears onto industrial control boxes, I wouldn't be surprised. But it would be pretty irresponsible. But it would be quite surprising to find MS DOS malware today on industrial controllers that was installed recently and not just a lingering infection from the last 20 years. Microphone 2? Did you find any conditions that weren't date based? Some of them do attempt to, some of them try and circumvent the date recognition. Unfortunately, it's very hard to brute force those. Some of them install themselves as what's called TSR, terminate and stay resistant. Which basically means that they will exit out, run in the background and continuously ask the actual system timer what time it is. It's a bit of a more risky strategy because the system timer might not exist, which would be unfortunate for the virus. So definitely there are viruses that have way more complicated execution conditions. I observed one sample that only activated after, I believe it was something silly, like 100 key presses, which is very hard to automatically test. Those sort of viruses require static analysis and statically analyzing 17,000 samples is a time consuming task. So we have a question from the internet. Do you have the source or what is the source of the malware that you analyzed here and did it publish somewhere? You can still find dumps of Vx Heavens and probably more modern dumps of Vx Heavens on popular torrent websites, but I'm sure there are also copies floating about on non-popular torrent websites. Over the microphone one. Hi Ben. I'm Job. Thank you for your talk. I was wondering did you learn anything from your studies of these viruses that should be taught in modern day computer science classes? Like more efficient sorting algorithm or some hidden gem that actually should be part of your approach to computing these days? My primary takeaway was x86 was a mistake. So I'm not seeing any more questions which oh no there is okay one more question from the internet. Have you found malware samples that did like try to detect dummy binaries or whatever to avoid easy analysis? Oh actually that's a really good question. So it's complicated. So some viruses would so maybe let's be dangerous. Let's try and go backwards on my home written presentation software. So come on. Too many slides. I have regrets. Yeah okay here we are. This slide. Okay so you know here how I'm saying that the malware infection goes to the end? Well some samples are really cool in the they don't change the size of the file. They just find areas of the files that are full of null bytes and just say this is probably fine. I'm just going to plot myself here which may have unintended consequences. It may mean that if a program is like a statically defined byte array of like a certain size and the program is relying on it being zeros when it accesses it for the first time it may get very surprised to find there's some malware code in there. But generally speaking as far as the underwear these this this deployment procedure works pretty well and it actually is very good at avoiding antivirus of the era which would just be checking like common system files and its size and you know if the size increases of command.com then that's clearly bad news. We have a question on microphone one. Are there any viruses that try to eliminate or manipulate virus scanners of the day? Oh yeah so a lot of the samples will actively go and look for files of other antiviruses but I am generally under the impression in that it's kind of hard to find them. There weren't actually that many antivirus products back in the day. I wasn't really I feel like it was a bit of a niche thing to be running. Microsoft did for a while ship their own antivirus with MS-DOS. So I guess you know what's new is old. So there were antiviruses out there. I don't think many of them were very effective. Any more questions there? Where? Oh right another one from the engine is interesting that the engine is querying MS-DOS all the time. Go ahead. Did you do the diagrams by hand or do you have a tool? So many hours. No so there's a couple of good tools to do it. ASCIFLOAD.org I think is a fantastic tool. I would highly recommend it. I think it's not maintained very well though. Microphone one. Are you publishing the tools you wrote? I will be publishing the tools at some point when they are less ugly. I will be definitely publishing all of the automatic malware runs and the gifs generated by them so that people can easily base Google for the virus names and get actual real time versions. The hardest thing that I found is when looking at virus names was literally just finding any information about them. And one of the things I really wish existed at the time of writing this talk was being able to just query a name and be like oh yeah this virus it looks like it does this. Since I saw a microphone one first let's go with that. Did you find any viruses that had signage in them not signage of today but the name of the author like he was very proud of what he wrote? Yeah there's some notable examples. Quite a few of them will try and name so DOS viruses do like have obviously sample names in the same way that you know we still today like give viruses names. A lot of the time you will just encode a string that you want the virus to be named it's you know somewhere in the file just a random string doing nothing it's like oh okay they clearly wanted to be called tempest. So that does happen one of the favorite examples is the brain malware which literally encodes an address and phone number of the author. I believe in Pakistan and there's a fantastic mini documentary by fsecure where they go and visit the people who wrote it. It's a super interesting watch and I would really recommend it. Indeed yes. Microphone 2. Did you have any chance to look at any kind of viruses that did not modify the files themselves? For example one of the largest virus infection time was a virus called Nymella which modified the master boot record. Yeah master boot record I did consider it was more of a time problem that I had in that getting to the point where you could brute-force time and date combinations and looking for master boot record changes was really hard. I am super interested in reviewing effectively the root kits of the era but yeah that's definitely something I will look into in the future. And we have yet another question from the internet. Yeah it's even from the same guy. Oh damn. Is the Ben x86 software open source or can it be found on the web somewhere? It probably will be. I wouldn't expect it to work in well in any use case though. It's effectively designed to like not work correctly right like you know what's the what was the spec it basically like fails at every single anything awkward I just went like oh that's fine we're probably far enough down there anyway. Where are we? Yeah like be aware this is the feature list. So is that a follow-up question from the internet? And no it's a it's a new one. Oh it's anyone good. And wouldn't I don't know how serious it is but wouldn't it be possible or a good idea to use machine learning to create new dust malware from the existing samples? It would not be a good idea but I like how you think. Actually I saw somebody trying to use NLP to generate viruses but okay that's a different you could probably do Markov chains with x86 to be honest. Please don't do that please. Don't try this at home. I have seen things I have seen just please don't do that. So I think we've run out of questions going once going twice let's thank Ben for this marvelous retrospective talk.


Hello, I'm Captain Ted O'Hara, Commander, Task Force 6-9. As part of Sixth Fleet, we are responsible for submarine warfare operations in the Yukon and Africom areas of responsibility. Our team is comprised of a lethal and agile force of attack submarines and guided missile submarines capable of destroying enemy ships, projecting power ashore with Tomahawk cruise missiles and special operations forces, and carrying out intelligence, surveillance, and reconnaissance missions. We silently operate 24-7, always ready to support anti-submarine warfare operations at a moment's notice in support of our national defense strategy. On behalf of the men and women of Commander Task Force 6-9, I want to wish Sixth Fleet a very happy 70th birthday.


So, for those of you who don't know me, my name's Armand Dodgar. You'll find me all around the internet as just Armand. And I'm one of the co-founders and CTO of HashiCorp. For those of you who don't know about HashiCorp, don't worry, it's very common. People know us much more for our tool names than they know the company. So, it's part of our regular education campaign that there is also a company behind it. Most folks are probably familiar with our vagrant tool, maybe Terraform. We make a handful of different tools, but really our overarching mission is looking at the challenges around delivering an application, whether you're an IT operator trying to figure out how to provision and manage infrastructure, whether you're a security engineer trying to figure out how do we deal with this brave new world of cloud and zero trust networks, or whether your application developer who just wants a runtime platform and want to find your upstream services and don't really care about the bits and bytes and how they all connect together. And so, because we try to look at so many different problems, we have quite a different handful of tools. So, I'm not going to spend a lot of time talking about most of them. Instead, I want to specifically spend time today on Vault and the security challenges we face as we go to the cloud. So briefly, I'm going to touch on a few different things. For folks who are less familiar with Vault, I'm going to give a quick intro there. What are some of the use cases? How does it work? And then, from there, talk about a new feature that I'm very excited about that landed in Vault 09 and then talk about Kubernetes integration since we're all here at KubeCon. So, for folks who've never heard about Vault, the way we like to describe it is it really solves three different challenges that are all pretty related. The first one, which is really why Vault exists, came out of the need to deliver secrets to our end applications. So, as we adopt tools like Chef and Puppet and Kubernetes and get more rich in our orchestration and deployment and automation, we have this interesting challenge, which is how do our applications actually get all of the credentials they need, whether it's database credentials, whether it's cloud API tokens, whether it's TLS SSL certificates, so on and so forth. We have all of this privileged material that our applications need. And as we're sort of driving towards this infrastructure as code vision, whether we're committing Chef code or Terraform code or our YAML configurations for Kubernetes into version control systems, we don't want to pollute all of that with sensitive information. Because otherwise, it's being stored in plain text. Most of the version control systems we use are not really designed for sensitive materials, so they don't really have things like access logs and fine-grained capabilities. And so, this is the first use case, how do we think about managing this privileged information? How do we deliver it to the applications that need it, but then still do it in an automated way so that these systems don't get in our way? Another sort of interesting class of challenge is, great, we're delivering our database credentials to our application and we're giving them a 32-byte encryption key and saying, God be with you, get the crypto right. This kind of works, but what we find in practice is that cryptography is really, really hard and key management is even harder. So, how do we make doing security right a lot easier? And so, how do we think about providing higher-level APIs, things like encryption as a service, so that we're not handing an application 32 random bytes and saying, have fun, but instead providing sort of higher-level APIs that are less error-prone? And then the last one is, as we're doing this, what about the people? We've spent a lot of time talking about machines and automations and applications, but there's still people in the loop. So, how do we as developers and operators and security people get access to the privileged systems that we need? And the last bit is that bit. It's the access management. So, with that, let's jump right into secret management. What does this world really look like? I think it starts first by understanding what a secret actually is. The way we like to think about it is, it's any piece of information that you can use for authentication or authorization. So, it's something that I can provide to another system that's granting me access to it or proving my identity or elevating my capability, right? So, here's a few examples of it. Sensitive information is sort of a different class. It's things we'd like to keep confidential but don't necessarily grant me additional access. So, maybe, you know, things that my customers order, I consider that sensitive, but it's not a secret piece of information. It's sensitive. And so, the open questions as we talk about secret management are really things like, okay, how do our applications get these secrets? How do our human operators get these secrets? How do we update these things? What happens if we need to revoke access? How do we know who's used what? And what do we do in the event of a compromise? These are sort of the hard questions in secret management that we'd like to have an answer to. The standard kind of state of the world, I guess, in some sense, the answer we typically see in many of these questions, is not great. Typically, it's what we would define as secret sprawl. These secrets are defined everywhere, whether hard-coded in an application, living in plain text in a config file, you know, strewn about GitHub or, you know, our puppet servers, so on and so forth. And so, these things kind of live everywhere. As a result of this, there's an extreme decentralization, which means we have relatively limited visibility into who's doing what, when, and where. And in the case of a compromise, we have, you know, minimal, if any, procedures around how to do break glass. How do we recover from this? And so, this was really when the kind of problem space that we started thinking about when we started working on Vault was, how do we get to a better set of answers for those questions than secret sprawl? And so, the kind of driving goal behind the system was to first and foremost get to a single source of truth, right? Moving away from this decentralization, this sort of sprawl to a single sort of broker for all of these values. And if we do that, then we have to recognize those two very different audiences for the secrets. There's our applications and our automation tooling, and they want automated API-driven access. And then we have our human operators who want the exact opposite of that, right? They want CLIs, they want UIs, they want to interact with it in a very different way. And so, as we think about these things, what's the right place to be on the spectrum between sort of theoretically perfect security and theoretically perfect usability, right? In some sense, I think this is a painful trade-off, but it is one that we have to deal with in security land. We can create these perfect systems that are so difficult to use that no user uses them, right? And that's what we see in practice is organizations who will adopt these things, and it's so onerous that one app uses it and everything else just skips it. And so, you have to find that balance of it being practical enough that applications will use it even if it's, you know, not perfect. So, not letting good be enemy of the perfect, or other way around. Last one is modern data center friendly, and what we really mean by that is, you know, pure software. It shouldn't depend on hardware devices because those things are relatively inaccessible if we're in a cloud environment. So, as we translate the sort of top-line goals we had, what does that mean in terms of the feature and functionality of all itself, right? The sort of table stakes for the system is secure bit storage, right? Bits in, bits out, encrypted at rest, encrypted in transit, and that sort of table stakes. Where I think things start getting a little more interesting is when we start talking about dynamic secrets, and I'll get into that. And there's a whole notion around how do we generate these things on demand, and what does that mean in terms of tracking their lifespan and dealing with things like revocation. And then we have all of our good bread and butter security things, right? How do we think about auditing? How do we think about fine-grained access control? How do we think about authentication? And so, while we're doing all of this, what are the set of principles we want to bring to bear, right? When we really look at, you know, what are the things that you have to care about when you look at security, they haven't changed a whole lot, right? The core bread and butter has been the same for a long time, right? We want to provide confidentiality of all data that's stored. We want to provide integrity. We want to provide high availability. But then more so, we want to do things like least privilege, right? Don't give access unnecessarily, because when someone inevitably gets compromised, right? There's a phishing attack and they push the link, right? The things that they're allowed to access will be accessed, and so minimizing what people have access to is a prudent thing to do. How do we get some element of privilege separation? So does everyone need to be an administrator? Can some people be read-only users? How do we sort of bracket who can do what? Privilege bracketing, you know, how do we time-bound access to things? So access doesn't mean access forever. Non-repudiation, how do we get to a place where we have sufficient audit logging that we can say with certainty someone did something at some time? And then how do we apply defense in depth? How do we not put all of our eggs in one basket? And so as we go through all these things, you know, I'll talk about how we kind of weave some of these principles in, because the goal is to bring all of these sort of security best practices into the way Vault works, into the workflow, but do it in a way that's sort of invisible. You're kind of applying best practices without having to be overly cognizant of it. So like I mentioned, the table stakes of the system is just secure bit storage. You want to be able to write something to the system and pull it back out, and that should be secured as it's going over the network, secured as it's at rest. And so, of course, we do that. Everything at rest is AES 256 bit, everything over the wires, TLS12. All of this is pure software, so there's no hardware security modules required. And the goal here is providing confidentiality and integrity. Doing it is actually pretty simple. So, you know, for all that we've mentioned so far, this is what it, you know, writing a secret and reading it back would look like. So here we're just going to write some arbitrary value. You know, bar is equal to bacon at path secret foo. The system's architected roughly like a hierarchical file system. And as we read it back, we get our arbitrary bars bacon back. Simple. So like I said, table stakes, the going gets interesting when we start talking about things like dynamic secrets. One of the challenges with something that's just a static bit store I put in bars equal to bacon is that every client that's reading that value out sees the same value. So to make this concrete, if I have 50 web servers that all go in and say, give me the database password, all 50 will see the same username and password. And so now we have this interesting challenge where it's like, what if that thing leaks? Which node was the point of compromise? All 50 of the machines were using the same credential. All of our developers who accessed it had the same credential. And so now it's really hard to understand, okay, where was the source of the problem? So we've lost that non-repudiation. We can't say with any certainty which actor was responsible. Instead, what we'd like to be able to have is a unique credential per client that's generated on demand. So when we see this client, this token that gets leaked, we actually have a very high level of certainty that actually it was Arman who accessed it at 2.45 p.m. And that credential was unique to him. And so how do we do this? This mechanism within Vault is called dynamic secrets. And so there's a few interesting fall-outs that come from this, and we'll talk about them. At the highest level, the way it works is pretty simple.  Vault ensures that I'm actually authenticated and able to do this and that I should be allowed. And then it goes and talks to the database and says, create a new dynamic username with random user, random user, random password. Restrict the set of things that username can do. So scope it down potentially with a grant statement. And then audit that Arman has created this thing with this particular unique username. And then hand it back to me. So as a user, I'm getting back a username and password. I go and connect to the database using my normal client driver. And I'm off to the races. In terms of the surface area of what Vault supports today, it's pretty huge. Almost any real RDBMS system. Pick your favorite NoSQL systems. Pick your favorite messaging queues. The list sort of goes on and on. And this is made possible because this is a plugin point. So what we've seen is the communities show up and say, hey, I need to support the system. And then they're like, here's the 300 lines of code that connects MongoDB or Cassandra or RabbitMQ. And this plugin is split away from the core. So this is one of the core tenets of the design of Vault is how do we minimize the size of our trusted compute base and have this sort of layered model for defense and depth. And so these plugins have very little access to like the innards of Vault and aren't really a security sensitive thing. And there's a clean separation of responsibility. What this comes with then is if this secret is dynamic and unique per client, we need some way of tracking it. We need some amount of metadata so that we know which client has access to what. And if these things are time bounded, when do they expire. And so Vault rolls all of that up into the concept of a lease. So when we generate a dynamic secret, the value is returned back with a lease. And so this lease says, this is for you, Arman. It's valid for the next two hours. This is your username and password. And the lease is something concrete that we can talk about. So we can renew a lease. We can revoke a lease. We can query what are the outstanding leases in the system. And so this actually lets us start defining break loss procedures. So as an operator, if I know my SQL was compromised, but I don't know who was exactly the source, I can go to Vault and say, you know what, just revoke all of the MySQL leases. And then we're going to do the forensics and figure out the point of compromise later. Or if you know it was just Arman's lease, just go in and revoke my lease, and there's no reason to impact all of the other outstanding leases. So why do this? This seems complicated. It seems like it adds a lot of nuance to the system. Partly it's because it gives us all four of these properties. It gives us this notion of privilege bracketing so that when I give you a credential for the database, it's not a forever credential. I can time bound as a valid for a day, a week, a month, and I can have some guarantee that the client is forced to move on to a new credential, otherwise that credential just gets wiped out. I get non-repudiation. So I know of my 50 web servers, this one was the point of compromise. I can also update. I can bound the time it takes for a secret to update. If I know every client is forced to play ball and respect this lease contract, then I have some certainty that if I change my policy, that within the time bound of the leases, all of my clients are going to get the updated version. And then the last one is it enables revocation because now we have this rich metadata on who has what, and it's unique. If it's a shared secret or it's something that's not time bounded, it's a lot harder. So then we get the sort of security bread and butter, the authentication, authorization, and auditing. As far as authentication goes, this is kind of an interesting part because here we have the split between what machines want to use and what applications want to use. As an application, you're not going to log in with a username and password and enter a 2FA code. You have to be able to use something like mutual TLS, bearer tokens, another mechanism we call app role, which is like a username and password for applications.  And what we do with InVault is basically map down these different authentication mechanisms into a uniform authorization language. So you apply a consistent set of policy over the different users of the system, whether they're an application or a user. And so while there's many, many different authentication backends, there's one way you handle authorization. And the system models this as a default deny sort of system. So it's a need to know basis. And this is really in keeping with least privilege. Unless you really need access to a secret, you probably shouldn't have it because that's just an unnecessary surface area of attack. And then with request response logging, everything can flow through a number of audit backends. So you can configure, you know, one to n audit backends. And the idea behind allowing multiple is it allows us to design the system in a fail-closed way. So we can allow you to set up two different auditing backends and then make the assertion that if none of the backends allow the request to get logged, the system rejects the request. So why would you do this? This allows us to protect that non-repudiation property.  Instead, you'd much rather the system say, you know what, I can't log this, so no, it's better to have an audit trail than to have this sort of gap in the history. So one of the interesting properties of a system like this is that you're pulling secrets out from it for your app to actually be able to function. Your app doesn't have the database credential until it talks to Vault and gets it. And so you kind of care about availability of a system like this, right? So from very, very beginning, Vault 0.1, the system was highly available. At that time, it only supported console. Now it supports etcd and zookeeper and a few other systems for doing leader election. And so you kind of point Vault at these systems and it'll automatically do an active standby model. So one of the Vaults becomes active and services request. The other one's forward things to the active instance. And if it fails, it will automatically flip over to one of the stand-byes. And this is providing high availability. One of the challenges of Vault is if we're providing this promise where we say data is encrypted and transit and at rest, it means we're encrypting Vault's own data. And so when Vault starts, it has an interesting problem, which is how does it decrypt its own data? One of the decisions we made very, very early on was to say this key must be provided online. You can't provide the unsealed key to Vault via a configuration file or a CLI flag, because inevitably what would happen is that that config file that gets managed by Chef would have the key back in it, which then lives back in GitHub. And so the entire sort of system becomes defeated. So this unsealed key, the one that allows the system to continue to function, in some sense acts as this key to the kingdom. Because if we had this key, we could just bypass all of the rules and access controls that are set up in the system. We would just go to the cold storage, decrypt whatever we wanted since we have the decryption key and not bother with auditing and access controls. And so we have to be very, very sensitive with this particular key. Particularly, we have to be worried about insider attack, people who do have access to the system and may have access to this key. So the approach we take is what's referred to as a multi-man rule. And the idea behind it is you actually just don't give out the master key to anyone. So Vault has this notion of a key hierarchy. So the sort of most inner keys, the keys that protect all of the data Vault stores are referred to as encryption keys. These can be rotated online all day and night. Nobody has access to these things. They're sort of an internal detail. That key, that key ring, I should say, is protected by the master key. The master key then allows you to get to the key ring, which then allows you to get to all of the other things that are stored. And so this is really the true sensitive one. This key doesn't even get stored anywhere. Instead, what we do is we split it out into a number of key shares. So we use this algorithm called Sharmir key splitting, where we'll split this one key until, call it five, 10, 50 parts, whatever you want it to be. And you need to provide some threshold of them that are then recombined together to rebuild the master key. So by default, Vault says, I'm going to take the master and split it into five and you need to provide three keys. And then what you do is you give these keys out to different key holding officers at the company. And when you need to unseal Vault, you know, it's kind of like the red October scene, right? Everyone puts in their keys and turns it. And once we get to the threshold, Vault is able to rebuild the master key, which is able to get back to the encryption key ring and boom, we're in business. This can be slightly annoying if we're doing automation and doing things like putting it inside of, you know, systems that are going to move us around. So there are mechanisms for doing it in an automated way. Effectively, we replace our human key holders with a machine key holder that we trust. And so this could either be, you know, a hardware security module that we're racking, or we say, you know what, we trust our cloud provider enough to be our key holder and we're going to delegate this to them. So there's ways of automating this, but out of the box, the sort of Shamir splitting approach is the default. So sort of high level summary, what we really set out to solve with Vault is the secret sprawl problem. It's really looking at saying, how do we get away from this decentralization, this sort of sprawl of definition where we have low access control, low visibility, bring it all into one place, where we can have a uniform policy, uniform way of auditing, uniform way of authorizing everything. And in doing so, we're really thinking about two different types of threats. The bigger, more real one is insider threat. It's, you know, folks that should have some access to the system. You know, it's people who are inside or are going to have access to our private GitHub and all of our keys and passwords are in there, right? And so how do we think about that? That's really about imposing things like ACLs and secret sharing and sort of isolation of the secrets. And then there's the external threat problem. People should have no access to the system at all, and that's really where having a strong crypto system comes into play. And lastly, we really want to do this in a way that applies security principles, but without making this painful, right? Like keeping to that practicality. So for folks that are familiar with Vault and maybe started using it earlier, a feature that we, you know, recently announced in Vault 0.9, which is only a few weeks, is the stronger notion of identity. And so at its core, what this is really about is saying, you know what, there's actually entities who are going to use the system and interact with the system in a sort of a longer-term persistent way. Like Arman as a user is going to interact with this on a daily, weekly, monthly basis. And so there's kind of different things you can do, different ways of thinking about it, when you actually model these entities as a first-class thing. So an entity could be a single person, it could be a single system, it could be a single application. It's sort of a, you know, a bit of a flexible concept in that sense. And what it lets us do is assign policies and metadata to an entity. So we can say, hey, Arman actually has this other metadata I'm going to annotate with, because I want visibility on that in audit logs, or I want to assign, you know, special or maybe more restricted privileges to this entity. And then what Vault lets us do is map this to multiple different aliases. So I might have one persistent entity, which is Arman, but I might exist in many different ways, which is my GitHub username and my, you know, LDAP sign-on, and maybe I have an X509 cert that's assigned to me. But all these things still represent the same entity. So whether I'm using a smart card that represents me or I'm typing in my LDAP password, it's the same person. So I should have potentially the same level of access without having to deal with me in three or four different ways. What this lets us do once we start modeling in this more sort of first class way is start grouping people together. So entities can be placed into arbitrary group hierarchies. So then you can start building sort of a group notion in a much richer way, right? And so, you know, one question is like, well, why deal with this in Vault if I have a notion of groups and something like LDAP? The challenge becomes, how do you start bridging all of this where you have one set of apps where your source of truth, your sort of set of trust, is Amazon. You're using Amazon as a trusted third party to say, hey, this is a web server, and you have users who are coming in via LDAP and you have a separate set of applications that are authenticating against your Kubernetes cluster and GKE. There's no longer a single identity provider, right? There's no single source of truth of who's in my web server group, right? And, you know, my Kubernetes cluster being in my web server group means I'm a pod with a particular tag, and Amazon it might mean I have a certain tag on my VM or I'm a certain AMI, and in LDAP it means I'm in the web dev group. And so what this really starts to let us do is sort of take a step back from the platform-specific nature of our identity and map all of these things in and to sort of a more cohesive shared set of identity. So I can just go in and say, you know what, my whole web dev group has one set of authorization and it doesn't matter whether they're authenticating against my Kubernetes cluster or as an Amazon AMI or they're coming in via LDAP. And so that's what it's really about is allowing it to be much simpler to have this consistent way of doing it so you're not sort of repeating the work three or four times. I guess I should have just gone to this slide. But yes, the basic idea is bridging across multiple different sources of identity as we're sort of losing this single notion of truth. So moving on a little bit now is the Kubernetes auth backend. We released this one with Vault 083 and we worked pretty closely with the folks at Google and other members of the community who have much more expertise in the platform than we do. And so the basic idea at the very, very highest level is how do we bind something in Kubernetes that's going to provide a sense of identity for our application, our pod, to something that Vault can actually verify in a sort of a cryptographically trustable way that we can assert strongly the identity of the other end, basically the caller. And so the mechanism that Vault really relies on here is the notion of a service account. And so what we're doing is binding roles within Vault to roles within Kubernetes using these shared service accounts. The nice thing about doing those is that we have this cryptographic property we can lean on of the JWTs to actually make sure that, hey, is this just a fabricated thing the pod has come up with or is this actually officially signed by the platform that we're trusting? And the nice thing is because these are built into Kubernetes already, there's no additional moving pieces. So for folks who have been kind of following the kind of Vault and Kubernetes, you know, the evolution of the integration story, for a while there was a few different approaches but they required using different service brokers. So there was a few different moving pieces that you had to kind of coordinate and as different APIs would break on the two sides, you had to kind of you were getting the paper cuts of that. So the nice thing here is it's a pure native integration of Vault, it's a pure native integration in Kubernetes and there's no additional moving pieces. So just to flow through what does it actually look like sort of as the kind of the bits flow. In this sort of starting state we have Vault running somewhere maybe within Kubernetes, maybe outside of Kubernetes, doesn't matter for this example, and then we have a pod that's been scheduled somewhere with a service account token. So the first thing that happens is when our container boots, it's reading the service account's Jot, this might be the app or it might be a sidecar that's doing some of this lifting on behalf of the app but we read the Jot then we send it over to Vault and say here's a thing that should prove that I am who I say I am, I'd like to have access to Vault now. Then what Vault does is verify that this Jot is actually cryptographically signed and valid, this thing says it is who it claims to be, and then we go even further and call into Kubernetes token review API and say can you also verify this and return any additional information you might have about this entity. And then finally once we've checked it out everything is verified and looks good to go, then we return the token back to the container. And at this point the container is holding a Vault token. So the nice thing about this integration as opposed to a few others that were sort of considered is this means the application can make use of any of Vault's capabilities. So not just thinking about Vault as sort of a back end for Kubernetes secrets where we're going to just do treat it as sort of a bit locker, but we can actually leverage dynamic secrets, cryptographic offload, dynamic SSL certificates, the whole sort of gamut of Vault is available because we have an actual Vault token. In terms of how we actually configure this thing, it's relatively straightforward. So the first thing we do in Vaultland is enable the Kubernetes back end. So everything in Vault is sort of off by the default so you have to sort of opt in. So we opt in and say I'm going to set up a Kubernetes back end as a system that I trust. And then the first thing we have to do is configure it and say where is the Kubernetes cluster? What are the CA certificates? How do we establish trust of the platform that we want to authenticate against? So this is sort of a one-time configuration. We set up the sort of linkage between Vault and our cluster. Then what we do is for each role that we care about, this might be on sort of a per service level, we define what that mapping is between something we trust in Kubernetes and the policy definitions within Vault. So here we're just writing a demo role where we say, okay, if you see anything that has the Vault service account in the default namespace, map that into Vault's Kube auth policy. So anything that logs in with that service account will get whatever is granted to it by the Kube auth policy within Vault. And then on the side of the client, let's say now we've actually scheduled something that has access to that jot, it boots up, grabs its jot, and it's effectively issuing the API equivalent of this call, probably not using the CLI, and providing the jot and saying, hey, I'm logging in as this demo role. And so once of all verifies it, you know, out comes the token. So here we'll see sort of the standard login where we're getting a token back as well as the policy that's associated. In this case, we can see the default as well as the Kube auth policies are associated with this token and some amount of metadata. So the metadata just helps us within sort of the audit trail understand what's going on, how does this thing tie back to the Kubernetes system. So briefly just to close out, you know, when we think about Vault, there's these three primary kind of things we talk about. The first one is really secret management and how we thread through these credentials, especially in an automated way, right? What we don't want to do is boot a machine and then file a ticket against our security team and say, by the way, this machine is running somewhere, you know, please go get a database password and put it on that machine, right? We'd like to be able to do end to end automation and bring our secrets along with us. The second use case that we didn't talk about at all is encryption as a service, which really is about how do we push the key management out of our applications and up to Vault and let Vault handle key versioning, key rotationing, key decommissioning, and then just provide high level APIs to say encrypt, decrypt, sign, verify, HMAC, whole nine yards. And the final bit is great, we've centralized all this, we've achieved, you know, we've defeated secret sprawl, everything is in one place. How do our human operators access everything, right? We don't want them to interface just with, you know, an API that's sort of painful. So instead, really thinking about identity and access control and sort of user management as a first class challenge within Vault. And so within that we sort of talked on the secret management bits, we talked about the Kubernetes integration, but there's lots and lots we didn't cover. So if this was interesting, please check out either Vault's website or, you know, search YouTube, there's a bunch of other talks on this, or check out the HashiHort blog. Thank you so much. And, you know, quick, there's some stickers up in the front. So if you use any of the Hashi tools Vault included, please help yourself. Thanks so much.


What is going on guys welcome back to the channel and I appreciate you being here today I am back with another destiny top 5 video and today we have top 5 Locky salute drops of the week but before we get into the video guys if you do enjoy it leave no like It really does help out and if you have any amazing clip You think he's worth sharing with the world send it into the email I just linked at the top of the video description, but let's get straight into this top 5 in the number 5 We have a clip sent to by just max 8 7 4 when you finally complete the show Throne solo and you get the ultimate hunter reward Or we have a clip sent to by Ali max zero two three seven When you are casually killing ads in the EDZ and you get that one title exotic you need Have a clip sent to by lumbar zero eight the ultimate forge rewards We have a clip sent to by burn as hell completing the scourge in the past and then getting these wow One we have a clip sent to by all war bird two amazing exotic to win the single forge activity Not bad at all people is the end of this top five if you enjoyed it Leaving a like really does help out if you are new and here and enjoy daily destiny videos Be sure to subscribe and if you have any amazing clip You think he's worth sharing with the world send it into the email just linked at the top of the video Describe share my guys. Thanks as always for stopping by and hopefully I will see you on that next one


Hello, I'm Mike McKee, the Bishop of the North Texas Conference of the United Methodist Church. I am in the sanctuary of First United Methodist Church of Richardson, where we will gather June 5th to begin the North Texas Annual Conference session for this year. The theme for this year's annual conference is vision. We have several guests who will join us as we explore that theme and the vision or purpose God is calling us to in our communities and in the world. Bishop Gregory Palmer, President Bishop of the Ohio West Annual Conference, will speak about the best of our United Methodist identity. His talks will remind us of who we are, where we have come from, and who God is calling us to be. And preaching at the Opening Worship Service on Sunday evening will be Bishop Cynthia Fierro Harvey, President Bishop of the Louisiana Annual Conference and President of the Council of Bishops. And serving as our worship and music leader will be the Reverend Dr. Cynthia Wilson, who currently serves as Executive Director of Worship Resources at Discipleship Ministries. She is a truly gifted singer and she brings passion and energy to everything she does and we are in for a real treat as we gather to sing together once again and I hope you'll be present for each of the worship services. We will also have the opportunity to participate in a special Together We Dine meal at Annual Conference. Reverend Richie Butler of St. Luke Community United Methodist Church established Project Unity in 2017 as a way to unify the community and improve race relations. And today the organization hosts community and educational events throughout the year, including Together We Dine meals across the Metroplex, the state, and even the world. I have had the pleasure of attending Together We Dine events in the past and I am looking forward to breaking bread with you at Annual Conference and participating in these vital conversations. Please register for this year's Annual Conference by visiting the North Texas Conference website so that the church and the Annual Conference can prepare for your presence here. As we worship, pray, and learn together. As we enjoy finally being with each other once again, we will clearly see the vision God has for us and the purpose God has called us to. So I look forward to seeing you on Sunday, June 5th at First United Methodist Church of Richardson as we gather for the North Texas Annual Conference. God bless you and I will see you in June.