So suppose a delegate from Mars comes to the boss and he's hungry and He would like to eat food, but he doesn't know what food is good for him So we're going to actually tell him what is good and what is bad what is healthy food and what is what kind of food? Well, this is good Actually will kill you right so in machine learning means learning from examples positives and negative examples and in order to do so what we would like Actually to build a neuro camera the neuro camera will face objects and will decide whether it's positive or negative Edible or poisonous in order to do so we're going to borrow actually from our own neural system So here's a schematic picture of a neuron it has dendrites that get in forget information and agglomerate it toward the cell body the cell body some the information and Form some sort of non-linear operation and outputs it through the axon and the information Actually flows bottom up. So this is actually the direction of the information. Here's an artificial neuron It looks almost like a real neuron in this case. There are three dendrites You get an input these inputs are multiplied by three weights Then they are summed and go through the cell body through a non-linear function Let me do it again in pictures so that you don't have to follow the map So think of the inputs as three bars the width are the values Then the weights are three bars the heights are the values Then when we take the product we get three rectangles the area of the rectangles is the product We sum them together and we get the yellow rectangle the last operation Which is very important is the non-linear transformation Which is the red body and this is really the essence the building block of neural networks Now we're not confined to a single neuron in this case We have five inputs and then these inputs go to three different neurons that constitute what we call a hidden layer Okay, each of these neuron perform, you know its operation and outputs is transferred to the next neuron Which is at the top now the question is are we confined to one hidden layer or to a single neuron and the answer to this Question is of course. No in the following example I'm going to show a for a depth for neural network with five inputs and four outputs And again the information goes from bottom to top. Okay, and this is called the inference procedure Now once we made a cap prediction, right? We need to learn we need to contrast it with some Information so we get a critique. These are the positive and negative examples and there is naturally divergence between what the network predicted and What is actually the truth and the result is what we call an error vector And the error vector is what we're going to use in order to learn now this slide is complicated But just think about that the information flows from below the error comes from above and then based on a Hebbian rule After Donald have we're going to modify the weights Okay, and this really conjure the process of deep learning.  And you'd like to know what to do for images We build special networks called con of nets which are based on locally tuned neurons that further passed to Regular network and then the output is this case, you know, it's edible. Bloopers are good for you kid. Here's one Example that actually is a for it to professor soon stock. This is using artificial neural networks to detect the structure of neurons of biological neurons Let me conclude with an example of Toy example of a self-driving car on the bottom left. You see a picture. This is the camera that the neural network sees okay, and Just follow the green line Based on the image the neural network decides whether to stay straight or veer to the left and in this example We actually see that it identifies the obstacle verse to the left and then continue to the right now It all looks very simple, but there's plenty of stuff underneath. There is a basic research and massive data sets Computation storage and networking. These are the algorithms. I'm the concrete man. I'm actually building trying to build the basic algorithm Thank you very much


Today, we continue on with Chapter 7, The Reality of the Kingdom. The Holy Spirit teaches one lesson and applies it to all individuals in all situations. Being conflict-free, He maximizes all efforts and all results. By teaching the power of the Kingdom of God Himself, He teaches you that all power is yours. Its application does not matter. It is always maximal. Your vigilance does not establish it as yours, but it does enable you to use it always and in all ways. When I said, I am with you always, I meant it literally. I am not absent to anyone in any situation because I am always with you.  You did not make this power any more than I did. It was created to be shared and therefore cannot be meaningfully perceived as belonging to anyone at the expense of another. Such a perception makes it meaningless by eliminating or overlooking its real and only meaning. God's meaning waits in the Kingdom because that is where He placed it. It does not wait in time. It merely rests in the Kingdom because it belongs there as you do. How can you, who are God's meaning, perceive yourself as absent from it? You can see yourself as separated from your meaning only by experiencing yourself as unreal. This is why the ego is insane. It teaches that you are not what you are. That is so contradictory. It is clearly impossible. It is therefore a lesson you cannot really learn and therefore cannot really teach. Yet you are always teaching. You must therefore be teaching something else even though the ego does not know what it is. The ego then is always being undone and does suspect your motives. Your mind cannot be unified in allegiance to the ego because the mind does not belong to it. Yet what is treacherous to the ego is faithful to peace. The ego's enemy is therefore your friend. I said before that the ego's friend is not part of you because the ego perceives itself at war and therefore in need of allies. You who are not at war must look for brothers and recognize all whom you see as brothers because only equals are at peace. Because God's equal sons have everything, they cannot compete. Yet if they perceive any of their brothers as anything other than their perfect equals, the idea of competition has entered their minds. Do not underestimate your need to be vigilant against this idea because all your conflicts come from it. It is the belief that conflicting interests are possible and therefore you have accepted the impossible as true. Is that different from saying you perceive yourself as unreal? To be in the Kingdom is merely to focus your full attention on it. As long as you believe you can't attend to what is not true, you are accepting conflict as your choice. Is it really a choice? It seems to be, but seeming in reality are hardly the same. You who are the Kingdom are not concerned with seeming. Reality is yours because you are reality. This is how having and being are ultimately reconciled. Not in the Kingdom, but in your mind. The altar there is the only reality. The altar is perfectly clear in thought because it is a reflection of the perfect thought. Your right mind sees only brothers because it sees only its own light. God has lit your mind himself and keeps your mind lit by his light because his light is what your mind is. This is totally beyond question and when you question it you are answered. The answer merely undoes the question by establishing the fact that to question reality is to question meaninglessly. That is why the Holy Spirit never questions. His sole function is to undo the questionable and thus lead to certainty. The certain are perfectly calm because they are not in doubt. They do not raise questions because nothing questionable enters their minds. This holds them in perfect serenity because this is what they share, knowing what they are. And from the workbook, lesson 47. God is the strength in which I trust. If you are trusting in your own strength, you have every reason to be apprehensive, anxious and fearful. What can you predict or control? What is there in you that can be counted on? What would give you the ability to be aware of all the facets of any problem and to resolve them in such a way that only good can come of it? What is there in you that gives you the recognition of the right solution and the guarantee that it will be accomplished? Of yourself you can do none of these things. To believe that you can is to put your trust where trust is unwarranted and to justify fear, anxiety, depression, anger and sorrow. Who can put his faith in weakness and feel safe? Yet who can put his faith in strength and feel weak? God is your safety in every circumstance.   Today we will try to reach past your own weakness to the source of real strength four five-minute practice periods are necessary today and longer and more frequent ones are urged. Close your eyes and begin, as usual, by repeating the idea for today. God is the strength in which I trust. God is the strength in which I trust. God is the strength in which I trust. Then spend a minute or two in searching for situations in your life which you have invested with fear, dismissing each one by telling yourself God is the strength in which I trust. Now try to slip past all concerns related to your own sense of inadequacy. It is obvious that any situation that causes you concern is associated with feelings of inadequacy or otherwise you would believe that you could deal with the situation successfully. It is not by trusting yourself that you will gain confidence but the strength of God in you is successful in all things. The recognition of your own frailty is a necessary step in the correction of your errors but it is hardly a sufficient one in giving you the confidence which you need and to which you are entitled. You must also gain an awareness that confidence in your real strength is fully justified in every respect and in all circumstances. In the latter phase of the practice period, try to reach down into your mind to a place of real safety. You will recognize that you have reached it if you feel a sense of deep peace. However briefly, let go all the trivial things that churn and bubble on the surface of your mind and reach down and below them to the Kingdom of Heaven. There is a place in you where there is perfect peace.  There is a place in you where the strength of God abides. During the day, repeat the idea often, God is the strength in which I trust. God is the strength in which I trust. God is the strength in which I trust. Use it as your answer to any disturbance. Remember that peace is your right because you are giving your trust to the strength of God. So again, today is a day to focus on God and strength and realize that trust settles every problem now. That in the strength of God, there is no situation or circumstance that could trouble the mind in any way. With God's strength, there is no cause for depression, anxiety, worry, concern. Today again is a beautiful sinking inward meditation, opening to a very, very deep feeling of peace within. He tells us, let go all the trivial things that churn and bubble on the surface of your mind and reach down and below them to the Kingdom of Heaven, which is perfect peace where nothing is impossible, where the strength of God abides. This stillness is perfect invulnerability, perfect safety. It cannot be assailed by anything. It is why you came is the purpose of everything. There is nothing but this peace. Do not be tempted to put your faith or trust in anything of this world, not in any desire, not in any outcome. Do not pursue anything in this world. Instead, allow your mind the stillness that is its inheritance. Allow yourself the peace that passes the understanding of this world. Allow yourself the Kingdom of Heaven, perfection, absolute stillness. As you say, God is the strength in which I trust.


I'd like to highlight two accomplishments, Project AIM and our continued work on safety enhancements in the wake of the Fukushima Daiichi accident. With respect to Project AIM, we expect to receive in the spring of 2016 one of the most important products in the Project AIM effort. And that's the rebaselining report that the staff is working on. That rebaselining effort will help us look carefully at the agency's activities in the context of its safety and security mission to look for opportunities to be more effective, to be more efficient, and to focus on the right things as we carry out our work. The second accomplishment is reflected in our continued work and continued success in moving through the safety enhancements that we've identified with the industry and other stakeholders after the Fukushima Daiichi accident. 2016 marks the five-year anniversary of the accident. But what we're seeing in the United States is a continued successful effort at implementing the most important of the safety enhancements, those things that we called tier one in the plan to address the Fukushima accident. We expect the industry to implement the bulk of these tier one enhancements by the end of 2016. And these enhancements provide substantial added safety benefit from the lessons we learn from the Fukushima accident. I found that my past experience has informed the decisions that I've been making because I've been able to draw on some lessons that I learned earlier in my career and also to think about the experience of the agency when it's confronted some difficult choices in the past. I've reflected on the different roles I had as general counsel and as a commissioner and as chairman here at the agency. As general counsel, you're really in an advisory role. You're in the background to some extent. But your job is really to provide options and to inform decision makers. Well, in my role now, I am the decision maker. So it's important for me to listen to people both within the agency but outside the agency. History, other stakeholders who have an interest in what we do and are concerned about safety and security as we are. One of my jobs as chairman is really to be out front as the spokesperson for the agency. And I think that's something that I knew but really didn't understand until I experienced it this year. And that outreach really goes to all kinds of groups, all kinds of people. It goes to the Congress. It goes to the industry. It goes to non-governmental organizations in the public. It's about engagement in the international community and with our fellow regulators in the agreement states. And I also try to get out to the facilities that we license and to visit our regional offices and to meet with our inspectors in the field. So that takes time. I find it very interesting. But then I realize I also need to come back and work on the decision making that the commission needs to work through. We have some interesting work that's going to come before the agency in 2016 and will extend beyond. We've seen interest in establishing consolidated fuel storage sites and expect to receive applications from companies starting in 2016. We also expect to receive the first application for a design certification for a small modular reactor or SMR towards the end of 2016. And that will be interesting work for the agency as we also try to make sure that we're prepared for the possibility of advanced reactor designs that may come in the next five to ten years or so. I think we'll have the challenge of both right sizing the agency but making sure as well that we retain the core expertise and core strengths that we've always had as an agency in terms of doing good technical work and assuring that we have the right kind of oversight in the field. We're at what I'd call a generational shift. You have fewer folks that look like me in the agency. We have a great new staff that we've added but there's always a challenge and this is something not only in the NRC but in the industry itself in terms of making effective connections through knowledge management between people who have substantial experience and those who are newer to the agency. We need to keep continued vigilance in the coming years over the threat environment. We have a robust security profile at nuclear power plants. We're working well with others with respect to source security but those are areas where we need to continue a focus. Another area involving security is the issue of cyber security which is on many people's minds. We established requirements a number of years ago for nuclear power plants to have cyber security protections and we're working through the first phase of the implementation of those regulations but that is an area I think all would agree that we need a continued focus on just not only in the nuclear industry but in other parts of the industrial sector in the United States. I have high confidence in our ability to meet our mission. Across the course of my career I've seen us move through an address and successfully learn from a number of challenges whether it's been particular events such as the Three Mile Island accident or the terrorist attacks in 2001 but also preparing ourselves for things like license renewal, changes to our licensing process for new reactors. Those are things where we've used our experience. We've thought carefully about how we can best address the challenges and I think we've been very successful at meeting them so I have a lot of confidence in the agency to shape itself and to respond and also anticipate the future in the best way we can. My confidence stems largely from the quality of our dedicated staff at the agency. We have extraordinary talented people in all kinds of disciplines, scientific, engineering, law, administration and they're the ones who contribute as a whole to the success of this agency. They have in the past and I know that they will in the future.


So thank you very much for coming and not leaving that means that whatever you want Okay, this is a turbo talk. It's not gonna be a one-hour long presentation. I think someone screwed up in the schedule I didn't receive a check. So It is what it is First of all, there is no agenda in this presentation. No crap about we are gonna see these and stuff like that No zone what's who art of war the stuff in this presentation either? I'm trying to make this campaign of no more free art of war stuff in security presentations It applies to everything. So come on. Give us a break No more free campaigns. See the dollar sign at the end. I Know that we have to pay bills, but come on man give stuff for free. All right No history of web scanners and no drama, but I think we have saw a little bit in the beginning of the presentation I'm sorry for that. It's not my fault my name if I interest Kind of 10 years enjoying IT security. I'm not going to say that I'm a pro. I think that's too high I'm proud member of the metal exploit team. This is my address and proud member Tambien of a country of Colombia So how many Colombians are here? All right, one two three and that's it. All right, man. Thank you very much By the way, please visit Colombia. Don't throw CNN Go there. I'm evidence that if you go there, you will not be kidnapped Well, no for now Okay, the intro this thing is about a web assessment as metasploit modules We're gonna run the models by hand or automated Still on early stages. They meet to the crisis. I had to work. I have to work a lot so my free time is getting smaller and HD more in metasploit in the conference sector in 2008 He said that well expect a big announcement soon. This is it and get in a hold of this I was my I was like shaking now. I'm shaking less All right So why I did WMAP?  Sorry the guys if you are from web inspect or one of those Sorry, I don't like it.  Sorry for the world But it is what it is right, I Also hate that it can only be run on Windows.  But the truth is that when sometimes a high is just a low and a couple of lows can get you high What is the real impact? That's why we're playing with metasploit. This is just a plugin So we can jump from the testing part directly to the exploitation methods, okay? How everything is working today? So I say like let's do something really cool. That's the damn graphic that I make So when you play with these tools with the net with the commercial tools I Like the commercial tools is just that I see that they can do better. Okay Now it's like playing with lemmings You just see that then packets are their quest just being gone through it and they just fall and nothing happens I cannot fit it take those results and feed them back to the engine.  So yes, you can use it sometimes But what is the good way to do it? Well, let's go back to basics the proxy the man in the middle proxy There are tons of proxies out there fiddler bar paros, etc man is awesome awesome awesome tools that actually does a much much better job than The one above, okay Okay, what is the objective of Dolima being a simple tool Command line based a way to testing methods with exploitation methods You can use it as scanner, but it's just an extension of the meta split framework and Again something easy to extend and modify. I didn't like the API is to Adjust or add new extensions to some of the frameworks out there So it takes forever to build something. We just need something easy. Okay Actually, we have to do a campaign like no more difficult tools or no more complex tools. Let's go back to basics dudes right Note no more web assessment no more the more web assessment tools the better For example w3af is awesome. That's Andres Bianchos is if he's over there or the people that contributes. Thank you very much a sequel map is awesome Bernardo dameles if he's over there or some people over that work in the project is around here. Thank you again, too and Each tool has its own limitations Choose the damn tool you like. It's just a tool Poor worksman playing his tools. I don't know who said that but I just copy and paste By the way, I left a blank field in the bottom so you can write there whatever you use Okay, I'm not saying that w-map is the shit. Okay. We are just starting and even if I finish with w-map still There are two good tools out there. So just put whatever you want there What is the design? The design is awesome. What I did is just Decouple everything just separating every single thing that is out there so I can have multiple Spiders or clients or devices that connect to the proxy The proxy just going to store every single request in through the metasploit database And then from the metasploit database W-map that is the plugin will take that information It will create a tree of the website and will store launching all the auxiliary modules one by one in the order That makes sense. That's it Actually, that's the way that the other tools does but This is simple. So why do we have to pay for that much? Okay, and then we just launches the test to the target and we can take the results because we own those results And we can do whatever they want like Put it again into the database or if you have something additional to do just throw it through the proxy And then you can test it again Easy the proxy the proxy only store requests in the database Initially when I started W-map I did a small plugin for Fiddler. Fiddler is made by Eric Lawrence from Microsoft I haven't released the plugin because it's ugly. So Still needs some improvement on the GUI because it's Windows patch for the patch that I use right now is a patch for a rat proxy from Elkentuff and and Also, thank you to a spin bad that has made some additional plugins for a Borg proxy Okay, but if you have anything that can take you the requests and store it in the database Do it man This is the Schema that you can use to store everything is very simple the host the port what else just that I don't see very well over there Or I'm here. That's why I use glasses. So I have to check this Okay, so we have the path we have the headers the query the body the response the response the response header The response code and the actual response when it was created. That's it. If you can put it in that format You're good to go Well, this is a small list of the auxiliary models are I Located right now in the metasploit framework Again, like I told you early this is on early stages, but if you see it the plugins that are here the modules They have the basic functionality that you need to have like a good understanding of your target and also You can use them Modified a little bit and create new tests without any problem. So please go to this directory Take one module. Just check it out. It's very simple to understand And all thanks to HD. Okay, it's not me. I'm just reusing his his work Let's start with some explanation with the With some type of the modules because I think the most important parts are the actual modules not the actual plug-in The plug-in just again just runs all of them on a certain order and This is just one basic example on how to run a Specific WMAP module just use it as any other module on the framework You set the specific variables for the module the target remember that each module can be used as a scanner you can put a range over there and We define the port In this case, we are going to use the WMAP SSL module So I'm going to say that we are going to use SSL and let's run it the specific example Just what to mention that I use What it does is going to query a bunch of certificates I'm just gonna show me the information from them It's a very good way to see the virtual hosts So if you have a bunch of IP addresses that usually happen on the pentest and you find those weird websites when you say, okay The web page doesn't show what is the virtual host this? This is a good plug-in to use. Okay a good module The other thing is for this example. I use well here. It says that I that is like the x y c Demco.com company, but in reality is one of those banks in America So what I did is I run it through a couple of IP addresses and just check the information that it returns it returns stuff like They're running a J-Balls and web sphere. So it's good good information to have I should only start for information gathering in this case. I'm gonna show you some The way that the that another module works that is the way that we detect directories on a web server Usually what happens is that the tools that you have out there they only check for the return code That sucks man because sometimes not nowadays. They have their own error pages. Well, thank you to HD and my work We were able to create a small module that has Signatures to check the error page. So if it returns whatever code it is it will really detect if Directly exists exists or not The second important part of WMAP is the data storage and reporting I think the most important part is that with the data We are we were used to only work with the data on the way that the developers were supposed to To give us access to in this case You can store anything you want in Put any name on it. That means that is in the future. There is someone that devices a new Attack technique or a new vulnerability and it requires a specific cool name then you can put it over there and Have that information is started on a tree that you can later reuse for analysis Like for example, let's check a little bit about this code The only thing that it has to do is is is going to find the report on The on the first part. It's gonna find the report ID is Based it on the host on the port number and if it uses SSL or not And then when with that report ID, you will be storing all the information that you want On the format that you see displayed is basically type Description that's it. Let's talk now about the WMAP plugin But basically what it does it identifies possible targets From the database that that all the requests are stored we build a target website with data structure and We run the models in a specific order reporting we already talked about it and If you see in the bottom of the presentation does the initial command to run WMAP One thing if we don't have time to run the demo Then the only thing that you have to do is to load the DB WMAP and then you just say WMAP Target to select and WMAP run and that's it Okay, there is no thing to select what plugins do you want to use you can do it on In other different ways, but it's very easy to use. That's what I wanted to do So how WMAP works basically we have a specific mixing where you can define Specific types of modules. So we have this type of Modules and what it does is I take the module I put the type WMAP the text what is the type and it will launch it at the right time For example, the first thing that it will do when I run something is that it will run every single module that is defined as a WMAP scan server, okay Then it will try to scan every single mod. It will run every single module that is for files or directories actually they run kind of At the same time then modules that require to execute at every single query and You see that I have something different that is called unique query because what it does is it will create a Signature of the query that you have a get request or a pulse request And I don't want to run it every time only because the values changes I only need to run it just because I see the same parameters all over again So I can do that just with the WMAP scan unique query Then also I have a Something to scan the body or to run or to work with the body of their quest The headers and something called generic that is always run at the end So this is where we can analyze the all the information that we have been retrieved Take that information and put it back into the database or fill it back to the proxy Okay, one basic example is the WMAP generic email extract This is the only type of modules on the framework that you cannot run it interactively or through the command line Because the only input that it takes is the database and this is just extracts all the emails the only email addresses so you can take it and run and use some social engineering with the e-mailer and using the tools that Jabra has Help me develop For the for the framework Okay, so how to build a WMAP module? Just take a metasploit module. It doesn't have to be an auxiliary model can be any module not only HTTP Just include the the mixing type and that's it so if you see there You just put the type that you need take this one run it Easy, right? Okay, what is the demo? We only have five minutes. Okay, so what I did Is I took a specific IP address on the internet and I did a small video But I think you guys deserve respect. So I did something like really a website on the internet Not something that I crafted here because that's kind of cheesy And let's see how it goes the full screen there it goes all right So what is what we're doing right now? We are learning the plugin we connect to the database I'm showing the comments that we that it has available We're gonna take the we're gonna define the targets So the first thing is we are going to reload Reload them from the database. That's it only one Sorry for the guy from the IP. I'm only browsing his web page. Okay, so no attacks were performed And the thing is then I just print the available targets And I'm going to select the first one actually I selected first before printing, but it works the same And what I'm going to do right now is I'm going to set the variables That usually are required by the modules again by the modules not by the by the WMAP plugins So these are the modules that are available You can enable and disable them through a profile to so check the documentation of the WMAP And now what I'm going to do is I'm looking at the website structure This is the most important part If you guys do a web app assessments, you will see that the structure is is where they the gold is always Then again, I like I said you set different variables in this case I'm setting the extension of That I need to play with Then at that time I receive a call from my wife, and then I have to stop a little bit. I kept going By the way, I'm gonna be at that so Thank you very much. I said the virtual host. That's all the other cool thing Some of the tools don't allow you to play with the virtual host. So You have a lot of range here to play with I'm checking the domain For some of the plugins that for example performs on brute forcing to identify additional virtual hosts Then I receive another call from my wife Yeah And there he goes now I'm going to type WMAP run the help Come on, man.  That's it, man We are not training the not flying brains here. Okay. That's why I think what that's what the type of tools that we require And I'm going to stop the demo here. Okay, because the time is over. So thank you very much, man


Hello and welcome to the Open Source Summit 2020 Europe. My name is Aftar Sirathan. I'm a developer advocate at Timescale. And today we're going to be talking about open source software and observability. Thanks so much for coming to this talk. I'm really excited to delve into our topic for today, which is going to be purpose-built observability solutions using open source software. Now, this is an entry-level talk. So you're probably watching because you're interested in open source and monitoring, but haven't deployed yet, or just early in your deployment journey.  You've come to the right talk. But regardless of your experience level, I'm going to make this talk valuable for beginners and pros alike. So let's get started. Before we get into the content for today, I just want to tell you a little bit about myself. So my name is Aftar. I am from South Africa, but I'm speaking today from New York City in the USA.  If you can put that in the live chat, that would be really great. This is an international conference. So it's always good to see where people are from and where they're tuning in from today. But the second thing is that I'm really interested in how to use technology to empower people. That's why I really enjoy being a developer advocate at timescale. And part of my job is to learn new things all the time. And I document that journey on Twitter and on my website, aftar.com. So if you'd like to follow along, you're more than welcome to. So let's get into the roadmap for today. There's four main things we're going to cover today. Four main topics rather. So the first one is why use open source software for observability. Under this topic, I'm going to be talking about why observability is important. Give you an overview of the areas of observability and compare open source tools with all in one proprietary tools. And we're going to look at why might you want to use open source tools for observability. In the second section, stories from the field, I'm going to be giving you concrete, real world examples, hopefully to inspire and motivate your own deployments and inform how you think about your monitoring system needs. Under this section, I'm going to be going through how three companies have set up production observability systems using open source software and give you the pros and cons of each of their approaches. In section three, we're going to get our hands dirty and build a sample monitoring system for metrics in five minutes. And here I'm just going to show you how to get up and running quickly with your own open source monitoring stack in five minutes to monitor Kubernetes cluster. It's going to be a sample system with tools like Grafana and Prometheus. And then lastly, we're going to do a recap of what we covered today and have time for your questions. So if you have questions at any point during the talk, please put them in the chat or using the question interface and we're going to get through it at the end. Now just to set a little bit of expectations, this again is an introductory talk. So I'll try to assume as little as possible and leave you with a lot of actionable next steps for you to continue your open source monitoring journey. My goal is that by the end of this talk, you will have the inspiration for how to build your own monitoring system from open source components based on the real world experience of other people, as well as a foundation in the different tools and different tips and advice to help you get started quickly. So I'm really interested to hear what you're hoping to learn from this talk. So if you can put that in the chat, that'll be really great and it informed me to understand your motivations. So if you can let me know what you're hoping to learn from this talk, just put it in the chat. It's good to see what different people are looking to get out of things. Okay, so let's delve into the first section of today's talk which is why use open source software for observability. Firstly, I just want to give you a quick overview of why observability is important. And at a high level, observability is using outputs from your system to better understand its internal state and its impact on the users. Now there's two main reasons why observability is important to developers. The first one is that it helps solve small problems before they become big problems. And the second one is that they help preempt issues before they affects users badly. There's a common saying among developers in the observability space that I want to see my problems before I hear about them from my users. So it's really a lot of preemptive by giving you information to preempt the problems so that it gives your users a better user experience at the end of the day so they don't have to deal with things like slow software or shutdowns or any other problems with their services or the product that you have. Now, what does this view into the internal systems, into the state of your internal systems allow you to do? So in other words, what does observability allow you to do? Now there's three quick examples that I'm gonna take you through. The first one is that it allows you to optimize things like your resource usage on your infrastructure and applications. The second one is a classic performance debugging and optimization you want your systems to run as quickly and as performantly as possible. And the third one is you get to respond to problems faster with things like an alerting system and things like that. Okay, now given the importance of observability, it's really important to choose the best tools for the job. These are tools that allow you to get value out of the box but also scale with your needs as your software gains more usage and your deployments become more complex. Now, generally you're choosing between all in one proprietary tools or a do-it-yourself approach using open source software. And many developers just turn to the well-known names in the observability field and those are usually proprietary software tools like Datadog, Splunk, New Relic, there's number of other ones that I can mention. Now, the problem is that while these tools are good, they're not a one-size-fits-all solution. And especially in today's times with COVID impacting the economy, money is scarce, budgets are cost, even in the tech sector, these tools aren't a viable solution for many teams. And the main reasons are that they're too expensive, they're too rigid and they're not useful enough for the needs of smaller fast-moving teams with specific needs and also teams on a tighter budget. So let's take a look why in a little bit more depth. And there's three main themes that we can compare proprietary tools and open source tools in. So I'm gonna introduce these themes to you now and you're gonna see this coming up later in section two stories from the field. And just a note about these points, the points that I'm gonna cite right now come from my own experiences but they're also a distillation of feedback from developers that I've talked to in my job as a developer advocate but also just generally from what I've read online. Okay, so the first factor is costs. Now most proprietary tools have a very opaque, difficult to predict pricing model. So for example, many solutions have per metric or per series pricing and this can leave you with large unexpected bills because these are the kinds of things that you're not really sure how to predict, especially when you're adding a new deployment or you're adding a new component to your system. Now this also develops, it also limits developer freedom because with per metric or with per series pricing, you may not be able to afford as many custom metrics as you need and this might actually limit the effectiveness of your observability efforts because you won't be actually be able to see the things that you wanna see because you're limited by price. Now let's contrast this with open source tools and open source tools are free or cheaper for hosted open source solutions and the big thing here is that cost is a big factor for a lot of smaller startups, smaller teams and as I mentioned earlier, during times like this where coronavirus has caused a lot of budgets to be cut, companies are really looking to save money wherever you can and so that's what makes open source tools a really attractive solution, especially during times like this. The second point is on flexibility and while many proprietary tools provide a good out of the box experience to have things like pre-made dashboards and whatnot, a lot of the problems that I've heard with them come one, three or six months down the line and the main downside is the limited features and customizability of the platform and for example, here you really limited the confines of what the platform has. So for example, the templates that they have and you might not actually get the most useful information about your system because of the fact that you can only really measure what the tool can allow you to measure. The other thing is that you might have a lot of unexpected maintenance work pushed back on you, that's another thing that I've heard from some of the users but the point that I put here in parentheses is scale and this is because as you start to scale up, pricing becomes really prohibitive because at this kind of per metric and per series, it might not necessarily allow you to monitor everything you want because in order to do that you would have to pay much more. And so this is where you get the worst of both worlds where you don't get something as flexible that allows you to monitor everything and it's also very expensive. Contrast this to open source software where traditional one of the benefits of open source software is that you have total control and extensibility where you can fork and modify the code as well. And then lastly on community support, one of the things that I heard during my user interviews with the developers which you're gonna hear about in the next section is that oftentimes these all-in-one tools and the companies behind them can be slow to release useful features and things that your team might want. One of the quotes was that the best you could hope for was an hour-long sales call and a vague discussion of how much a feature might cost and when they eventually deliver and then you're left waiting for six to 12 months and you can't actually use the thing that you want to. You might have actually experienced this before. So the main con there was that it's slow to release useful features and not really responsive to user feedback. Contrast this with open source tools where especially for projects like the CNCF projects, the Cloud Native Computing Foundation, these projects tend to have a breakneck pace for new features and fixes and just progress in general. And in general in the open source tool space, especially in observability, there's a high velocity of new and useful features. So you have the support of a large community and what this means is that the tools become more mature with the larger community that they have and they also become more well documented. And this makes it easier to bootstrap from the community best practices. So right now you're probably sold on the value of open source software for observability. You like the cost aspect, you like the flexibility aspect and you like the community aspect. So this is you going on Google to get started and you're searching open source observability tools. But then bam, you're hit with a seemingly endless array of projects and configurations. And this is one of the things, one of the downsides of open source software is that there's so many options to choose from and things can be a bit confusing. Do you use Fluent D or do you use Fluent Bit? What's the difference between these two? Do you use Prometheus? What would you even, what components do you pick and what do all of these things actually do? So this is one of the areas in which I wanna help you today. Let's get started with section two, which is the main event stories from the field. Who better to learn about open source software and observability systems from than the teams who've actually built and deployed them to production. So today I wanna share the stories of three companies who have actually built production of observability systems using open source software. Now for each company, you're gonna learn why and how they've gone about implementing their open source observability stack and how their configurations have fared as well as the pros and cons of each approach. Now I'm sharing these stories in order to give you a concrete real world example or many examples in this case to inspire and motivate your own deployments and to help you think a little differently about how you view your monitoring needs. So the goal here is really to help you to understand when, where and why to use open source technology. We're gonna cover for each company these four things, which I actually already mentioned. Okay, so let's get on to the first company, which is when I work. So when I work is a employee scheduling platform, they're based in the USA and their use case is metrics monitoring. And the reason why observability is important to them is because if their platform fails, if they have a problem with their software, employees don't report to work. So the users, the employees of their users don't actually report to work and the customers will actually be really mad because of that. So it's pretty high stakes and it's actually really important for them to build something that allows them to, one, make sure that their systems are performing properly and really get visibility into what's going on. So let's look at why they decided to use open source software. Now the story is that when I work was originally using Datadoc and they decided to move to an open source monitoring stack for two reasons. The first one is a pricing model. So they didn't like the paper series and paper metric model that Datadoc had. And they ran into the limits of this pricing model once they started to add custom metrics. And as I mentioned, this actually stifled developer freedom because they couldn't have as many custom metrics as they needed. And that mainly gave them less insight and less useful information about their system. The second point of why they moved was dashboard lock-in. And this is because at when I work each product team maintains their own dashboards. And so that meant that moving and sharing required a really big effort. And that's because the dashboards are saved in the proprietary system, Datadocs UI, in their file format. And they have to be rewritten entirely if you wanna move. So it's basically you gotta do all the work again. You're not actually benefiting from any of the work you've put in the specific tool. And they really wanted something where they could just switch providers overnight and not have to rebuild the whole system. And that's actually something that they've done with their new model. Okay, so let's take a look at their observability stack. So I'm gonna go through how folks like when I work have set up their stack. And then I'm gonna show you how to get up and running with your own open-source monitoring stack using some of these components. For example, Prometheus, Grafana, TimescaleDB in five minutes during the demo in the next session. So just hold tight for that if you're interested to see how to build some of these things or how to get a system with some of these things yourself. So there's really a few things to mention in this architecture diagram. So let's take a look at how where the metrics are coming from and what they're using each of these components for. So the first one is that they use AWS CloudWatch to collect metrics from their managed services. So that's not pictured here, but that's also a source of metrics. Then they also use a tool called WorldPing from Grafana which provides external monitoring and Pings to make sure that their system is up and running. So that's a WorldPing tool. That's not something that we mentioned before people do use it. And then they use Prometheus as the main metrics monitoring and metrics collection system to collect metrics from their Kubernetes clusters. So you can see here from their app cluster, they have Prometheus here. And then their GitLab cluster, they have two Prometheus, two Prometheus, I think that's the correct term here as well to monitor the actual monitoring system. But then also in the application itself. And so what happens is that these metrics get collected from the application and get sent to timescale DB in order to aggregate them and store them for the longterm. And they also take metrics from the monitoring cluster as well. And you can see somewhere there's an arrow here. These metrics also make their way to timescale DB. Okay, so timescale aggregates and stores the metrics for the longterm.  So they have both a staging and a production version here. And they also have Prometheus Alert Manager, which is the thing that they use for alerting, very popular with Prometheus that allows you to define a bunch of rules on your metrics and when you can trigger alerts. And that's set up to Victor Ops and their Slack so that engineers can get notified. And a bit about why this deployment is important to them. When I work said that their customer-facing deployments are actually auto-scaled based on the usage metrics. And then also another thing that this provides them is their status page on their website is actually driven by a Grafana query. And another example of where they've used these metrics is they've actually had to roll back a broken release before it caused a lot larger problems. And that was all thanks to the real-time metrics that they were picking up from Prometheus here. So you can see the setup has actually been very useful for them so far and they've quite enjoyed working with it. So let's get into some of the pros and cons of this setup as reported by when I work. The first big pro is the simplicity of service discovery and custom metrics within Prometheus. That's been a huge advantage for them. The second pro is the variety of data sources supported in Grafana. So Grafana really has allowed them to read and compare and alert from multiple data sources. And previously they've had different data sources but they were never able to fully combine them despite things like the data dog integration with AWS. So that's the second benefit, the variety of data sources in Grafana. The third major benefit is allowing them to test new versions of their software in parallel with the production deployment, test new versions of the observability software and versions of the new tools that are coming out. So for example, there was a new connector between Prometheus and timescale DB and so that now they can actually test this in parallel with the old connector. So you can actually see what performance improvements and things you're getting. And then the fourth point is the fast progress of the CNCF projects Prometheus and whatnot. The advantage here was that there's always new fixes and there's always new features being released from these projects and there's always a bunch of useful stuff to use but limited time in the day. So that's actually a very good problem for them to have. On to the cons. The first con is just the time it takes to develop the system. And so they took them a while to understand how they can fit together different pieces of the observability puzzle and they needed to decide which components work best for them. And the second con was the time to learn some of the advanced features. So things like Prometheus rules and timescale DB continuous aggregates. And the issue here was really teaching the rest of the team how to use things like Prometheus rules and timescale DB continuous aggregates. And that's because they have sometimes a steep learning curve and that means that they went through this phase of a lot of inefficient dashboards where they had a bunch of duplicate or inefficient queries powering their dashboards whereas if they use some of these advanced features which they now do, many of these queries could have been aggregated ahead of time but it's just being difficult to teach people how to use that. So that was one of the corny time to ramp up with some of the advanced features. Okay, so that ends the first company. Let me know in the chat what you've learned from it. Jungle is the second company that we're gonna feature in today's talk. And Jungle is a energy and industrial monitoring and forecasting company. They're based in Portugal. So I thought it'd be good to use a European company for this conference. And their use case involves metrics, logs and traces. And here the main thing is Jungle provides you that visibility for their users to gain control of their assets at a very high level. So let's see how that actually manifests. So the motivation for Jungle to use open source software is mainly for these three reasons. The first one was cost. So they were budget constrained and so they decided to pursue open source projects because of the cost factor there. The second point was the previous experience with some of the technology. So some of the team had previous experiences with certain tools and the integration with Kubernetes. And then the third and the reason that really sold them was the community support. And so as they built a proof of concept they found that the support of a big community around major projects like Prometheus and FluentD they had constant updates and they were always releasing new features and bug fixes. And they also found that many of the new features many of the features that they actually wanted either existed already or were being merged into upcoming releases. And so that really made them confident about the maturity of these tools as well as the quality of the documentation around in order to actually use this in a production deployment. So that's why they moved over to open source tools. So let's take a look at Jungle's observability stack a bit of a less complex diagram than the previous one. So Jungle stack is composed of a mix of metrics logs and distributed tracing between the different components. So the first point is that they use Prometheus to collect the metrics they're using Prometheus to collect metrics in order to gauge the health of their services. The second tool that they're using is Yeager which allows them to trace requests from their product in order to see how they travel through the various microservices. The third tool that they use is FluentBit and this is a tool for getting the logs from their container in order to help them detect different application errors. And then they also use timescale DB once again to store and query their metrics data. So what they're doing is sending all the metrics data to timescale DB and they deploy a single timescale DB instance per Kubernetes cluster. So this is all running inside Kubernetes as the icon here is supposed to indicate. So you can see here in this stack and then they also use Grafana, sorry for displaying metrics and logs information so that they can visualize some of the things about their systems. So you can see here this stack involves a little bit more areas of observability versus just a metrics monitoring stack that I showed you previously but that's how you can fit together some of these different components. So let's take a look at the pros and cons that they found so far. The first pro has been the operational cost they found is just much more cost effective compared to paid alternatives. The second one has been the ease of maintenance. So this is something that you might not often hear from open source projects but this is mostly around the great community around the core projects like Prometheus and FluentD and Yeager. That's made it easy for them to get updates and new features and they've really enjoyed that. And the third benefit has been the easier ingestion and querying of large volumes of data on timescale DB. So they're able to have this large volume of data on a single database without the downside of maintaining the integration of several systems by themselves. Okay, and the main con that they asked me to relay is that it really took more time to build than other of the box solution and this was good for them because they had planned ahead and they had a lot of time in order to execute the system but just be wary of the time and effort that you might need to build and you don't want a situation where you're pretty crunched for time and have to try and put together a solution that needs more time to actually become effective. Okay, so that's the case of Jungle. And next I'm gonna move into a company that I know very well because I work here, Timescale, the makers of Timescale DB. All right, as I mentioned, Timescale are the makers of an open source time series database called Timescale DB. They're based in the USA but they have a global team and a global user base. There's actually a lot of people in Europe on the engineering team. And this is a bit of a meta example because it's a case of an open source company, the makers of an open source database, using open source software to monitor its own systems. And in this case, the system that is in question is the managed database as a service product called Timescale Forge. And there were already three reasons why they decided to use open source software to monitor this database as a service product. The first one is cost. They found that Datadog was too expensive for their needs and the budget that they had. The second one was customizability. And they found that even with all-in-one solution like Datadog, it wasn't possible to get the exact views that they wanted of their data. And they also wanted a more integrated solution where they also wanted a more integrated solution where they could do things like linking with their internal identifiers. And they felt that they'd be more comfortable doing this with open source software and building their own system. And then thirdly, they found it was easier to actually bootstrap from the open source community. So again, having a big community around some of these projects gave them things like free charts and allowed them to have a more plug-and-play element as well as queries in order to monitor certain things where you don't have to come up with your own queries. There's existing ones there to monitor common things. So let's take a look at their observability stack. All right, so again, pretty complex diagram like when I work, but we're gonna make our way through it one by one. So the first thing to take note is of Prometheus. Forge, timescale forge runs in Kubernetes on AWS. And so they monitor things across the AWS stack. So you have node exporters that on the different EC2 instances that are running. And they also run Postgres instances for their customers because that's the nature of the product. And so they need Postgres exporters. So all of these things in the Kubernetes node here, you can see prom exporters. This is actually sending data to Prometheus. The other thing that they have is the default metrics exporters for their clusters. So you can see here in the Kubernetes master, you have your KubeState metrics and your Kube metrics server here. And then they use Go Prometheus to monitor the other microservices. And then they have alert thing using the Prometheus alert manager. So everything in yellow here is supposed to be Prometheus related. So alert manager here to have rules on their Prometheus data in order to trigger alerts. And this sometimes wakes up my team member Sam at 2 a.m. to go and check out what the problem is here. So that's how the flow goes there. And on the next, onto the next point, so that's how they use Prometheus. You can see a number of different exporters that export metrics in Prometheus format. The second point is the use of timescale DB. So they use timescale here again to store metrics for the infrastructure and their product itself. And it allows them to have things like Kubernetes metrics to power the user facing graphs in the application. And essentially they store a lot of metrics in timescale DB, which they wanna join with user data. So let me show you an example of that. So here's an example of the hosted and managed product that timescale has, which is timescale forge. And here, this is just an overview of the metrics of a particular database that I have. In fact, let me just show you this in my timescale forge setup itself. So I'm just gonna take a certain instance that I have here and then I can view metrics about it. And so these are metrics that I actually collected from a by Prometheus about the service running in Kubernetes. And they're put in timescale DB so that you can power graphs like this where as a user I can actually see all I wanna know my CPU and my memory and storage at a certain point in time. So they have to get that information from somewhere and then put it in a form that the user can actually make use of it. So this is why they use tools like Prometheus and timescale DB to do this. Okay, another example is the ability to join the data from the metrics and user data in order to make dashboards to get information about specific users and how they're using the system. And that's because when you're dealing with a managed product, you need to be able to support the product quite well. And so they're able to leverage open source software where for example, in this case, my teammate Rob, this is the different services that he has and you can see here like the status of them just to help support the project a bit better. There's also a number of other dashboards some of which I can't actually show you publicly but this is just the taste of the fact that you can actually join your metrics data with your user data to get more informative dashboards and just views of your information. Okay, so that's some example of how they use the timescale DB to join the metrics data with the user data. Next, they use Grafana as a visualization for the DevOps data. So in this case, the metrics data but also they use it to visualize platform metrics and things about user behavior and whatnot. And here's an example of a dashboard that's made in Grafana about a sudden instance. So again, this is important for things like supportability and so you can see the information about the services running so that if a customer says, hey, I've had this problem, you can actually see, okay, they've actually had this problem. This is the information about the database. So in this case, you can see things here like the network traffic every five minutes, a disk usage, CPU utilization, the instance state, whether the service is enabled or disabled information about it. So you have tons of dashboards like this for the customers and this is all powered by Grafana and Prometheus and Timescale DB itself. All right, so that's the Grafana visualization use case. I think that you've given a good taste of what do these dashboards actually look like. To end up here, Timescale also uses Yeager for tracing on pod requests. And then for logging, they use elastic search mainly because it's very battle hardened and it allows you to really search and index your log data really well and they use the Kibana front end in order to visualize their logs. And then they use FluentBit. So often you'll see the AlkStack, but in this case, they've opted to use FluentBit instead of LogStash as the agent to pre-aggregate the logs. And here that's just pulling the logs from their Docker containers and shipping them over to elastic search. So that's an overview. Pretty complex architecture, especially if you're seeing it for the first time when you're just starting up, but hopefully that's a shed some light on the different components and what they're used for. So let's get into the pros and cons of this setup. So the first one is just the pace of upgrades. The team have found that they've benefited a lot from the constant improvement and quote unquote, when they upgrade things, they actually get a lot better. And so the Prometheus services, projects like Prometheus are constantly evolving and they're really benefiting from the fast moving upgrades that they're getting from it. The second point is the documentation and the community. So the benefit that they've felt here is that you can actually Google a lot of your problems and you'll find that there's a lot of documents or GitHub pull requests describing the problems that you have and they tend to be widely available. And the third benefit is the full control and custom features that you get or that they've been able to build rather and they've had the freedom to extend and modify things to suit their needs. So for example, let's go back to the previous point. So for example, they've had to do things like custom authentication using Google or so that people can access the internal Grafana pretty easily. And that's only really possible with open source software. And there's a bunch of other features, Fox to Elasticsearch and Fox to different things that you need to modify for their specific use case. And then the fourth point here is the tighter integration with the application itself. Building an open source software allowed them to do things like link their metrics to internal user IDs so that you can get a more overview of the system and a better overview of what's actually going on so that you can better support and create a better user experience at the end of the day. And then going into the cons, some familiar cons here that we've heard in the previous two examples. The first one is the time and energy to ramp up on the best practices. And here there's a lot of things to learn and sometimes the docs can be lacking or convoluted and there's a lot of upfront time in order to understand the best practices for how do you use different tools? What tools do you use together? Hopefully some of these have been minimized a little bit from you coming to this talk. And then secondly has been the resource overhead and deployment complexity. In particular, they found that Elasticsearch has been very resource intensive and scaling out the logging stack is pretty intense in the words of the engineer that I spoke to. So that's a con that you might want to look out for but they chose Elasticsearch because it's the battle-hardened solution and it's serving them pretty well but it does have that resource intensive downside. All right, so now that you've seen how three different teams have actually deployed open-source software in their production observability systems, I want to show you how you can get started quickly building your own system using a tool called TOB so that you can apply what you've learned today. So before I get into that, I wanna know in the chat if you could just type the same what are you looking to monitor or what you want to monitor or what are you monitoring right now just so I can understand how you could apply this and so you can think about how you can apply what I'm about to show you to your system. Okay, so let's go through the sample open-source metrics monitoring system. So I'm gonna spin up a metrics monitoring system to monitor Kubernetes cluster as well as the nodes and the pods running within it. I'm gonna be using Prometheus to collect metrics about the Kubernetes cluster and the services running in it using Prometheus exporters.  And then lastly, I'm gonna use Grafana as a visualization tool so that I can query things from Prometheus and from timescale DB itself. Prometheus uses a query language called PromQL and timescale you just use a good old SQL. So those are the components you may have seen some of them in the previous examples. I'm gonna show you how you can get up and running with those ones now. So let's actually go over to the terminal and let's actually set this up. So I'm just gonna clear my screen and so the thing that I'm gonna use today, let's go back to GitHub, is something called the observability stack for Kubernetes. So what I'm gonna do is just install it. I have a Kubernetes cluster here. So let me just show you the contents of what's in my cluster. So this is in one of the namespaces called demo. I've got a bunch of microservices here. This is actually a shop. So I've got things like a payment service, checkout service, I wanna monitor this. And I've got another namespace here called monitoring that I've set up before, but there's actually nothing in it. So let's put some resources in monitoring. So what I'm gonna do first of all is going back to this GitHub. I'm just gonna follow the instructions here. I'm gonna install this tool. It comes as a command line interface, just as an easy way, but you can also do the Helm charts by themselves. So I'm gonna install this tool. It's gonna download complete. And then I'm going to install it in the namespace called monitoring. Okay, so while this is installing, let's actually take a look at what's actually going to be installed on this next slide to tell you a little bit about this tool called TOBS, the observability stack for Kubernetes. And really what this, one of the benefits of the open source community and open source tools is the volume of tools that allow you to get started really quickly. And this is one such tool, the observability stack for Kubernetes. And this is what it's composed of. So the main difference between this and the previous slide that I showed you is that you have this additional component called prom scale, which is a component that comprises both the connector between Prometheus and timescale DB, as well as timescale DB itself. So this is one component. And again, we have, and also have an additional component called prom lens, which allows you to build queries and it's basically a tool to allow you to you interact with that using promql and it's very useful for interacting with Prometheus. And let's take a look at how our installation is going. Okay, so this has actually been installed. And so what I'm gonna do is show you some of the benefits that have come from it. I'm just gonna start up my Grafana so that I can show you some of the dashboards that we get for free here. Okay, so let's take a look at what's actually been installed. So as I've showed you the components, I wanna shrink, okay. So this is the components that actually get installed. So I've showed you in the diagram, let's just match some of these things. You have the Prometheus server, you have these node exporters, which I mentioned to you there that actually collect the metrics from Kubernetes pods as well as the Kubernetes itself. We've got timescale DB for long-term storage and analytics. And then we've got promscale, which comprises the connector and timescale DB. And then we have this tool prom lens that I've just mentioned to do things like query building for promql. And then you have Grafana, which is running up here for visualization. What I'm gonna do here is very quickly just get the passwords. So you can find all this information on GitHub. I'm gonna give you a link to it now. So I'm literally gonna do what I usually do is and look at the docs for the commands because I haven't memorized them. And what I wanna do here is just get my password for Grafana so that I can, okay, one more time that I specify the namespace. So this is just my password for Grafana. Obviously you can change it and there's actually commands to show you how to change your password here. But since this is a demo, I'm just gonna keep it the way it is. And so what I'm gonna do here is I've actually port forwarded Grafana already in this terminal window to a local port on my laptop so I can access it. So let's actually take a look at Grafana so I can show you some of the dashboards that you get for free. Okay, so in this case, is it still setting up? Okay. Okay. So in this case, you get a lot of things for free using this TOBS, the tool that I'm showing you. Going back, one of the benefits of this is that you don't actually have to set up each component itself. So let me just go back to the command line here. So all these tools are already set up and talking to each other, their data is being sent from Prometheus and TimescaleDB sent from Prometheus to TimescaleDB and TimescaleDB is already connected to Grafana and Prometheus is already connected to Grafana.  So one of them is this dashboard for custom monitoring which shows me information about the different services that I have running and their states and number of nodes and the status of the pods and whatnot. Obviously I just set this up now and this is something that I don't actually make. It just comes with the tool. It's something that you actually get for free. And then the other one that you get is this Kubernetes hardware monitoring one which shows you information about your Kubernetes cluster. So things like the memory usage, CPU, file system usage, the network and you can actually choose which node of the cluster. I'm running a three node cluster and you can see the difference between the different nodes and then you have a bunch of other dashboards that come for free here. So for example, your CPU usage, the memory usage of your different processes and things like that.  You don't have to start from zero. But going back to the different components here, all of these components are open source and they have a large community around them. So for example, Grafana, Prometheus, what this means is that you get new features all the time as well as all of these components have production users as well. The one con of this tool that I've shown you so far is that it only does metrics. So you can add in your own logging and tracing components as well. And the other thing to note is that these two components, Prom Scale and Timescale DB are actually optional components. So if you don't want to use Timescale DB or Prom Scale, you want to use another long term storage, you can just swap them out by editing the Helm chart. So you can find more information about this on the TOBS GitHub page, which I'm gonna link to just now. But you can see here, this is the thing that I've been looking at. Okay, so that's some of the benefits of the tool that I just showed you, TOBS. One more thing to mention is that with Timescale DB and Prom Scale, you can actually, so let me go back to the Grafana dashboard. One of the other benefits is that in addition to these dashboards that you get for free, you actually have a connection that's already been made between Prometheus and Timescale DB. So now I can create new dashboards in Grafana. It's already being configured for me as these as the data sources. So now I can create dashboards and new queries in PromQL using Prometheus as well as SQL using Timescale DB. Okay, there's a bunch of other benefits that you get from this tool as well, including things like long term storage and Timescale aggregation and down sampling as well as custom retention periods per metric. But those are some more advanced things and you can read more about that on the GitHub page itself. So I'm gonna go back to the presentation and we're gonna wrap things up. So this is the stack that we just created in five minutes using the observability stack with Kubernetes. And you saw some examples of the ways for you to get started quickly as a result of the work that other people have done in the community, in this case, in order to monitor Kubernetes. Okay, so the last thing that I wanna leave you with before we end is some advice from folks who have used production, used open source software in production for their observability systems. And I asked them the question, what advice do you have for those looking to build their own observability systems using open source components? And this is what they said. They gave me seven tips to convey to you. The first tip is to look at existing models and documents or rather documentation. And the key point here is that there's a lot of working stuff already available. A lot of them have decent documentation to help you go through the process of combining the different pieces of the observability puzzle. So for example, there's models out there that already exist to help you collect metrics and logs from the infrastructure. And there's tools like TOBS that I just showed you to help you get started quickly. That's the first. Point number two is some factors to consider when evaluating open source tools. So how do you actually pick tools? So you wanna look at things like how active is the community? Go and check out how many GitHub issues they have, how active their forums are, how active the Slack is, the Discord, the email list, what kinds of things are people talking about? Are they talking about things that are actually relevant to you and developing features that are actually relevant to you? The second thing is who is using the tool? So you wanna look for things like case studies of different companies and different users' experiences with the tools. Maybe such hack news to see like how someone has used a particular tool and what they've said about it, or just Google the different tools and who's using them. And the third point here is how active the project is. So you wanna look at things like cadences of new releases, look at their release notes to see how many things are actually they adding in different release and look at the different issues in GitHub as well. So you wanna pick things based on one, how important they are to your core business, and two, wanna pick things that have a larger community, a lot of people using the tool and as an active project that tends to be a good way to go. So point number three is about knowing what metrics you care about and which dashboards you want to use them. So here there's sometimes a tendency to over dashboard. So you end up with these large dashboards with 30 to 40 panels. And if you're not sure what to graph, you're just gonna graph everything. And what you end up with is a dashboard with more noise and not enough signal, which is opposite of what a dashboard is supposed to do. And the antidote to this is to really define what's important to you first and then go from there. So perhaps you wanna look at just thought with things like response times and durations. And thinking about this beforehand also helps you define what metrics you care about in the long-term. So for example, for long-term storage and aggregation you wanna use in your analysis down the line, thinking about it first and thinking about it in the beginning actually helps you simplify what you actually need to build and what you actually need to visualize at least on the dashboarding side. Then the fourth piece of advice is to get start quickly with a well-maintained Helm chart. Just to see, I've shown you an example of how to do this using TOBS. So what TOBS does is install the Helm chart underneath and you can modify the chart as you go. But this point number four really starting with a maintained Helm chart gives you a head start. Then point number five is to consider your team size and your maintenance costs. So often you have to choose between self-managed open source software or hosted open source software. And generally smaller teams wanna do less maintenance. Just you can consider a hosted open source software for your team if that makes sense to you. But just a point here is that the software is free but the human cost isn't free. And also the human cost of using proprietary software isn't free. It's often just a bit less upfront. So keeping that in mind and considering your maintenance costs and you're seeing what other tools have worked for teams in your industry and how much maintenance they've had to do is important. Point number six is really to understand things like the cardinality, the count and the volume of your different metrics and things that you're measuring. This helps you better understand and better predict your pricing as well as the performance that you're gonna get. And then lastly, perhaps one of the most important points especially since this is an open source conference is to contribute back. And so the best thing you can do here is contribute a PR and really share your story so that others can benefit from it. So for example, the timescale team contributed back a feature to the Postgres Prometheus exporter to help increase the number of queries they can expose. So even small things like that can actually, these things add up and they turn into really massive benefits for the community. So being part of the community contributing back is a final thing to do. So I hope that gives you a lot to think about as you go about the journey of building your own systems. I've also linked to a bunch of resources here to help you get started. So what's next in this journey? I'll put up the slides and resources from this talk on my website. So you can go to this URL and then check out to get the slides here, save them for yourself, go through them later. I also put up a recording of the talk there once it's out. And then we also have the link to the tool that I showed in the demo called TOBS. You can go to this short link here to go to the GitHub page that I was showing. And then lastly, it is a plug for my employers, great company timescale DB. They make Postgres for time series both open source database and a hosted and managed solution as well. So you can check them out at timescale.com. Again, thank you so much for coming to this talk. Really appreciate you. And I hope you learned a lot about open source software and observability solutions. If you wanna get in touch with me, you can do so on Twitter. My handle is on screen there at Aftar S and then also keep in touch on the website Aftar.com where you can find some of the other things that I've done. And if you have any questions, you can email me at Aftar at timescale.com. Thank you so much again for watching.


Good evening! Have you heard the exciting new news? Team Fortress 2 has a new 10th class, at least that's what people on the internet are saying. That's not exactly true though. What is true is that the game's new Summer 2023 update has come, bringing the game to the highest player count it's ever seen, and with that a lot of confused noobs. Now, that's not to shame anybody. I for one am happy to see this game continuing to grow and sprout new players who try to shoot through glass or make use of ambient fire extinguishers. And that's not a joke. But this influx of new players combined with the fact that the new update was not only the biggest the game had seen in seven years, but was also kind of a train wreck is leading to a lot of confusion in the community that I would like to try to help settle in this video. If you're someone who's played the game for a long time, chances are you already have a decent idea of what's going on with this commotion about a new update and a new character. But if you don't, or if you just want to be 100% sure you're up to speed, I'm here to give you a rundown on who this weird buff shirtless guy is, why he was suddenly added after nearly seven years of the game receiving no major updates, and why it's such a big deal that everyone online is freaking out over. I'm also going to talk a little bit about what this update means for the future of TF2, because it's kind of a big deal. So let's start from the top. The headliner. This summer update has added, for the first time in the game's 15 year lifespan, a new playable character, which is kind of crazy and has led to a lot of jokes and talk online about how Valve has finally added a 10th class. However, I find all that to be a bit misleading, since if you boot up TF2 and Q into a game, you're not even going to run into this guy. In my opinion, calling him a class at all is kind of a stretch. He's not a selectable mercenary in regular matches, and in fact, right now the levels he is supposed to appear in have been temporarily removed from the game's map pool because they're broken. But as soon as the multi-billion dollar company figures out how to put the smiley face puzzle back together, those matches will be re-enabled, at which point everyone will be able to queue for a match to play around with this guy, which is the main reason everyone is drooling over this update.  But who is this new character? Well, you probably have already figured this out, but he's a guy named Saxton Hale, and he was introduced as part of a very special new game mode called VSH. Very silly hotdog, I think it stands for. Or maybe versus Saxton Hale, I don't actually know. Anyways, the entire game mode is based around teaming up to try to kill Saxton since he's super strong and powerful. It's sort of like Juggernaut from Halo if you've ever played that. And the reason this game mode and character being added is such a big deal to so many people is because it has a very long history in the game's community. The fact that Valve was willing to officially induct it also potentially marks the start of a more active era in the game's life, which personally makes me very excited. Almost as excited as I am to interrupt this lovely video to tell you that it's being brought to you by none other than Rage Shadow Legends, the fantasy RPG game with over 700 unique champions and counting. With 15 different unique factions, it's safe to say there's something here for everyone. I think the Sacred Order faction is pretty cool. Take on 12 terrifying dungeons, each containing a brutal boss that will take all your sweat and skill to defeat, but will reward highly satisfying loot. Battle against players from all around the world in 3 PvP modes, including an exciting real-time live arena mode. Unlike TF2, Raid is also regularly updated with new champions modes and features for players to enjoy. There's always a new challenge to take on, so you'll never get bored. Raid is available on both mobile phones and on PC, so you can play it anywhere or anyhow you fancy. It also has endless customization, so you can tweak your champion builds, develop your artifacts, and create a team of unbeatable warriors that's completely unique to you. Think about it, 400 million players in over 190 countries can't all be wrong. The new fearsome boss, Accumory the Phantom Shogun is guarding everything you need for accessory ascension, a new feature that allows you to upgrade your gear to even greater heights. And if you somehow manage to miss out on the incredible animated limited series Raid Call of the Arbiter, then you can check out all 10 episodes on the official Raid Shadow Legends YouTube channel now. With all of this exciting stuff and more coming to Raid, if you haven't played yet, then what are you waiting for? Use my link in the description or scan my QR code to get insane bonuses, like an epic champion Drake from the Lizardman faction or other useful things like energy refills, skill tomes, and XP boosters. Once you're in and crushing your enemies, come find me under the name Richter Overtime, and if you're fast enough, you can join my clan. So, visit the description today, and I'll see you on the battlefield. Alright, now that that's settled, let's talk a little bit more about the Man of the Hour. The first appearance of our boy Saxton here was actually all the way back in 2009, 14 whole years ago. The Sniper vs. Spy update released with a promotional page including a list of weapons it added, and then at the bottom there was an advertisement for the weapon company that supplies the Mercs, Manco, and in it, a picture of this fella. With his weird shorts, long legs, crocodile hunter hat, and bare chest, his appearance was striking and very memorable.  Whether that's intentional or not, I'm not really sure to be honest. Saxton also appeared in the update's Gerati comic, loudly proclaiming himself as its inventor, basically implying he was the first guy in the entire world to ever pee in a jar and throw it at someone. So I guess this guy is just filthy. He's also Australian, which is the reason he's so brawlic and huge. If you didn't know, Team Fortress was originally created by Australian people, so in the game's story, everyone from Australia is a super macho man thanks to a metal-called Australian that basically gives them superpowers. That's an actual plot point. Moving on, Saxton proved to be a hit among the game's die-hard fans. So Valve brought him back as a main character in TF2's official comic series, where we learned about his longtime love interest Maggie, who he used to ruthlessly fistfight Panthers with, as well as his arch nemesis Charles Darling, who Hale hated for wanting to keep animals locked up in zoos when they were so clearly meant to be fist-fought by humans like him instead. The final comic entry was never finished, so we don't really know what was going to come of Saxton's arc or where this plot with Maggie and Charles Darling was going to go, but I did a video covering why it never came out and a supposed leak of it not too long ago if you want to know more about that. Anyway, the love people had for Saxton Hale translated to an undying need in the community to see him in action. This fan rendition of Saxton Hale was one of the first custom TF2 character models ever made, and it was immediately put to a very interesting use. See, Saxton being an all-powerful, super-strong, superhero-type character gave a team of community members an idea. What if there was a game mode where a full team of mercenaries had to try to take him down? Like a big boss fight. It was a really interesting and novel concept, and when someone wrote a plug-in to turn it into a reality, it quickly became probably the most popular TF2 community game mode of all time. I personally have a lot of fond memories on old VSH servers, even if I was never very good at the game mode itself. The 2-4 desk was always my favorite map, even if it didn't make a whole lot of sense. Eventually, other characters began being introduced as bosses, like the fan favorite community abomination, the Vagineer, and the horseless headless horseman from the 2009 Halloween update. And with time, so many boss concepts and designs were piled on that VSH got forked into a separate project called Freak Fortress, which is still around and includes a truly demented assortment of bosses and levels. Right, he's left for dead, Bill?  Bill, no! But despite how well-loved and influential Saxton and his game mode were in the game's custom server community, the character himself remained somewhat obscure to the masses for years, since he only ever officially appeared in comics and update pages. However, this would all change with the release of the Jungle Inferno update in 2017. Alongside the Jungle Inferno update came an animated short that gave backstory to its mercenary park level, where we finally saw Saxton in action 8 years after his initial debut, boasting a new model and official voice actor, ripping through yetis and throwing a terrified scout through them as they explode. I can't lie, it was pretty awesome to see this beloved character finally brought to life in an official medium, even if the update itself had a pretty mixed reception. One thing I always found interesting is that the Jungle Inferno short also seemed to tease that Hale would be the announcer on the mercenary park map, which wound up not being the case. I don't know if that was something they ever actually planned, but it does make me think about how interesting it would be to see custom announcer packs in TF2, kinda like what Dota 2 has. Anyways, Jungle Inferno ended up being the last major update TF2 got for a long time. It actually still marks the last time the game received any new balance changes, Valve made maps or new weapons. The community also unfortunately never got their hands on the new official model of Saxton Hale, though someone made a pretty good remake of it that was quickly cycled into most community VSH servers. And with the game falling into disrepair over many many years with no sign of new content coming, it really seemed like this Jungle Inferno short was most likely the end of Saxton's story, which was sad because it really felt like it had only just begun. However, fast forward to 2023, Valve announces a new summer update and asks the community to make content for it. What? The build up to this moment honestly deserves a video of its own, but it turns out that Valve hasn't abandoned TF2 yet, at least not completely. It turns out a team of community members had petitioned Valve to add some new fan made levels that would actually stay in the game's map pool year round. Given how dormant the team behind the game had been for years, few people thought it would amount to anything substantial, but apparently it did. This blog post went out and immediately resulted in the biggest surge of activity the community had seen in years. People began making maps and hats like crazy knowing there was finally a decent opportunity to get non-holiday themed content added into the game. And in the midst of this special opportunity, a small team of modders decided to completely rebuild the classic vs. Saxton hail community game mode from the ground up, using vScript, a scripting language Valve had recently added support for. Once this remake was officially published, it quickly hit the front of the game's workshop. But a lot of people, including myself, thought it was unrealistic that Valve would ever add it. Surely they wouldn't grab a remake of a beloved fan made game mode and abruptly drop it into the game 10 years after its creation with absolutely no hype-cycler prior announcement, right? But lo and behold, 13 years after the character's introduction, Saxton hail and his game mode are officially added to the game through these four v-script powered maps in the summer 2023 update. And on launch, zero of them worked. At the time of me recording this, if you want to play the official new vs. H game mode, you'll have to give it a test run on community servers, because if for some reason it was crashing every time it ran on Valve's official ones, and it's now even been temporarily removed from the matchmaking queue. Here's a quick update. In the two weeks that I spent editing this video, Valve tried to fix vs. H like two or three more times, and it kept just not working. But as of today, it does look like vs. H is officially back and has returned to the matchmaking queue, and I do recommend checking it out. It's actually pretty fun. Personally, I think something as momentous as officially adding a decade old beloved game mode should have definitely been handled a lot more tactfully. As it stands, the official first new game mode added to the game in literally like a decade includes fan voice impressions of the mercenaries in hail, as well as that fan-made remake of his model. That, combined with the fact that the whole mode is currently completely unplayable, is kind of insane. But it's still undeniably cool and exciting that Valve is willing to try stuff like this out. We can only hope that in a future patch they'll bring back the original voice actors for Saxton and the mercenaries to replace these impressions, because even if they're good, there's really no reason Valve should be shipping them when the voice actors are so clearly pining to reprise their roles. The maps themselves are also pretty cool, and some of the most faithful looking ones that the game has seen in a long time. And in general, as much as the role out of Saxton has been botched, it's still a childhood dream come true for this classic fan project to get added to the game officially. However, this whole ordeal does have some interesting implications for the game's future. I'm not really sure if I like the precedent that anyone can create something as giant as a new v-script game mode and get it permanently added to the game's matchmaking queue with little fanfare, especially with some of the incompetence Valve demonstrated with this update's rollout. And the abundance of tweaks this update brought to classic maps like Steel makes me worried as well. This stuff is permanent after all, you're toying with things people have loved for over a decade now. But with the floodgates open, one thing is undeniable. It is an exciting time to be a TF2 fan, the most exciting it's been in over half a decade. If this is your first time experiencing a big TF2 update, I don't want this video to take that joy away from you. But I do think there's plenty to be desired about how this whole thing was handled, and that the story that got us here is kind of fascinating. Anyways, hopefully this video helps catch at least some people up to speed with what's going on with TF2 and why it has a new character in game mode. If not, hopefully it was at least entertaining. Also, before I see a giant wave of sacks and hail memes start popping up, I want to remind everyone that the real coolest character in TF2 is engineer's grandpa Radigan Conniger.  That's all. Also, don't forget to download Raid Shadow Legends using my link or QR code to get cool bonuses in the game. Thanks for watching, subscribe for more, and have a nice day.