Hello everyone. We are super excited to be here at the DevCon conference because it's actually our first one, I think. And we are here to talk about the cloud native project that we both develop and maintain in open source area, and it's called Thanos. So after this, I would like you to know, you know, what why you need metrics, what Prometheus is, what things it solves, what Thanos is, what problems it solves, and last but not the least, how to use Thanos and what are, you know, design decisions that we made while creating this scalable, distributed and durable metric system. But before that, a short introduction. My name is Bartek Plotka. I am engineer working at the Red Hat OpenShift team, OpenShift monitoring team. I love open source Golang and solving problems, especially in Go programming language. I'm part of the Prometheus team, and I also co-author of the Thanos project. And with me, there is Lucas. My name is Lucas, and I'm Marine. I'm also working on the Red Hat monitoring team and everything OpenShift kind of for the last two or three years, I came from the CoroS side and yeah, I work on everything that's monitoring, but also I'm really interested in machine learning and distributed systems and networking also. And our job is focused, you know, on building scalable observability platforms and solutions for OpenShift. However, the major part of our work is also maintaining open source projects, including Prometheus and Thanos on a daily basis. And those projects are focused on enabling monitoring via metrics for infrastructure, service application, for example, microservices, running on Kubernetes. So first, it's important to understand what monitoring is. So let's dive in. I hope I don't need to repeat that, but it's useful to reiterate what are the reasons behind the monitoring, right? And there is a saying that I learned kind of the first day of my work as an SRE in my previous role with production cluster. It goes like this, running a product without any monitoring is like not running the product at all. It's kind of strict, but what I mean here is that if you want to run something that means you probably will be accountable for downtime and bugs. This means that you have to have reliable monitoring in order to prove or at least being kind of aware that something is running or not. There's no point or little point in creating a system that has to run 24 hours per day if you can't prove if it's working, right? So, well, the same message you can read in the SRE book that I really recommend if you want to run things on for production. It is a SRE book written by Google guys and there was like this muscle pyramid for infrastructure need. As you can see, you know, kind of monitoring is the foundation of it, right? It's very essential. It's actually before building the system itself, which is on the top. So it's really important. It actually gives you a couple of things.  It gives you the ability to debug deeper if needed. It, I mean, you can see long term characteristics. You can learn from that, allowing, yeah, you can essentially have data driven decisions, which is amazing. And finally, you can automate things. So you can have, yeah, auto scaling or self healing based on things that are happening with your system. So monitoring is essential in this sense. Now, as you probably know, there are different signals as well. And this is, yeah, typical way we divide the monitoring. So we have tracing, which is any bit of data or metadata that is bound to lifecycle of the request of the transactional request in the system. Logging, which is essentially a discrete event. And finally, metrics. So samples over span of time, composed into logical counter, gouch, histogram, things that you can aggregate. Now, it's not the end. There are other signals as well. So, you know, for example, continuous profiling, something that actually our team is touching as well. As this name suggests, you, it allows you to continuously collect profiles, application profiles. So for example, for Golnack application, you continuously collect, go-piprof profiles, heap dumps, threads, whatever, go-routine dumps. And this is massive help for the developer, because, you know, when something is happening, the application is using too much memory, the CPU spike happens, and you got notified. It's actually too late. Like, things just happen. So it's too late to profile this application. And with this method, you can actually retroactively look on the profiles that happened, yeah, a couple of minutes ago to actually find the root code. So this is amazing as well. And that's not the only one. Maybe, you know, at some point, we'll figure out further kind of other signals that will improve overall monitoring capabilities. And, you know, how those signals come together, that's interesting as well. This is a, like, typical journey of anyone that runs something on production. And let's say, have an incident.  And it's mainly probably triggered by your metric system. Then you go to the Grafana, so static dashboards to locate the problem or, like, what services are actually affected. If you want to dig more, then you can always do ad hoc queries. So, you know, kind of analyze your data that you have based on metric signal as well. And to be honest, in 90% of the situation, that's totally enough. Like, you already know what's going on. You can apply the fix. You can resolve the incident. But sometimes not. Sometimes you need to dig further. So, you have a log aggregation, right? Number four, where you can narrow the issue a bit as well. And if that's not enough, you can move to distributed tracing and check the latencies for certain operations on the way of the request, and that's useful. And then, finally, hopefully, you will find the bug and fix it and deploy it. So, you're good. And during this talk, we'll be focusing on metrics. So, enabling users to do those things like alerting and analyzing your data in the quickest possible manner with help of the prompt use and Thanos. Cool. So, in order to be able to make use of metrics in the first place, we have to instrument our code no matter what metrics we're using. If that's Prometheus. So, if that's Prometheus or if that's any other service, you always have to instrument your code to actually know what's going on. So, you have to say, what are the things that I actually care about inside of my code? You know, inside of my service, what is important to me? What are the things that I actually care about for defining my SLI or, like, for my SLOs? Like, what are the things that actually matter to my customer and to us as a team to help us debug? So, really quickly, let's go through an example application. So, let's say that we define some service that we're writing in Golang that's a Golang server that exposes just one single API endpoint at slash API. And it does the most minimal amount of work possible. So, 50% of the time, it'll return one status code that's status okay. And the rest of the time, it'll report status teapot, which is also very important. So, how would we go and instrument this application? The first thing that we would do is we'd say, okay, let's define some metric that actually matters to us. So, in this case, we're defining something called HTTP request total. And we can keep track of, you know, what are the total number of requests that we're serving for any single HTTP method and then also what's the response code that we gave for that endpoint. So, in this case, we should expect to see, you know, some time series that ends up with labels that are code and method and those values could be, you know, 200, 418, and the methods could be getPost, et cetera, et cetera, right? So, now that we've defined a metric, we actually want to instrument our code to use this metric. This is typically pretty easy as well. The only thing we have to do is, first, you know, define some registry and expose this new metric that we defined so that it can be scraped by our metric system. And then, whenever we get a request, we increment our counter and say, okay, when I get a request of 200, I'll increment with status 200 and I'll also add a label that is the method on it. And then, once our metric system is actually scraping this application, we should expect to see some payload that looks like this, which is time series for every label and value pair, so code and methods. So, here we have code 200, code 418, method gets, and we see, for example, we got fit through request, 50 get requests that ended in 200s and 53 requests that ended in 418. So, that's just an example. And this is how easy it is to do with Prometheus, for example. When you have a more built-out application, you can get a lot of different metrics, pretty much for free from your client library as well.  But before we go any deeper into instrumentation, let's talk a little bit more about Prometheus in the first place. So like, who here has actually used Prometheus? Oh, okay, cool. All of you do. That's awesome. So, Prometheus, I'm glad that everyone's using it. We love it as well. One of the things that we love about it is the fact that it's super simple. So, let's cover some of the simple concepts that make Prometheus really good in the first place. Prometheus, first of all, is a super simple application, really, really stripped down, right? Where like, it's just one single binary that you can run on any machine or on your cluster. And the only thing you have to do is point it at your application and scrape metrics, right? Now, the thing that's great about Prometheus is that it's super simple and then it's also really efficient to use, right? So, you can either query Prometheus directly using Prometheus' own query language as a promQL, or you can also define dashboards that are like really complex and can have lots of different useful queries like built into the dashboards. Or you can do things like writing alerting rules and recording rules. So, for example, you can get notified when my service is serving too many 500 requests over the course of the last 10 minutes, I can get paged of your pager duty or maybe an SMS on my phone. And one of the things that's also particularly cool about Prometheus is the fact that it scales pretty well in itself. It can handle tens of millions of time series just on one single application if you give it enough resources. And then we're gonna be talking a little bit more about how this scales going further on. One thing to note about Prometheus is that it's a pull-based model and this differentiates it from other monitoring systems because it makes it also easier to configure. So this means that the only thing I need to get my monitoring system up and running is run my binary and point it at the applications that are actually exposing metrics. My client libraries for this reason can also be really simple. My client libraries don't have to do anything like retries, backoffs, buffering of different metrics. When the client libraries end up taking all this into account, the complexity of the client library can end up being greater than the complexity of the application I'm actually monitoring. And Prometheus, on the other hand, has really simple and stable client libraries that make it super easy to instrument your application. And it also means that Prometheus server itself can kind of regulate how much data it's ingesting and it doesn't have to do anything like throttling or rate limiting clients that are maybe spamming the Prometheus server. And this way, you would overload your metric system. So today, let's see, Prometheus has been an open source project for about six years, and it's super well adopted. And one of the things that's beautiful about this is that there's tons and tons and tons of different open source projects that already exposed Prometheus metrics. But it also means that if your open source project, for example, Memcast or something, doesn't expose Prometheus metrics, there's probably some ecosystem project that's an exporter that can wrap that component and does expose metrics for the application that you care about. And it's really easy to write exporters as well. OK, so we talked about why Prometheus is really great, why it's super cool, it's super easy to run on just one machine. But there's also a lot of times when running just one single Prometheus on one node isn't enough, and we have to scale it out. So let's cover a little bit, why would we even need multiple Prometheus instances? We just said that you can handle 10 million time series in one single Prometheus server. It's super easy to configure. So why is this ever even an issue? And let's go back to this example where we have just one single Prometheus that's scraping a bunch of applications. And we said that this scales pretty well. We can ingest a ton of different time series. But imagine that I'm actually running real life We work in distributed systems, we work in OpenShift every single day, and things break in distributed systems. Really unexpected things happen. Let's say imagine, for example, that I'm running on an OpenShift cluster or some Kubernetes cluster. And the node that's running Prometheus goes down or dies for any reason. Or maybe my Prometheus server also dies for X reason. And it can't scrape my application. And I also can't query Prometheus. And maybe right when this node is down, I'm having an incident. And this is exactly when I need to be querying my Prometheus server to get metrics. Or maybe something that's much more routine, like not a failure or disaster scenario. Maybe I actually just want to do a rolling upgrade of my Prometheus. And I want to switch from one Prometheus server to another one. And I don't want to have any downtime on my Prometheus server. So for this reason, it can be super useful to actually be running two Prometheus servers side by side. And we call this running highly available pairs of Prometheus where every Prometheus server scrapes every single application that I'm also running. This way, if one Prometheus server is down, the other one is likely still up. I can also do a rolling upgrade. So I can take one down, upgrade it, take the other one down, and upgrade it. And I'll always have high availability for both queries and metrics ingestion. So this is just one example for why we might want to run multiple Prometheus. Yep. So we will cover that, I think. Yeah, we'll cover that. But I think the member here is that in this highly available scenario, both Prometheus are scraping all of the different applications. And they're also evaluating alerting rules. So cool. But that's not the only reason why you essentially have a need to have more than one Prometheus instance. So let's cover other use cases. So let's see our example that we provided before. And we spoke a certain amount of metrics on each of those applications. So let's say we have four applications. Each of those has some number of metrics that it exposes. Within Prometheus, we hold almost 2K of metrics. And this particular query tells you that information, like how many series it stores for fresh metrics in something called head block, which is essentially in memory. And this is kind of number of those series. It's called cardinality. And this is super important because cardinality tells you how large the Prometheus instance will be. How much resources it will use. This is the main thing that affects your resource consumption. So let's assume we have this situation. This is fine. For 2K of series, my Prometheus will run super smoothly, like probably almost no resources will be used. And maybe one gigabyte disk is totally enough for that. But in real world, we have more applications. And with this equation, let's say we have one node and just 30 apps per node, like maybe Kubernetes node. And each of those has roughly 300 metrics exposed. Suddenly, we have 9K of series in our Prometheus instance. And that's totally fine as well. That's a low amount of metrics. Probably you need to have one gigabyte of memory reserved for this Prometheus. But that's totally fine as well. However, who runs the huge cluster with just one node? That's unrealistic, right? So well, let's scale it up. Now, things get maybe more tricky. We suddenly have like 3,000 applications to collect metrics from. So at the end, Prometheus collects almost 1 million of series. And that's still manageable. That's totally fine. You need to have more resources for that, but maybe 10s of gigabytes of memory. But that will do just nice. But then we scale more. We have suddenly 500 number of nodes, which is still doable in these days for Kubernetes. So it means like 15,000 applications running. And then in our equation, it means like almost 5 million of series. Now, this gets kind of tricky. You have almost hundreds of gigabytes memory right now. The operational aspects are getting tricky. You suddenly need to have more buffy VM or hardware to support this instance. So it gets more pricey and scary, right? And also the disk, you need to have large SSDs to cover those number of series, essentially. So OK, but let's scale up. Maybe we have much, much more nodes. Or maybe we can change other number in this equation. Maybe more pods per application. Maybe a couple of applications can have like 10,000 metrics exposed. So this goes out of the limits. Because we set, as a Prometheus team, we are kind of setting 10 millions as a threshold for something being kind of easy to operate. Because at this point, the startup of the Prometheus will be slower, because we need to replace certain things. And yeah, crash recovery and all of that stuff are super, super tricky. So we really recommend to have like 1 million of series. That's the best. But suddenly, well, we have those apps, and we want to collect those metrics. So how to, what's the solution to reduce the number of series per Prometheus and be able to run maybe smaller VMs, right? So the solution is called sharding. And this is showing like the functional sharding, which means that you separate your application per function, or maybe per namespace, or maybe per project, or maybe per customer. And yeah, you have separate Prometheus scraping those metrics. And being weak are able to limit the size of each Prometheus majorly. And then there is kind of connected. But if you don't have separate functions, or you don't care about like, OK, you don't have information what function is for what application, you can use something like consistent hashing, which you just tell Prometheus how many Prometheus's there are in your cluster, and it will automatically hash and allocate each application to one Prometheus in a way that it will be equally evenly distributed. So that's a nice feature as well, and you have that out of the box. So this is another way of sharding, right? But what it means? It means essentially that, again, we have another use case for running more than one Prometheus server, and that's really realistic these days. And there's one more. Well, we run more than one cluster, right? These days, OpenShift or any other Kubernetes can be started in minutes. So it's really realistic that companies will have many, many Kubernetes clusters or any other clusters running around the world, maybe for geolocation reasons, maybe for HCI high availability reasons. So this means you need to have more than one Prometheus. Well, you could say, OK, but why can't I have one Prometheus which collects the data from many clusters? This is not recommended, because your Prometheus, because of pool model, should be in the same failure domain, in the same network as your services that it's monitoring. So remember about that. This is very, very important, and this allows Prometheus to be very reliable, and you can trust it more, because we limit the unknowns that are between services that are being monitoring and Prometheus. Which means, anyway, again, multiple Prometheus instances, at some point, you can escape the scenario. OK, but Prometheus by design works as a distributed, does not work like a distributed systems. And as Prometheus team, we kind of designed this being like this way, to be a single server application, super simple to run, super simple to maintain, and understand. And a simple one binary time series database. But to be honest, it makes this project actually super focused on matter, and I love it. That's why I love to contribute and maintain to Prometheus. That's why it's super popular, and to one of the best projects with one of the best and kind of friendly community. So this is because it prefers simplicity versus over-engineering. And yeah, it allows integrations. However, it is focused as alone into things that matters in these projects. However, this means that not everything is solved out of the box. And we'll be talking about exactly this point. So yeah, let's run through the challenges when having more than one Prometheus. First of all, let's say we want to find a number of metrics we store in each of those Prometheus's in the current moment. So we do this query. And normally with query this, we will have the number of series in the head, and then we will sum all of those by cluster. Because we want to know by cluster what's the number behind each cluster, how many series we have in each of those clusters. However, if we do that, we need to, well, to do so, we need to kind of query each of those Prometheus instances separately. Because this database is not replicated in any form. They're not connected in any form. So it's kind of tricky to calculate that. We would need to manually do so, which is not really convenient, right? So there is no global view. This is the first challenge that we want to mention, the global view. So global aggregation on top of multiple Prometheus's instances. Yeah. And then it actually gets even a little bit trickier when we start talking about highly available Prometheus pairs. So in the previous example, we had functionally shorted Prometheus i in one cluster, this US1 Kubernetes cluster. And now here we have functionally shorted and highly available Prometheus in our Kubernetes cluster. And it gets even trickier because you see how one of the Prometheus i says, yeah, we have 200,000 time series and the other one has zero. What's going on? How do we reconcile these two differences? Why does one Prometheus have so many time series and the other one has none? And actually, why did this happen in the first place? And the answer is that there's actually some valid reasons why two Prometheus i in one cluster will have different time series. For example, we remember the fact that Prometheus scrapes on interval, right? So periodically scrapes my applications, like maybe 15 seconds or 30 seconds. And it's totally valid that maybe in the time span that I was scraping, something happened in my application. Maybe the application got restarted. So one Prometheus will show one set of numbers for time series. The other one will show a different set of numbers. One might show certain metrics that existed when it was scraped the first time. The other one won't show it. Or maybe my Prometheus got restarted and it just doesn't have anything in a CSDB. I lost something on disk for any reason. And it's expected that our highly available Prometheus pairs are going to end up with different numbers of time series. It's simply a fact of the matter. But how do we reconcile it? When I'm querying these different Prometheus i that are both on the same cluster for the same time series, I can end up with one Prometheus that shows, yeah, I have a bunch of data and I'm missing it in the first time span. And then the other Prometheus is saying, yeah, I have that data. But I'm missing it in the later time span. So again, it makes sense that maybe one Prometheus was down for some reason. The network connection between the one node and that application was buggy for x reason during that exact scrape interval. But the question is, how can we reconcile these two graphs? Like, it would be great if we could have one graph complement the graph from the other Prometheus. So the question is, how can we make our querying of Prometheus highly available? Not just the scraping, but also the querying. Another question that comes to mind when we're running Prometheus at scale is retention of our metrics. So normally, when we run Prometheus, we are querying metrics for the last 36 hours. These are the metrics that we typically care the most about. Because when I'm diagnosing my application, what I care about is, what's happening in my application right now? What is the state of this time series exactly right now? Or maybe I'm debugging an incident from yesterday. And it's like, OK, what was happening yesterday at 5 AM when we got D-dust? Or maybe it's what happened 10 minutes ago so that I can evaluate an alerting rule and I get paged or something. So my Prometheus normally is looking at really short time ranges of data. For that reason, typically Prometheus has a time retention for its database of 15 days. And that means that after 15 days, the blocks in this Prometheus time series database just fall out of the database and we eliminate them. This means that it's difficult to look at data from last year. Even though there are some cases where maybe it's really useful to analyze data from last year, because I want to look at long-term trends maybe for my application, maybe for I want to analyze my SLIs over a long period of time, or any other reason. But if I'm keeping all of this data on disk, I suddenly have to increase the amount of disk space that I have available for my Prometheus. And disk space is really the only external dependency of Prometheus. We want to have some persistent disk. But if I'm keeping a years' worth of data, my dependency goes from being disk like this to disk like this. And the problem is that this isn't totally scalable. Like when we scale up the disk sizes, we also have to think about scaling up the operations that we do on these disks. That means backups, resizes, restores of the backups. And this makes it really difficult to manage Prometheus service with a long retention time. The other thing that happens is that when I query for a years' worth of data that's scraped every 15 seconds from Prometheus, that's millions and millions and millions of data points. So imagine that I query an application every 15 seconds, and that means four times a minute, 60 times an hour, 24 hours a day, 365 days a year. That's 30 million plus time zeroes if I'm querying for some metric over an entire year. And number one, this takes a lot of processing power to compute, but also it takes a ton of network bandwidth just to send all this data from my Prometheus server to the client that's querying it. And what's more is that 30 million data points is way more than I can even render on my screen. If I had an 8K monitor, I would need 4,000 monitors side by side to render this in full resolution, which I don't have at home. And so what we really want is to have lower resolution data for these really long time ranges. So for example, can anyone point out the difference between the graph on the right in full resolution and the graph on this side in low resolution? And the answer is no, because they look exactly the same. And when we have really long time ranges of data and we do down sampling correctly, we can still get really valuable insight into this data and we can preserve the accuracy even while we're losing some resolution. And we can still make the valuable inferences, but we save a ton of processing power and it also means that we can go from responding to a query in a minute's time to responding to a query for years with the data in under a second. So we want to have some kind of long term retention and down sampling in Prometheus, but this doesn't exist right now. So what have we covered so far about Prometheus? Number one, monitoring is super important, do monitoring. Number two, it's really easy to do with Prometheus and you should get started with that. We often have to run more than one Prometheus for different reasons, maybe high availability or just because to manage the load, but doing so is pretty difficult and there's a lot of challenges surrounding that. And also we want to be able to keep maybe long term data, but this is also difficult in Prometheus. Yep, and all those challenges we mentioned with running Prometheus at scale, like global view, lack of global view and handling of high availability, really hard story behind enabling long term storage of the Prometheus, those did not come out of nowhere. It was a feedback for many, many users of Prometheus and that wanted metric in this form. And in the same time in my previous place I had kind of improbable startup at London, we suffered those problems as well. So that's why we teamed with Fiband Raynards who was in the CuraS back then, also Prometheus maintainer and we then together kind of created a TANOS project which was essentially fully open source from start because we knew that everyone wanted to solve the same problem so we want to collaborate together on it. And the project has two years now. We joined Cloud Native Computing Foundation, I think in summer, and we were quite lucky to have really, really amazing and friendly community that helps us to maintain this project. And we kind of have that from kind of start so we were super, super lucky. And through those years we managed to build quite diverse maintainers team as well. Most of them from different companies so this is pretty great as well. And there are many users who are actually using us right now, the project is free, open source. And more users means more chances for awesome contributions and feedback so it's pretty great. And I think the key value of TANOS on why TANOS is kind of popular is that with single, it has single mission to scale Prometheus, nothing more. So TANOS is also aimed and focused on simplicity and low maintenance burden like Prometheus is. So this is additional kind of things that we are focusing on. So why is that? Well, we try also to reuse as much as possible from Prometheus code. So we literally import Prometheus code for query, for querying, for kind of TSDB for us, the storage format, and many, many other like alerting. So once someone fixes something on Prometheus site, it immediately fixes things on TANOS which is great. So we are all together in something called Prometheus ecosystem, we call it. So Prometheus and projects around that. And we also team up with the Prometheus. So it's great, it's open source. But let's focus on like how enable TANOS in our case where we have like multiple Prometheus instances, like how to enable that. And we actually have some kind of seven steps and I would say pretty simple one. So let's go. How to essentially transform the existing Prometheus setup to solve global view, HA and long term storage. So let's say we have like two clusters and three Prometheus is overall. And yeah, step number one is to add a sidecar. So TANOS sidecar is a small Go binary that has multiple features, but we cover two of those here. First of all, it translates the internal Prometheus remote read API that exposes some metrics that we selected, select. And it transforms that to TANOS GRPC API called Store API. And also it has some other features, like for example, it reloads the configuration of Prometheus or rules or alerts automatically from, for example, Kubernetes config map or stuff like that. So to have TANOS enabled, we need to have the sidecar added to each Prometheus instance and in Kubernetes it's kind of easy because we have pods. Now, step number two is to add a stateless TANOS querier component which is essentially, it essentially performs a PROMQL. So native Prometheus square language evaluation on the global level instead of just in Prometheus local level, right? And it connects to the Store API via GRPC and it actually connects to any Store API. It doesn't know if it's sidecar or something else, it just, it's an API. So, and then it can evaluate the query and by fetching the data from the nodes, leafs, let's say. So actually it will ask Prometheus directly and you know the same kind of, the querier exposes the same HTTP API as Prometheus does, so all the Grafana dashboards all the toolings works because they think it's just Prometheus, but it's something more, right? And it also allows us to answer our previous query, right? Like how many series are per cluster and because it has access to the data from both and it can aggregate in this particular example sum, all the things from each leaf and provide us with the correct result. Cool, now do you remember the kind of the problem with the HA on multiple replicas and how to provide the output that kind of makes sense given two replicas having some gaps here and there? Well, querier has built in the duplication layer and thanks of extra label that you can configure in each of those Prometheus replicas, it is able to tell that this data should be the same and there is like a certain penalty algorithm that detects if there is a gap, then feel this missing data from another replica if there is any. So user at the end transparently, I mean the process is transparent to users, so user doesn't know and will not even notice that there are multiple replicas running, it will see like the one signal which is exactly what we need because now we can for example, upgrade Prometheus version, and no one will notice finally. So this is how we solved high availability as well and global view. But let's say you want long term storage retention, right? Right, so what then? In order to solve long term storage, we introduce what is essentially the only dependency, external dependency of Thanos and this is object storage. So what we configure pretty much is you essentially give the sidecars in Thanos credentials to your object storage provider and right now, Thanos supports a ton of different object storage like S3 on AWS, Azure, Google, and also some in China like 10 cent in coming Ali Cloud and essentially what happens is that when Prometheus outputs a block in its time series database format, the sidecord takes this block of data and it uploads it up to object storage. Now, one thing that's important to note here is that this is not some kind of streaming API or something like this, right? We're essentially doing lazy upload of all of our data. After two hours when Prometheus actually produces one complete block of data and although this is not real time, it actually provides other really beneficial characteristics and the fact is that we don't need to have a super low latency connection between our object storage and our Prometheus. We can have them in separate regions or totally different places or whatever and we can just upload this chunk of data when it's ready. And now in order to actually make use of this data we introduce a new Thanos component called the Thanos Store Gateway. So the Thanos Store Gateway implements the same GRPC API as a sidecar and the querier and the thing that's important about the Store Gateway is that it makes instead of reading the data directly from the Prometheus API it reads the data that it's serving queries for it reads the data from object storage. So now whenever I query for data that's very recent my Thanos querier will probably get data from my sidecars and directly from the Prometheus that's running on a cluster and when it's looking for data that's more long term for maybe my years worth of data it'll get that data from object storage. But there's one caveat to this and that's that remember that we were saying that when I look at data from a year ago I might be getting 30 million time series and we need some way to solve that. So in order to solve this issue we introduce one more component that's called the Thanos Compactor and this performs a couple really important functions. Number one it does this down sampling that we were talking about. So I can take really high resolution data that's sampled every 15 seconds and down sample it to maybe every minute or even lower or something like this like an hour and I can essentially when my querier is looking for hey give me this time series from a year ago I can give it much fewer samples and make the queries a lot faster. But the other thing that it does that's very important is that it combines multiple Prometheus TSDB blocks into fewer larger TSDB blocks. So instead of having thousands of TSDB blocks that are only two hours long I'll have fewer TSDB blocks that are two weeks long. And this is a very important optimization for speeding up our queries because rather than having tons of different indices in every single block and a lot of different metadata associated with that we can essentially reduce the size of these indices and combine them all in larger blocks and make this way our queries a lot faster. One last component that we're gonna introduce in this talk is the Thanos Ruler. And the purpose of this component is to provide a global view of our alerts and our recording rules. So normally we always advise and Prometheus that you wanna keep your alerting and your recording rules local to your Prometheus cluster. So that means you want to evaluate the rules on your Prometheus that's running in your cluster. But there are some important places and some cases where you would actually do want to have a global view of your alerts. And one good example of this is maybe you have some alerts that aggregate over multiple clusters. Another one is maybe you want to have some alerts that take into account years worth of data or very long term metrics. In that case you want to be able to draw. You want to be able to evaluate these queries with data that's in object storage. Or maybe you want to do some kind of meta monitoring and you want to know like is this query, is this cluster or is this Prometheus up or down or not, things like this. So this is kind of an optional component. But one important thing to note is that it actually serves these recording rules via the same store API that both the sidecar and the store gateway also implement in the query year. So there was something that was common in all of these architectures and it wasn't just Thanos. One thing that was common in all these architectures that's very important is the store API. And we kept saying this word like what is the store API? The store API is a GRPC API that isn't defined inside the Thanos project. And this is really, in my opinion, is one of the key innovations and things that the Thanos community actually brings forward to the ecosystem. It's implemented in all of the Thanos components that allow evaluation of time series or create time series like the sidecar, the ruler, the store gateway, et cetera. But it's also implemented by some third party projects that want to be able to maybe act as a bridge between some different format like OpenTSDB and Thanos. So now anybody that implements the store API can integrate with Thanos community. This is what the store API looks like. Anybody that implements this can integrate with Thanos. I think that we don't have that much time to talk about this. So let's get to the more interesting things. Sure. One more thing that you can add as a bonus I would say and something that actually we are still working so this kind of still working progress is caching. So all of those things that you query and you maybe run some Grafana with some dashboards and imagine that 20 users are, and there was incident. So many people wants to see exactly what's happening and hitting the same dashboards which has probably 20 different graphs. So 20 different API calls to your metric system. So caching in this case and all those users are doing exactly the same API. So caching the response for those particular queries are super, super nice to have. And this is where Open Source is amazing because we have friendly project called Cortex which is kind of similar. And then we are kind of friends with them, with maintainers from that project as well and they created something like Cortex query frontend. And this allows, this actually works against Prometheus and Cortex and also against Thanos. It uses some shared caching for example memcached or just in memory. And you can put that on top of Thanos for example to cache all the responses that are evaluated for the query and then those consequent kind of, sequent, well next query is that he's exactly the same data. We'll respond very, very fast without kind of resource consumption. So this is something that we're still improving. Like for example, it doesn't support the sampling yet but probably we will make sure like we can collaborate with the team to make something that works nicely for Thanos but caching is essential at some point. It allows you to fasten things up drastically. And one more thing, one more bonus is streaming. So at some point we were talking about pool model based so for example Prometheus is pulling, collecting metrics from application. A query is pulling data from the leaves but at some point you need to have some kind of push method. Like you can't escape it in certain cases. For example, your cluster is very isolated and you don't allow ingress traffic, you only allow egress traffic, right? So at this point maybe you want to push the data to some centralized place. So for this kind of rare let's say cases we created another component receiver which is essentially another story API. Well it exposes story API as well and it allows to integrate with remote read in the protocol that Prometheus uses which essentially streams all the samples which requires again load latency maybe network and there is some delay in metrics visible in your centralized clusters but you know this is a trade-off that you can decide on why deploying Thanos. And the key part about all of this is that all those components you can shape this deployment as you want. You can have just courier and sidecars and no object storage at all. Maybe you want to have certain cluster having no object storage but actually kind of data from this cluster will have long-term storage retention so you have the Stargate way for example or maybe you have just one special cluster that requires kind of streaming so pushing the metrics from the cluster instead of pooling and then you have some receiver somewhere as well just for that purpose. So you can mix things together. You can just use one or three together so it's very, very flexible and all thanks to the same API as well in some form. Cool, so kind of instead of live demo we thought that it would be nice to instead having some nice interactive tutorial to be for you to try in offline mode as well so we don't need to like prepare something that will go to garbage essentially and actually there is a nice Catagoda platform that allows that if you go there there is actually only one scenario yet but we are working on other stuff as well and if you click on that scenario you can essentially play with Thanos because this Catagoda platform allows you to run some communities cluster or Docker from your browser and access everything access from your browser you don't need any hardware it will run on some other kind of cloud provider and there's tutorial that explains step by step what you should do and how you can play with it so we really recommend to go for that and we are trying to make it up to date so it should get latest version or something so yeah, check it out and yeah, let's sum up so overall let's sum up what we learned today we explained what monitoring is that it's very essential so please monitor your application that's for sure and metrics helps a lot that's our opinion we explained the cases where you have why you need more than one Prometheus we explained why that brings some challenges and finally we presented Thanos and hopefully explained how in few steps you can gradually transform your setup or Prometheus setup into Thanos and there's a very important note like let's say you don't have monitoring yet and you want to instrument your application don't start with deploying Thanos that's distributed system which has some complexity just start with Prometheus learn first how to use querying how to use alerting how to operate it how to learn Prometheus well first and then once you hit the scale issues and you are sure okay that's not enough then you can gradually install without you know it's not like you are replacing this Prometheus no, you are building on top of stuff adding more components and gradually installing Thanos so it's very very flexible and that was our design behind Thanos as well cool, so that's it from us you have some links we are also hiring for a monitoring team at Red Hat and yeah, thank you for listening to us I think we have time for questions so yeah, we support GCS, S3, AWS, Azure Tencent Tencent, Ali Cloud we also support Swift Seth, yeah but like Swift, OpenStack, Swift anything that exposes S3 API so like Minio, stuff like this yeah, exactly so any S3 compatible buckets as well so yeah but if you are missing any just put GitHub issue on that yeah, it might be more people interested in that cool, questions yeah, good point oh, I read the question so essentially you are asking can we provide any user stories or where you can find the user stories behind the deployment architectures different deployment architectures because you have many components and you can put Store Gateway in different places together with centralized place and maybe in the client cluster so the question was what's the recommendation I guess and where you can find and read about those so I would recommend the blog posts and like I think if you go to our Thanos page or like GitHub page there are some links to the blog posts which essentially are from different companies and how they use and architect the deployment and particular question with the Store Gateway yes, I think it's really reasonable to have a Store Gateway in some centralized place in the same region as bucket because the latency might matter here, so that's our recommendation and we run Thanos in many places as well in production we are on call as well I would suggest this yeah, so the question is is there any roadmap to integrate OpenShift and Thanos or put Thanos on OpenShift and the answer is yes, actually coming in the next release of OpenShift we're having user workload monitoring this has Thanos on the cluster this is providing functionally sharded Promethii on the cluster one for cluster monitoring and then another Promethii for user workload monitoring and then you can still have a global view using a Thanos Quarrier and this means that now you can actually configure say I have a bunch of I have a bunch of applications that I want to monitor OpenShift can you please provide me with a Promethiis to monitor my applications using the Promethiis operator and all the nice like service monitor CRDs and stuff like this so you can get that for free and then still use Thanos to have one single place to query for everything there's as far as I know there's no roadmap to deliver Thanos itself like hey like OLM or something like this like some kind of you know operator installed Thanos or anything like this but basic sidecar and like querying functionality this is coming in a soon release of OpenShift and actually you have Promethiis operator right as well which deploys out of the box Thanos sidecar and ruler so it's coming in soon so yeah super easy to deploy yeah so that's integrated we have to integrate in the cluster monitoring operator and the user work the monitoring this is integrated in the cluster monitoring operator but anything any more advanced features of Thanos this is not yet on the roadmap however you can integrate that on your own as well yeah Alex yeah so I think the question is we are talking about number of series and how they affect reasons consumption the number of samples the retention that we are having the data for and that's a very valid question and the answer is that samples and like this that because there are two dimensions like the time and the number of series cardinality right and the problem with cardinality is that this dimension is not compressible however the samples are very compressible so there are really really nice methods to compress it yes we chunk it we have like very special gorilla kind of compression format and it's actually not a major addition to the resources if you add more samples or less right at some point of course it matters if you store you know like one day retention or one year retention however yeah but in fact most of the time when you add more samples like higher higher interval scraping these samples become more compressible because they are more similar and like because of the encoding of the samples it actually compresses very well and the thing that really becomes a limitation oftentimes or vast majority of the times in Prometheus is not number of samples but it's number of time series absolutely it's true that it has become a limiting factor when we were talking about sending millions of samples over the wire for a long-term query like this can become a limitation for sure cool all right I think we're done please find us later we'll have you talk thanks


Hey everybody Forest here ripping 2022 gilded and finest baseball hobby for a four box mixer number seven random team style So again, this is one gilded and three finest check the results on screen anytime at leighton sports cards dot com Let's go ahead and get started y'all Yep, this is number seven Let's start with the three finest finished with gilded I'm gonna pull gilded superfactor or any superfactor really Who's it gonna be vaughn the hardened should be hardened Who do you think should be Jake? Who's your replace kairi and the eastern all star It's gotta be brunson or hardened. It can't be anyone else, right? I'm gonna go gary hairs Well-deserved Uh, let me what's do you have an order number nap time? Ricky refractor where once you can try us pirates Or whatever the first and last name whatever the first and last initial is Yeah, it could be tray I've put in the wrong initial maybe Oh that I put drew I think I put drew a instead of dave a That's exactly what I did because there's no other first name d Everything else is correct Sorry about that dave one sec. Yeah, that's exactly what I did I'm just gonna repost the results you can change it by hand there if you want to I'm just gonna redo it One sec dave Everything else is good. All right.  That's Julio Rodriguez. That's what's up, man 64 of 300 You go ahead and make this there should be some check right near your leg down there Yeah, usually it keeps in 35s over there Get better sleep for this one get that mag for you mariners Good stuff the 300 rookie j-rod base j-rod as well. What's up, James? Jeremy Pena rookie Salvador Perez bobby with jr. Refractor devbers You just you just bench them and then Huh get green speckle Byron buckson 42 of 125 for the twins got a gold refractor rookie Auto Mets Tyler McGill 27 to 50 nice hit there for the Mets Thomas Miranda Fractor charlie blackman That's his name chuck nasty Aqua shimmer kebrian haze pirates 63 of 175 That's the first rule j.  Yeah, they're just out there getting paychecks. That's all Wonder how much they get paid for this It's something i just don't know what it is It's just it's a full stadium though Refractor trevor rogers. It was a good idea to move to vegas get people to actually want to go Instead of Orlando or even Miami rookie green lava george kirby mariners mariners having a good break here 70 of 99 Hey, Brennan. What's up, man? That's your fondix. I want to ball this the fondix Suzuki rookie refractor auto marcos de blonde for the orioles jakes bag refractor marcus semian Abrams are of excellence water franco.  I could it looks really good. It's very good so far I've never played the game the game the game. I'm thinking about just buying Thinking about just buying a playstation and just playing the first game to play the game refractor finando tuttees No, I know as soon as I saw the first episode. I'm like, yeah, I gotta play the game Aqua blue vapor yordan alvarez astros number 11 to 250. I'm like, man, I'm just gonna buy a new playstation spend just money that I shouldn't Who needs a serious exit into playstation honestly D backs rookie refractor auto drew ellis Filing taxes. Yeah. I mean, hey, it's been good so far. It got through some artifacts, which is always awesome refractor andrew mccutchin as a brewer Episode three was Probably the best of so far For like acting and story rookie rosegold a's kevin smith 1975 How much is buying a playstation gonna help the economy? Yeah, sony's doing sony's hurt and any they need some more money Sony and game stuff. They're the ones that are struggling Sorry guys land tax me about about hockey cards right now.   He's probably wearing a rose rain of shers. Do you remember that hoodie right there?  Is it evans? Is it mclanahan refractor mookie bets? aqua refractor trevor story redsox Number 40 of 199 Dude the game looks so good. Yeah, I'm for sure getting a brennan. I have the deluxe edition I just hope it ships on time because I bought it um from target rookie refractor auto tether gilbert for the debacks But then again, I'm pretty busy on tuesday anyway, so even if I have it by tuesday, I won't really be able to play it much but Something my next day off is saturday after that so saturday I'm just gonna put way too many hours into it all of the all of the hours that I have available We're going in the hogwarts on saturday jas tiz i'm refractor Bondard, do you guys purple refractor rangers marcus semian number 80 of 250 I'll for sure have to teach my wife how to play it and then it'll be her xbox Yeah, plus the game looks legitimately really good. All right gilded collection Show us a huge hit here. Come on We should get this uh trampoline and like big like landing pad set up for the shop I'm gonna have an extra hit in here. I believe there's one more chrome card than normal It's high cop gold edge 96 and 99 tigers Ashrow's rookie gold edge hosay siri That's a raised legend right there Numbered 44 of 99 astros Nice rookie. Oh, here's an ice rookie auto mckenzie gore go padres 186 of 199 got marlins joey wendell auto Not the raised legend 13 of 50 on the joey wendell auto for the marlins Rookie mini diamond kyle muller braves 41 of 50 And come on. I'll see a big frame hit here. Come on legend time big legends It's a dave winfield as a padre 55 of 80 framed out of dave winfield padres There we go Give me just a second. We'll do a recap everybody. All right. Here we go with uh the chrome autos here We had tyler gilbert Brandella cruise vedal brujan drew ellis marcos de plan Mackenzie gore to 199 for the padres joey wendell numbered 250 for the marlins Tyler mcgill gold rookie auto for the metz to 50 Framed auto winfield padres to 80 This one padres and rookie sky blue hulia rogeria's mariner stuff 300 Now it'll do it. Thanks again everybody. Another one of these coming up next


It's time to process apples into applesauce. This is one of the first things we're doing with apples. I like to have a mix of apples, so I've got some yellow, I've got some red, some tart, some sweet. And so what I'm going to do now is move them through from picked state into applesauce. And the way I do that is I weigh out. I'm going to weigh out about 23, 24 pounds. I'm going to quarter them. I'm not going to peel them or core them. I'm just going to toss them right at the pot and then I'm going to cook them down. And as they cook down, they break down. And once they're broken down, then I'm going to run them through a food mill. And the food mill just takes out the cores and the peels, spits them out. I feed all that to the chickens. They love it. And then all the applesauce gets put back in the pot. I might add a little honey. We'll see how sweet it is. If it's palatable alone, I won't add any honey. And then I'll put it into seven quart jars. Okay, now comes the real test. Come here.  Let's see. What's the verdict? More? More? Okay. It's good. Hoop. More? Yeah. Okay, good. Okay. This is the last one. A little more? Are you done? Okay. Kind of now. Nope, no more. That's it. Thank you. Thank you for your services. And then a little teaspoon of lemon on top. And then we can them and it's good to go. It's real easy, real quick. We'll get to save a lot of these apples for the winter. It's hot out. It's about 90 degrees out. In here, at the top it's only 100, but down here it's got to be 130, 140 degrees. So what's happening is the black mesh in there heats up and pushes air upwards as it heats through convection. So it pulls colder air in underneath, heats it up, and then it gets to the top. It runs all the way up into this box. And then it runs through all these racks holding about, well it was a pound of apples each. Most of that's water weight, probably three quarters of it. And so what I'm doing now is I'm just going through and having a check. I find a thicker piece, still got a bit of juice in it. So I'm going to let it go a little longer. Now just rotate through every hour or so. Make sure I rotate these screens, check on them, maybe turn them over. Not really necessary for apples. They dry pretty uniformly. Whether or not I turn them over to tomatoes, it's more important. And this is probably how I put up half of our apples, maybe not half, but quite a lot of them. Not half counting cider. So later in the afternoon, the sun has gone behind the trees, so we're not going to get any more power out of this. Almost done. I'm going to have to pop them out again tomorrow, but they're pretty close, maybe another hour would have done it, but ran out of time. I have 10 of the pounds of apples in here. It does five in a day for sure yesterday, but just couldn't do 10. That was too much for the sun we had today. They're pretty good. I should mention, you shouldn't eat too many apples, dehydrated apples on an empty stomach and without a lot of water. Last year I was actually hospitalized because I ate about a quart of dried apples, which is about, I don't know, four or five pounds of raw apples.  So don't eat too many of them on an empty stomach when you're dehydrated, which is exactly what I did was exactly wrong. Make sure you only treat them as a snack and make sure you're drinking lots of water. So there's that. I actually love these.


Anomalous Item Entry Item number SCP-030-J Object Class Uncontained Special Containment Procedures At this time, due to the widespread cultural impact of SCP-030-J, containment is considered inadvisable. As such, SCP-030-J is to be considered uncontained until further notice. The possible change to Euclid in the Pacific vector of infection can be quarantined. As of the death of the carrier, SCP-030-J will be considered neutralized after the death of all humans who witnessed SCP-030-J's infection vector prior to May 16, 1990. Description SCP-030-J is a specific combination of sounds that cause a mild compulsory effect upon subsequent hearings of the trigger phase. This compulsory effect is a call and response of several phrases, with no other significant cognitive effects. At this time, Foundation scientists are unsure if SCP-030-J is of a medic effect of the original carrier or a simple info hazard. Data points to the former, as SCP-030-J infections have sharply declined since the death of the original infection vector, PLI-2844. SCP-030-J reached near epidemic levels due to the actions of PLI-2844 during 1969. All infected human subjects became carriers of SCP-030-J and spread the infection further. Further infections from these carriers did not produce the same carrier effect. It is projected SCP-030-J will be neutralized by 2090. Foundation agents are still in the process of uncovering the original cause of SCP-030-J within PLI-2844 with little success. Attached Addenda Addendum 1, Interview Log, PLI-2844 The following interview was recorded in 1971 between Agent Joaquin Casim and PLI-2844. It had come to Foundation attention that SCP-030-J was being spread deliberately by PLI-2844. Due to the celebrity nature of PLI-2844, containment of PLI-2844 was considered untenable with the Foundation level of technology and amnestics available at the time. So you were aware that SCP-030-J was anomalous at the time you infected several million people? I was actually. I don't feel particularly bad either.  What do you mean by that? So it's not a big deal for all of you if I go, PLI-2844 speaks the trigger phase. Agent Casim repeats the counterphrases and expresses mild annoyance. Please don't do that again. I don't appreciate being compelled. Sorry, I just find it funny. Before I would be muttering ma- Agent Casim holds up a hand. Sorry, the phrase, constantly. I was going out of my gourd. I thought I was going to end up in a padded room. So you decided to infect millions of people? Well, every time someone else gave back the response, I felt a little better. It came out less and less. I figured if you had a million or so people doing it, I'd get a little peace. Okay, last question. Did the TV show come before or after SCP-030-J infected you? PLI-2844 chuckles. Before, actually. Addendum 2 Transcription of SCP-030-J's Phraseology Manamana Manamana Manamana


The results of this study suggest that manual selection of tidal flow volume loops using standardized criteria provides a reliable alternative for lung function measures in awake infants with interrupted breathing cycles in a real-life setting. This article was authored by Karen Eileen Stensby-Baines, Refna Katrin Godmonstitia, Martin Fardig, and others.


...the Jack Benny program presented by Lucky Strike. ... In a cigarette, it's the tobacco that counts. And LSMFT, Lucky Strike means fine tobacco. First, last, always. LSMFT, yes, Lucky Strike means fine tobacco. Friends, its only common sense that fine tobacco is what counts in a cigarette. And season after season, independent tobacco experts. Men who spend their lives buying, selling, and handling tobacco can see the makers of Lucky Strike consistently select and buy that fine, that light, that naturally mild tobacco. Fine, light, naturally mild tobacco. Real Lucky Strike tobacco. Fine tobacco that means real deep down smoking enjoyment for you. So smoke that smoke of fine tobacco, Lucky Strike. So round, so firm, so fully packed, so free and easy on the draw. At 50, 7... 59 American. The Lucky Strike program starring Jack Bene with Mery Livingston, Phil Harris, Rochester, Dennis Day, and yours truly, Don Wilson. And now, ladies and gentlemen, let's go out to Jack Bene's home in Beverly Hills where, even as you and I, Rochester is filling out his income tax. Let's see... name Rochester Van Jones, occupation, butler, chauffeur, cook, gardener, ballot, masseur, window washer, and author of what to do in your spare time. Now let's see, exemptions. If married and your wife or husband had no income or if this is a joint return of husband and wife, list wife or husband. I better read that again. If married and your wife or husband had no income or if this is a joint return of husband and wife, list wife or husband. Yep, that's what it says. That's what the man, that's what the form says. Unfortunately, I'm single and have no wife or ages. Oh, hello Rochester. What are you doing? I'm filling out my income tax, Mr. Benny. It certainly is complicated. I'll help you out if you like. I sure would. All right, let's see the form. Hmm. Oh, Rochester, look at all the occupations you listed. You don't work that hard. Don't I? No. I got housemaid's knee clear up to the hip. What? And the back point, middle-aged spread takes over. Rochester, if you want me to help, you pay attention. Now let's see.  I don't want to tell them what my salary is. Why not? Boss, you pay it to me and I'm even ashamed to tell you. Rochester, you've got to put down your salary. Can I write it in red ink? Red ink, why? I want them to know I'm blushing. Now for the next question, list any extra monies you received as gratuities, gifts, or bonuses from your employer. Oh, boss, calm down. Well, let's get on to business expenses. How much did you spend last year? Let's see. There was $70 for uniforms, $20 for a lawyer, and $50 for advice in preparing last year's income tax. I thought I only charged you $40. You're lucky it was $50.  Now we come to dependence. For every close relative you support, you can deduct $400 from your income tax. Well, let's see. There's my mother.   I take her to drive every Saturday night. Your mother? I happen to know that every Saturday night you take my car and go up on Mulholland Drive. Rochester!  Last Saturday night, I followed you and I distinctly heard you mention Lena Horne. I said, mother, don't lean on the horn. Now, Rochester, after you sign this, all you have to do is... I'll get it. Hello, Jack. Well, hello, Mary. I wasn't expecting you. Come on in. Jack, it was such a beautiful day. I thought we might go for a walk. Yeah, let's do that. I feel like going out today. In fact, if you hadn't come along, I was going to call up my girlfriend, Gladys Obisco, and invite her out. Oh, is Gladys still around? Certainly, why? I thought Louis would be mayor auctioned her off last week. I don't like the way you're always picking on Gladys. She's got a nice figure. Did you ever see her in a bathing suit? Yes, and she looks like Gary Cooper. Only from a distance. So what if she is a little thin? She's homely, too. Now, wait a minute, Mary. Gladys may not be the most beautiful girl in the world. I mean, I won't even say that she's the most beautiful girl in the United States, or in Los Angeles, or in Beverly Hills, or in... You know, Mary, I think you got something. Well, I'll get my coat and we'll go for a walk. Okay, Jack, I'll sit here and read this letter I got from Mama. Well, a letter from your mother, eh?  My darling daughter, Mary, everything is fine at home, and the weather is getting to be real nice. We're pretty sure that winter is over now, because last month a groundhog came out of his hole, saw the sunshine, and went back in again to wake up Papa. Papa came out, saw me, and punched the groundhog right in the nose.  Now, now, where was I? Oh, yes. Even though it's nice now, two weeks ago, we had a severe blizzard, and when your uncle Harry came in from the barn, his milking hand was frozen. Gee. I hope it thaws out soon, as I'd like to get the cow out of the house. I are going to the movies tonight and see a new picture. I heard that the picture is all about Jack and his violin. It's called The Beast with Five Fingers. No other news, love and kisses from your mother, the shocking Mrs. Livingston. That's the silliest thing I ever listened to. Oh, wait a minute. P.S. I think your uncle Harry is thawing out, as I just heard a little squirt in the bucket.  No, who can that be? Say, Mr. Benny, I came over to ask you if it's all right to... Oh, hello, Dennis. Hello. I came over to ask you if it's all right to... Hello, Dennis. Hello. I came over to ask you if it's all right to... How, uh, how do you feel, kid? Thanks. I came over to ask you if it's all right to... Dennis. Dennis, what happened to your face? It's all cut up. I know. When I shaved today, I tried out a new blade. Gee, the way you're cut up, what kind of a blade was it? Single edged or double edged? Double edged. Gee. Well, maybe you didn't have it in the razor, right? Oh, razor! Well, certainly. You're supposed to put the blade in the razor before you lather up. Oh, lather! Dennis, if you don't know how to work a safety razor, why don't you get an electric razor? I mean, that would be easy for you. Well, I tried an electric razor, but that doesn't work either. What do you mean it doesn't work? All you have to do is plug it in. Now, anyway, you haven't got a very heavy beard. I know. I take after my mother. Now, look, kid. Kid, what did you want to see me about? Well, I wanted to ask you if it's all right for me to sing. You can't see the sun when you're crying on the program next Sunday. Yeah, I guess it'll be all right. Let's hear your arrangement of it. Okay. We'll leave in a minute, Mary.  Beautiful songs I've ever heard. And I thought the way you... who came in? Dennis went out. Well, why don't you say goodbye or something? Maybe you're not paying him enough. Could be. Well, come on, Mary. Let's go for a walk. Wait a minute. Hello? Hello, Jack. This is Don. Oh, what is it, Don? Well, Jack, this quartet situation is pretty serious, and I think you... Don, I don't want to talk about the quartet anymore. But I just got a special delivery letter. I don't care if you got a... It's from the sponsor. Oh. What does he say? He says, Dear Mr. Wilson, unless Jack Benny has a quartet on his program immediately, next Sunday's show will open as follows. The Lucky Strike program starring Al Pears. Al Pears. With Tony Romano, Maddie Mayne, Spike Jones, Steppen Fetchett, and yours truly, Harry von Zell. What? L-S-S-O-S, L-S-S-O-S. Don, do you think the sponsor is really serious? I'll say he's serious. This letter is written on a Lucky Strike tobacco leaf. She used his personal stationery. Well, I'll see what I can do about it, Don. Goodbye. So long, Jack. Come on, Mary. Let's go. Okay. No, Mary, this quartet situation is a lot more serious than I thought. It certainly is. Starring Al Pears. You know, he's pretty good. He only made one mistake. What was that? He used to go on the air every week and say, I hope, I hope, I hope, I hope, I hope. And what happened? Hope became a star. I wonder what's the best thing that... Who's that? Hey, Jackson. Hey, Jackson, come here a minute. It's Phil. Oh, yes. Hey, Phil. Phil, where'd you get that trailer? Huh? That trailer you're pulling. Where'd you get it? Holy smoke, it's hooked on. What? That's my garage. Oh, for heaven's sake. I better wake up the maid. She lives upstairs. Don't bother, Phil. She probably likes the ride. What are you doing around here anyway? Look, Don called me up this morning and told me you were in a lot of trouble with that sponsor about the quartet. Phil, don't worry about it. I ain't worried, Jackson. I'm thinking about you. What? I'm young and cute. I can get a million jobs. You don't have to worry about me either. I can go back to pictures. But, Jack, they talking them now. Oh, stop. Look, Jackson, if you don't get a quartet, you're going to be in a lot of trouble and you know it. Well, yes, I guess so. Why don't you go up to a talent agency and audition some singers? I don't know who to go to. Well, why don't you try my agent? He's terrific when it comes to picking talent. Look what he did for me. Huh? He fixed me up, so I'll never have to do another day's work in my life. No kidding. What did he get you? Alice. Phil, you mean to tell me your agent got Alice for you? Sure. And when we get 10 kids, we have to give them one.  Now go and see your agent about a quartet. Okay, here's his card. Thanks, Phil. Thanks a lot. So long, Jackson. So long, Libby. Come on, Mary. We'll go and see Phil's agent. Here we are, Jack. Here's the talent agency. Yeah, let's go in. See, there are a lot of people in here. Jack, I think that's the agent sitting behind the desk. I'll find out. Oh, Mr. Mr. Yeah. Are you the man who halves the talent? Yes, Nelson is the name. Low State Nelson. Oh, well, this is Ms. Livingston, and my name is Jack Benny. Well, well, I'd have known you people anywhere. It certainly is an honor that you came here first. Thank you. Benny and Livingston. Yes, sir. I'll make a note of that. Auditions are Thursday, and don't forget to bring your piano player. What? Look, I'm not here to audition. I'm trying to replace a quartet on my show. Oh, yes, I've heard them. I've got a quartet here that can take their place, and nobody will know the difference. Good, good. They're right in the next room. The Seal Brothers. Seal Brothers? Well, we can change the name. Oh, no, you can't. Why not? Because they're seals. What? I'll get them for you. Okay, seals, you're on. Oh, for heaven's sakes, four seals. There used to be five. I gave my wife a coat for Christmas. Look, I'm not interested in your wife. You can send those seals back to their cages. Okay, you can go back. He doesn't want you. What are they mad about? You broke up their gin rummy game. Come on, Mary. Let's get out of here. Okay, but I've got another quartet for only $60. I don't care of $60? Yes. Sit down, Mary. Let's hear them. Come on in, fellows. They turn out to be golfers. There's going to be trouble. Look, Jack, he isn't kidding. It's a real quartet. Yes, yes. Mr. Nelson, what are the names of these four boys? John, Charles, Irving, and Thomas. What'd you put Irving in there for? To keep from being sued. Well, okay, Mr. Nelson, let's hear them. How'd you like them, Mr. Benny? Well, I don't know. What do you mean you don't know? Their voices are beautiful. They sing like larks. That one looks like a bald eagle. Mr. Nelson, these fellas may be all right for something, but I've got to have a quartet that can do the commercials for my lucky strike program. Oh, they can do that. They can? Of course. Listen to this. Go ahead, boys. Why'd you stop them? What's wrong? Well, they're not exactly what I want. Very well. All right, boys, you can go. Come on, Mary. I'm sorry I bothered you, Mr. Nelson. Oh, don't go away, Mr. Benny. I'm sure we have something that would... Excuse me. Hello, Nelson Talon Agency. Mr. Nelson left town. What's your complaint? And no complaint? This is Nelson speaking. Yes, we have talent for every occasion. For a wedding, you need what? A violinist. You willing to pay how much? $6 and dinner? I'll look in my book and see. Violinist, violinist. No, I'm sorry. We don't have any violinists available. Oh, Mr. Nelson, don't hang up yet. Yeah, what is it, Mr. Benny? Ask them what they're having for dinner. It's too late now. They hung up. Oh. Now, Mr. Nelson, I think I'll run a walk. Now, wait a minute.  But I didn't like those four fellas you had on. All right, so you didn't like them. I have other talent, you know. Hey, girls, come out here. Girls? Yes, their names are Maxine, Patty, Laverne, and Irving. Irving again? No, Irving Schwartz. Now look. Take it, girls. Mr. Nelson. Mr. Benny, why did you stop them? They're wonderful. But they're not what I had in mind. Mary, what do you think? I like the seals. And would you like to interview them? Well, all right.  I am. I don't know about the other three. Well, I have to be very particular about my show. Where did you girls work last? Oh, our last job was in San Francisco. We worked on the top of the mark. Well, that's pretty good. How long did you work on the top of the mark? Two months. Why did you leave? We finished painting the roof. Well, look, Mr. Nelson. I'm sure I can't find the kind of a quartet I want here. So I think we all... In just a moment. The Nelson Talent Agency. What? Oh, yes, just a minute. Fido. Fido. Fido, you want it on the phone. Mary, it's a dog. Yes, and one of the smartest talking dogs in the country. A talking dog. I don't believe it. I'll show you. Here he comes now. Fido, how old are you? Fido, how old are you? Fido, say something. I don't know what to do with him. He's been acting like this ever since he saw Nora Prentice. Some talking dog. Fido, if you don't talk, when we have dinner tonight, I won't let you bury your bone in my mashed potatoes. Now, come on. How old are you? Seven years old. Gee, he can spell too. Now, Fido, spell cat. Seven years old, he can't spell cat. Look, Mr. Nelson, I didn't come here to get any trick bought of a lax. All I want is a quartet, and you haven't got one that suits me, so I'm going. Now, just a minute, Mr. Benny. I've got another quartet that is positively sensational. Look, they're Russian. A Russian quartet? Yes, I'll call them. Tavarovich, Ivanovich, Stepanovich, Baranovich, come here.  Mr. Nelson, look, I can't use a Russian quartet. You haven't heard them yet, but all right, boys. I don't want to hear them talk. I want to hear them sing. Boys, boys! Now, Mr. Nelson, I don't want to hear them talk. Let's not waste time. If they're going to sing, let them sing. Okay, come on, boys. One, two. One, two.  Ladies and gentlemen, during the past week, over one million young people, as well as adults, have been observing National 4-H Club Week. 4-H Clubs are made up of boys and girls who are agreed to learn to do by doing some phase of farming, homemaking, or constructive community activity. I, for one, am happy to salute the fine work of America's 4-H Clubs and urge all boys and girls between 12, 10, and 21 to join an all-qualified adults to enroll as leaders in their locality for the continuation of National 4-H Clubs. We'll all be back in just a moment. But first, here is my good friend, Basil Riesdale. As you listen to the chant of the tobacco auctioneer, remember LSMFT. Lucky strike means fine tobacco, and fine tobacco is what counts in a cigarette. Speaking of fine tobacco, here are the words of a man who really knows tobacco, Mr. Frank A. Brown of Stoneville, North Carolina. An independent tobacco warehouseman for 25 years, he said, Season after season, I've seen the makers of Lucky Strike by good right tobacco. Tobacco, you just can't beat for smoking quality. I've smoked Lucky's myself for 29 years. Year after year, at the market after market, independent tobacco experts like Mr. Brown can see the makers of Lucky Strike consistently select and buy that fine, that light, that naturally mild tobacco. Fine, light, naturally mild tobacco. Remember, LSMFT. Lucky Strike means fine tobacco. LSMFT. No doubt about it. Lucky Strike means fine tobacco. And this fine Lucky Strike tobacco means real deep down smoking enjoyment for you. So smoke that smoke of fine tobacco, Lucky Strike. So round, so firm, so fully packed, so free and easy on the draw. Brown, where's Jack? Well, Mary, he had to leave. He's on Phil Harris' show. Gee, I didn't know he was going to work on Phil's show. Yes, I heard them talking in the hall and Phil offered him some money. Oh, good night, folks. This is NBC, the national broadcasting company.


State of America, the Honourable State of Israel, of the State of Israel, Lincoln, Secretary of State of the United States of America, the Honourable Benny Gantz, Minister of Defense of the State of Israel, Rabbi Yisra, Chairman of the Yad Vashem Council, Mr. Danny Dayan, Chairman of Yad Vashem, Holocaust survivors, ladies and gentlemen. We have gathered here in the Hall of Remembrance at Yad Vashem, on the Mount of Remembrance, Jerusalem, to commemorate the six million Jews murdered by the German Nazis and their collaborators. The Ankor Choir, directed by Ms. Daphna Ben-Yokhanan, will open the ceremony with a song, Walk to Caesarea, written by the young poet and parachutist, Hanna Senesh, who was killed behind enemy lines in Hungary in 1944. It's President Biden to rekindle the eternal flame in the Hall of Remembrance, to rekindle the eternal light in this Hall of Remembrance before the sacred remains of our martyrs. We unite with the blessed memory of six million of our people who died a martyr's death at the hands of the German Nazis and their collaborators. Of the Jewish communities destroyed in a wicked attempt to eradicate the name and culture of Israel. We remember with veneration the fortitude of the fighters who kindled the sublime flame of rebellion among the besieged masses of the ghettos. The heroic and persistent struggle of the masses of the House of Israel on the threshold of destruction for their human dignity and their Jewish heritage. We remember the righteous among the nations who risked their lives to save Jews from persecution and death. I am honored to invite President Biden to lay the wreath on the slab under which the ashes of Holocaust victims from the extermination camps are buried. We'll now recite El Malei Rahamim, a prayer for the souls of the martyrs. Ladies and gentlemen, the ceremony is concluded. President Biden will now greet Holocaust survivors Dr. Giselle Gita Sikovitz and Mrs. Rina Quint. Please remain in your places. Thank you for those two Holocaust survivors. One of them, Rina Quint, spoke to her, who earlier at the Holocaust Memorial Museum. Let's take a listen to what she had to say ahead of what we can evidently see is an extremely heartfelt encounter. I can't help thinking that as a child of 10, a little girl who didn't speak English, who didn't have a mother, who didn't have a father, who didn't have a home, came to the United States very frightened and very much alone, and I became an American citizen. I was adopted by a wonderful American Jewish family. I married a wonderful, wonderful American man and had children there, but I never thought that from a child who had no education and no possibilities to all of a sudden be meeting the President of the United States. It's really quite an honor. Yes, and now we're back to those live images of the American President at the Yad Vashem Holocaust Memorial Museum, taking his time to have a genuine conversation with two Holocaust survivors of American descent who came there. And we can see we, you know, sometimes all cynicism should be put aside. You can see what a genuine, honest interaction it is between them, how those lovely Holocaust survivors are really taking the opportunity to speak to the American President. And this is, again, the opening note of the Biden visit here in Israel. Once he concludes this conversation and this visit at the Yad Vashem Holocaust Memorial Museum, he will be heading to the King David Hotel, where he will basically finish his agenda for the day, but we can safely assume it will include a whole lot of preparations ahead of an extremely busy day tomorrow. And so to get a better understanding of what is it exactly that is expected for the American President tomorrow, joining me now in studio, Dr. Fadi Ismail, former senior, Stafford U.S. Congress and Research Fellow at the International Institute for Counterterrorism, and Mr. Udo Weintraub, lecturer at the School of Media Studies at the College of Management. Thank you, gentlemen, very much for being with us. Again, we can still see those live images of Joe Biden, the American President at the Yad Vashem Holocaust Museum. As we've mentioned numerous times in the broadcast, it's not his first visit to there. As an American President, and his record when it comes to Israel is to an extent, it cannot be disputed. And even while we're mentioning that, he's not necessarily being perceived as such here in Israel when it comes to public sentiment. What would you attribute that to, Mr. Weintraub? Well, I think that many Israelis got used to Trump's closeness to Israel, and it's not the same style. Also President Biden, even though he truly is a friend of Israel, and even the Holocaust he learned about it personally when he was a youngster in his family, was a Zionist. He didn't hesitate from time to time to speak his piece, and there were some altercations in some previous visits, though he was voicing his own sentiments. But if you look at his record, he has known Israel since actually it's established at the beginning. And as a junior senator, he came to Israel and continued on. This is his first visit as a president, and I believe that the record speaks for itself. Yes. Indeed, and he arrives here at a very, let's call it shaky timing to an extent when we're talking about the Israeli state of affairs. It's no secret perhaps that the Biden administration was fond of the experiment, let's put it that way, of the so-called unity government, the Naftali Bennett, Yair Lapid led government. Now we are amid election campaigning season. The American president expected to meet, of course, both Prime Minister Yair Lapid and opposition leader Benjamin Netanyahu tomorrow. What do you think we should be expecting during those high-profile meetings, Dr. Ismail? The meetings tomorrow. Yes. Some of it will be probably nice cities, diplomatic nice, some of it. It depends. Well, that's a part that we'll see on camera. If there will be a part outside the cameras away from them, then I will assume that they will speak about the issues of the day, the way governments speak to each other. It is true that the president, Biden, knows Israel very well, he's been here, I don't even know how many times, they say 10 days that we know of, and he met all Israeli leaders since Gundam-Irisen, so he knows the Israeli political system really well. The guys who are going to talk tomorrow with all due respect, both Prime Ministers, the former one and the current one, to him are basically youngsters, young men. He can tell them things that the Israeli government has done when they were still in high school, literally in high school. So not even in the case of Lapid, not even in the school. I just saw one of the Israeli reporters tweeting that the reception ceremony at the International Ben-Gurion Airport, your Lapid, the Prime Minister, told Biden, not sure if you remember, but eight years ago I met you at your office, you were Vice President, and you told me that if I had your hair, I would have been the American President. If I had your height, I would have been the Israeli Premier and look at them now. So without a hair and without the mileage, they still got you to this point. But putting aside politics, every single character we've mentioned, they have their fair share of politics to deal with. If we're talking policy, considering this, again, sensitive political timing here in Israel, the honest truth that Israel is somewhat of a stopover for Biden on its way to Saudi Arabia, is anyone really expecting major breakthroughs during this visit, or is it so-called maintenance work? I think to a large extent it's maintenance work, but there have been reports that there's going to be a declaration, a Jerusalem statement, or a Jerusalem declaration tomorrow, in which President Biden will recognize Israel's right to dissuade, and also America's right to dissuade Iran from acquiring a nuclear weapon, and also recognize Israel's right to dissuasion and to self-defense from a variety of threats or a combination of threats. So this is sort of a veiled language, and parts of the agreement will not be made public to note that Israel can undertake certain operations. We can only imagine what they are to dissuade Iran and other enemies. So this is perhaps the major diplomatic effort. But his main thrust, as far as I see it, will be the second leg of his voyage, which is the GCC plus the Gulf Cooperate.