I will give you an example of a manufacture of a compound called Azadiractin actually a bio pesticide I do not know if you know AID parry makes it it is been talked about you know this big controversy about neem a lot of these neem seeds have been powdered and used as fertilizer in this country but suddenly these companies come along and they want to in fact they patented and said here after used by farmers in India they pay this is one of the few battles that we actually won I think it was Dr. Michelle Kerr CSI our director general who spearheaded this attack and finally there is a class here that says the traditional wisdom cannot be patented and if patented you cannot chart the guys who had the traditional wisdom in the first place but we have a lot of such a cases but this is a bio pesticide taken out of neem seed actually long go there was a consultancy project that came but the way the industry develops this is simply by empirically doing it in the lab then scaling it up and so on but they do not do the fundamentals what I wanted to in fact they came to me I told them these are the fundamental measurements that are required so I will trace that I will give you an idea as to where we use our face equilibrium information what you do is actually you take your neem seeds this is a bio pesticide these the whole area of bio pesticide became important after they discovered in the US that the golden eagle eggs were breaking because of DDT content in the shell we are using DDT very widely as a pesticide it sort of went through the chain and finally somewhere along the I think the rabbit see at it and the golden eagle ate the rabbit so finally DDT persists so the level of DDT got so high that the shell of the egg broke very easily therefore the number of golden eagles decreased this is a study by an environmental group but finally they decided that these pesticides have more harmful effects than short term benefits so they were looking for bio pesticides one of the bio pesticide that is used that is recommended strongly is called asadiractin I have no idea about its structure and things like that but the way it is made what you do is to take neem seeds and you grind them this is some ball mill some mill of some kind comes under your unit mechanical operations I do not know if you still have a course you have a course called mechanical operations still you have a course you do size reduction particles ball mills and rod mills and things like that you do not have a description of that huh but you have it in the curriculum I forgotten whether you have it anyway this powder is taken typically what they do is the process could be batch or this thing you add water and you take out the extract the water dissolves a large number of components including asadiractin will describe the process and then in what you have is a liquid liquid extractor this is water this is ethyl acetate asadiractin is basically a non polar compound as a thumb rule non polar compounds are more soluble in organic solvents than in polar solvents like water so you take a liquid liquid extraction you have distribution of asadiractin let us say the component that I am interested in is a this is asadiractin so you have a in both faces a w and a e a ethyl acetate the asadiractin both this extract is taken out and then in some kind of a heater the ethyl acetate is evaporated out and the asadiractin is collected as you increase the temperature the salt the what you really do is you get you get to a point where the ethyl acetate evaporates and leaves behind more and more concentrated solution of asadiractin finally the asadiractin is separated out you broadly the process that they use water extraction use regardless of the fact that the solubility of asadiractin water is very low but water is about only 10,000 times cheaper than ethyl acetate so you can use 1000 times the amount of water and extract the compound first along with it will come some undesirable compounds then what you do is to choose a solvent in which the distribution coefficient that is the solubility of asadiractin in ethyl acetate divided by the solubility in water is very high so with a small amount of ethyl acetate you can get a large amount of asadiractin and one of the problems that ad parry had they developed the process in industry is just as capable of when there is a monetary drive they just as capable of getting a process through in fact they will develop a process faster than the university because university will keep asking fundamental questions one of the questions that you have is what is the distribution coefficient between these two what is the solubility in water because this at most can be a saturated solution so thermodynamic quantities of interest are what is solubility of asadiractin in water and of course how does it vary with temperature because it is variation of temperature will determine whether you are going to have a boiler that heats the water up brings the water at some temperature normally pressure affects the negligible in the condensed phase so normally its pressure is not as you can ask another question here about the ball mill also you can ask is there an optimal the merriest optimizations of the ball you can ask how many balls should you have inside it what should be the size in what should be the rotation speed and these are all very poorly understood so a lot of it is caused by empirical correlations in chemical engineering one of the things that is very still not well understood is simply mixing second is grinding the energy required broadly speaking the energy required is create voice for creation of surface area incidentally in thermodynamics courses we write du is equal to Tds-pdb you have to write plus sigma da where sigma is the surface energy per unit area so you have to add this term 99% of the application this is very small but if you are looking at something like a ball mill where energy is created this is negligible this is the significant part of it so per unit of area created if you have very small particle the surface area per unit volume increases dramatically therefore you have to pay more penalties but this kind of thing is envisaged already you should make this DE because you should include potential kinetic energies you should include other forces and their conjugate variables but this is a mechanical operation and then you have this extraction with water normally these are done counter current you know water goes in and comes out and that is the thing that is collected but this is one question the second question is what is the distribution coefficient for other directing between water and solvent the question is what is the best solvent and there are 100 considerations for the best solvent ideally if thermodynamics progresses to your point where you understand it fully you should be able to sit at a computer and choose the best solvent some extent you can do that now the data available is very extensive and you can do that to a large extent eventually it simply you will put ethyl acetate water as a direct you will you will not put ethyl acetate you will put put in water and other direction and say what should be the next other solvent for the next step and it will go through all these you hit the idiot box on the head you actually just move the mouse around and click best solvent so you do not even have to be very educated in fact as spent tech has a lot of these flow sheets everything is marked there so you click design distillation column and will design this you just have to need to know how to read the output but let me come back here as far as thermodynamics is contained you still have to supply this data or even in aspen tech there is a thermodynamics package which is where you need to so all I have to do is if I call this to this is water plus as a direct in this is component 1 this is component 2 so I said mu 2 solid is equal to mu 2 liquid this is mu 2 solid is mu 2 solid pure so this is I just leave this as pure solid this has to be written as mu 2 liquid as a direct in as a solid so you write this as mu 2 star then as I told you you do this whenever you do not you have a star that you cannot get away get rid of an equation thumb rule divide by T differentiate with respect to T so you get minus h 2 solid by T square is equal to h 2 bar star minus plus R partial of L and ? 2 x 2 with respect to T so you integrate this equation if you know this value or if you know that h 2 is solid sorry h 2 bar star will become infinity because you know you can go to infinite dilution so this is infinity you can go to infinite dilution because when you divide by T and differentiate with respect to T first of all you divide by T this term is independent of composition so the left hand side should also be independent of composition so you can go to the limit as composition goes to 0 so h 2 bar star is the same as h 2 bar infinity same as the partial molal enthalpy at infinite dilution this is a measurable quantity so you have you can have data for this by T square if you like R T square is partial ? 2 x 2 with respect to T even if you do not have this data you can treat this as a constant and fit the data you need an expression for ? 2 so log ? 2 in this case as a direct and self is a non-polar solute and the polarity of water does not affect it enough for you to worry about these theories you can write this as a x 1 square I will say example so you get log of a which is a constant it is a function of temperature and x 1 square then you have x this logarithm is a x 1 square then you have log x 2 so you can integrate this equation with some data on solubility that is you have to have measured solubility of azadiractin in water ad pari did not have this data actually we had to do some measurements to get this data this is trivial all you do is get some azadiractin put it in a beaker of water maintain that water at some temperature and wait for infinite time in a stir rate ideally you should wait infinite time normally you wait an hour and then it comes to reasonable equilibrium then you make a measurement you stir it but anyway having done this so you can get these constants and therefore you can get x 2 so you can get x 2 as a function of temperature from these equations and some data I have to give you a bit of data and see if I have the data in any case I cannot give you the industrial data because it is there but I will see if I will just take a look and see if there is data I will give it to I give you the data then you can fit you treat this as one constant the other constant is this is approximately constant because you are treating with water the temperature range that you are talking about is going to be something like 30 to 80 degrees centigrade not much wider in absolute terms the small range of temperature 300 to 350 degrees k so you are talking of a small range in temperature so you can treat this as constant this is one constant this is a constant B that is a constant a so you will fit two constants and you will be able to get expression for x 2 as a function of temperature from this equations so that you do and then you can ask later in an optimization program you have to work this also you have to have a heater here you have to supply some Q so this is at room temperature and this is some desired process temperature you have to ask what is an optimal value for TP normally what will happen is in any industry they will have a boiler they will have some waste heat so it is convenient to pipe that waste heat and do the heating anyway so that energy cost may not be for them it may not be an explicit cost in industry those are local variations if you have to set up a boiler it may not be worth it but anyway this is one part of it the other part of it is the distribution coefficient that again what you do is to ask what happens if there is water plus as a direct in this is component 1 this is component 2 this is ethyl acetate which is component 3 plus as a direct in which is component 2 in practice actually you will get some ethyl acetate here plus water here there is solubility of ethyl acetate in water and solubility of water in ethyl acetate so they were worried about recovery of ethyl acetate as well so you have now essentially equations like this mu 1 water is equal to mu 1 ethyl acetate this is the ethyl acetate layer this is the water layer 1 is for water mu 2 water layer is equal to mu 2 ethyl acetate layer and mu 3 water is equal to mu 3 ethyl acetate layer there are two layers and you are asking for equilibrium between the two I think one is water you will find that these solutions are sufficiently dilute so you must remember that this is these two are solvents this in the water layer actually in both layers I can say ethyl acetate in this case in both layers water and ethyl acetate are solvents and as a direct in is solute this is solute as far as we are concerned this is a solvent or a solute depending on the phase in which it is at temperature and pressure of the extraction so at this temperature and pressure that they are using these two are liquids water and ethyl acetate are liquids so you can use the solvent models for them so in both phases here you will simply write mu 1 liquid pure plus RTLN gamma 1 X 1 W W means W phase this will be equal to mu 1 liquid pure so for solvents this mu 1 liquid pure will cancel on both sides because this is the chemical potential of pure water at temperature and pressure of solution at temperature of the at temperature pressure solution that is the same here so this animal will simply cancel and all you have is gamma 1 X 1 is equal to gamma 1 X 1 water phase is equal to ethyl acetate phase notice the water phase will have it will be predominantly more than 90% it will be water so your X 1 W is likely to be almost 0.9 or higher so it is because gamma 1 is much higher in the ethyl acetate phase that you are able to get this kind of a distribution then if you look at this mu 2 W again this is 2 is what 2 is solute so you have a small problem here you have to write mu 2 star in the water phase plus because remember when you refer to mu 2 the solvent counts the medium in which it is there when you have a star here gamma 2 X 2 mu 2 star ethyl acetate phase plus RTLM gamma 2 X 2 is gamma 2 X 2 in the water phase this is gamma 2 X 2 in the ethyl acetate phase this will be exactly like that for the last one which is again a solvent this can be rewritten very simply what you really have in the liquid liquid extraction system is some amount of water they told you plus as a direct in plus ethyl acetate this is the water phase this is the ethyl acetate phase this again has same it has like water plus as a direct in plus ethyl acetate but if you are doing a batch process for example it is simply the amount of you can vary the amount of water in the amount of ethyl acetate so that is another control variable you have so you have to worry about the amount of water you have phase you have per unit of ethyl acetate phase you have because what you will get is an input to the system that essentially has input and then you take two outputs the water layer and the you separate these two is done by gravity all you have is you have to be careful not to disturb the interface and pull out the water phase the ethyl acetate phase but this in so what you do is a mass balance of as a direct in that comes in equal to the z direct in that leaves in the two phases so you will have three mass balance equations for water ethyl acetate and as a directed you have to combine them with these three and then solve you can vary the amount of ratio of ethyl acetate to water that you take in the beginning the extent of the phases that is a control variable you can therefore control these but you have to solve these three equations call this one two and this is three together with there is a mass balance the mass balance equations you can write them down there is a mass coming in and a mass to sing leaving you are assuming that you are giving enough time in the system for equilibration to occur again if you mix this system well and allow it to settle it usually comes to equilibrium very quickly so these are near equilibrium processes many of these process they come under I think now will be taught under mass transfer one the other course is about processes in which you do not reach equilibrium in the in the first set of processes thermodynamics is very important because the equilibrium calculation is what tells you how much will be what will be the composition these are these two will be in equilibrium this is what comes from your previous extraction it is water with azadiractin in it because it is actually the actual process is complicated by the fact that not part from azadiractin you have many other solutes neem seed does not oblige you with just azadiractin usually has about 12 13 components so you have to write all the components and the balances and solve these equations together you have to worry about whether those components affect the activity coefficients so you will have a multi component expression already you have three components so you write write a multi component expression so in actual practice I would not give you the in an undergraduate course one does not give you the details but I think one assignment I will include a three component system so that you do one calculation to get a feel the calculations too complicated for you to do in an exam or a thing the whole thing is not about the exam knowing how to do the calculation so basically this is the only equation you have to worry about really so you look at log ? 2 x 2 in the water phase by ? 2 x 2 in the ethyl acetate phase this is log ? 2 x 2 in the water phase – log ? 2 that is equal to µ 2 ? ethyl acetate – µ 2 star in the water phase by RT what you are really interested in is x 2 W by x 2 E this is the distribution coefficient n of azadiractin in the water phase by azadiractin in the ethyl acetate phase x 2 ethyl acetate is much higher than the solubility in ethyl acetate is much higher so you actually expect n in this case the way it is written it will much less than one so actually in a sense using water first to do the extraction is a waste but there are two advantages of using water one is it is inexpensive so you can use a lot of it to extract the azadiractin second is it tends to take out the other non-polar compounds in smaller quantity if you use ethyl acetate the bulk of the seed neem seed when ground consists of in organic matter all of it will be extracted into ethyl acetate if you use it directly then you will have to separate the components in the ethyl acetate layer this way azadiractin comes out more dominant it so happens that azadiractin solubility in water is low but it is higher than the solubility of the other organic components in the neem seed. So it is an advantage to use water first so what you do again here is to differentiate and if you differentiate you will get partial of log I leave this ? 2w by ? 2e into n into 1 by n is it x 2w now into n with respect to t is equal to this is h 2 bar infinity in the water phase – h 2 bar infinity in the ethyl acetate phase by RT squared h 2 bar infinity in the ethyl acetate phase if I differentiate this I told you these things are independent of composition therefore I can replace h 2 bar star by h 2 bar infinity and again h 2 bar infinity now if I subtract this solid enthalpy of azadiractin h 2 solid from both I will essentially get ? h for ? h 2 you like in the water phase – ? h 2 in the ethyl acetate phase by RT squared that is add and subtract h 2 solid pure if I write small h 2 it is pure solid this is essentially the heat of solution of water this is the heat of solution I mean heat of solution of azadiractin in water the other one is ziractin this data is available and usually this is a constant so you can treat this is a constant µ 2 is not constant that is why you cannot do this stunt here the ? h is more constant than the chemical potential so you do this and once you have this data you have to integrate this normally the solutions in water and ethyl acetate are both sufficiently dilute for you to treat ? 2 in this case in these two they would not be the ratio will not be one the ratio will be constant so ? 2 w by ? 2 e very often in the process is treated as a constant so you get one more constant in the fitting or if you differentiate this log ? 2 w by ? 2 e you can neglect that so effectively I will say approximately this is what is used for correlating data you get ? log n by ? t that is ? 2 w by ? 2 e is not one the advantage of having ethyl acetate is that it is the activity coefficient ethyl acetate is much lower so you get higher solubility but this ratio is independent of temperature approximately independent of I will say ? 2 n w by ? 2 e is independent of t in the range that you are talking about therefore you get ? log n by ? log t this will be some constant is we will get is equal to a by t square take this along with the R then you get log n is equal to – a by t plus b so you do two experiments in the at two different temperatures you can get both your constant A and B you have to measure this distribution you have to do this you keep this in the lab and measure the concentrations both in the actual process I told you this could be 90% of it of this vessel could be filled with water 10% will be the ethyl acetate phase or you can vary this and you can ask what is an optimum so essentially this is where thermodynamics is involved in a process in this process you will simply ask what is the right temperature to use what temperature should I use so that I get an advantages distribution coefficient you solve all these all this can be solved directly on a computer when doing it by hand is a pain but you can do it straight on at the best way to do it as far as industries concerned is to use a spreadsheet.


What's up everyone? It's your girl Kinsey here from Layton Sports Cards, and I'm ripping a personal for Brett W. We've got another 2022 Panini Phoenix football hobby box. Alrighty, Brett, good luck. Thanks again for hopping back in. See what you can get you here. Winston, Kway Walker. We've got Yellow, Adam Thielen, Vikings, 275, Randall Cunningham, Fire Forged, Cook, Stewart Goff, Danny Gray. We have Aqua, Justin Herbert, that is 135-150 and Matt Corral, Contours rookie. Sweetie Bonito, Pink, Terry Bradshaw, Fire Forged, 20 of 175, Jalen Hurts, Flamethrowers, Kittle, Bateman, Baty. Got Lava, James Cook, Rookie to 175, Hot Routes, A.J. Brown. Smills, Garrett Wilson, Rookie Base. Got Regimarch Chase, Bangles to 250, Rookie Rising, Silver Rookie, Traylin Burks. JP, one second, Bowel Mountain. We've got 250 on the Orange Rookie Tyler Algier. Nice one there, Falcons. Yo, Brett, Jeff M. Yours will probably be the last one going today. Kenneth Walker, the third. Saw that come in for some impeccable. It'll probably be the last one here in about 15, 20 minutes. Ellen Stafford, Ferguson, Purple Hot Routes, DK Metcalf to 125, Kirk Cousins, Flamethrowers, Filler, Holmes, yeah, Pickens. We've got to 149, Christian Watson on the Rookie Jersey, Watson. We have nice, Colour Burrs, George Pickens Rookie. There you go. Nice one there. Prescott Lance, Jabbo. We've got Red Rookie of Christian Harris to 250. Hot Routes, Drake London, Silver. Andrews Anderson, McBride. We've got a, is that blue or purple? Blue to 35, Tom Brady. Nice low number, Brady there. Great box here for you, Brett. Flamethrowers, Zach Wilson, Collins, Field, Derek Stingley, Aqua Flamethrowers of Derek Carter 150. Hot Routes, George Kittle.  Diggs, we've got Justin Ross, Silver, James Conner. And it's going to be Colts to 299, Silver RPA of Alec Pierce. Colts. All righty, that will do it for you, Brett. Better second box for sure. A nice one there. Congrats on the hits. We'll get these right out to you.


Good afternoon everyone. It's been a great day here today at Cubeday India, the first major CNC event in India. So it's a pleasure speaking here and there have been some amazing talks all day long, really great energetic conversations and we'd like to continue this momentum a little further. The zero interest rate phenomena is truly over. Companies worldwide are focusing on managing their costs in tooling, also like parallely to their performance optimizations. We're going to be talking about how a few years ago we made a few mistakes that almost cost us upwards of $100,000 and these are mistakes in our long-term metric from your storage and yeah, that's what we're going to talk about. We're all humans. We're all going to make thousands of mistakes in our lives. It just so happened that these mistakes synchronized so beautifully together that each mistake exponentially increased the cost. So let's dive right into it. So who are we? Well, I'm Shubham and this is Ankur, we're the founding team at ZenDuty, an incident response platform. We have accumulated years of experience making mistakes, learning from them and advocating for best practices in production in SRE teams worldwide. So what is this talk really going to be about? Firstly, we're going to talk about why we needed a solution like Thanos in the first place. What other solutions were present in the market? So this was Victoria Metrics, Cortex and we're going to see how they compare against Thanos. Then our production setup and our expectations from that. And then the good, the bad and the ugly, not the smooth movie but what went wrong, how we fixed it, the actual cost and the potential cost for our missing configurations. So just to give you some context into what we're building and the scale that we could run into, this is what ZenDuty looks like, the incident response platform. So we connect with the entirety of our customers' infra support and comms channels and deliver them alerts across all platforms. With the context straight from the monitoring stack, we have to automate everything from their first alerts. So naturally, when our business started picking up, our production scale also needed to ramp up that fast. With that, we took the cue that our observability stack needs to be just as reliable and we need to step that up a little. So why would we need Thanos? Well, Prometheus has a simple and reliable operational model. It's capable of tracking millions of measurements in real time and with powerful and pretty capabilities of visualization and querying via Grafana. But after a certain scale, it has some shortcomings. And these shortcomings, they fail to answer questions that naturally fast-skating teams come across. Questions like how can I store petabytes of historical data? How can I store that without sacrificing response query times? Can we access all the metrics from a single API query? How can I merge replicated data collected via Prometheus HA setups? And turns out, Thanos was the solution to all these questions. So, yeah, a few major reasons why we chose Thanos. Firstly, the global query view. So Prometheus encourages a functional sharding approach. Even a single Prometheus server provides enough capability to liberate users from the complexities of horizontal scaling. And regardless, you would still want to access all this data through a single UI or API, which is the global view we're talking about. So for example, you can generate multiple queries for a single Grafana graph, but they're all going to be against a single Prometheus server. With Thanos, however, you can create, like, you can query multiple Prometheus servers at the same time because they're all available from the single endpoint. Next, we have reliable historical data storage. Naturally, Prometheus sidecar watches, so Thanos sidecar watches Prometheus for new blocks of persistent data, and it pushes this to the object storage, which in our case was AWS S3. Then we have down sampling. So once you start querying historical data, you'll realize that there are some fundamental big complexities that make your queries slower and slower as you retrieve weeks, months, and eventually years' worth of data. And the solution to this problem is good old down sampling, which is reducing the sampling rate of the signal. So with down sampling, you can zoom out to a larger time frame and still maintain the same number of samples and thus keeping your queries responsive. And lastly, we have high availability. So Prometheus has an HA model that essentially collects data twice, which is as simple as it could be. However, the merged and redeplicated view of both streams that Thanos provides is a huge usability improvement. So we considered a few different solutions at that time, but there were a few problems in every one of them. So a few of them weren't community-driven. They were either too young. They either did not support object storage, or just their general integration with the rest of our observability stack was not as efficient as we expected it to be. So Thanos is where all the great engineering teams were at that time. That's where we decided to be. But if you were having this conversation today, it could go very differently because you have some good competition from Victoria Matrix and Cortex depending on what you're looking for. I'll give a quick shout out to even last line who is trying to build something in this space to check them out as well. And so let's take a look at how we implemented Thanos, a beautiful Titanic before it hit the iceberg. So Thanos runs as the sidecar to the Prometheus instance. And it pushes blocks of object data storage to your S3. We had implemented Prometheus functional sharding, which means that we were scraping the same data twice from multiple ACs. This setup requires Thanos' ruler to set up the promql equations for alerting. And it should be in turn connected with the Prometheus alert manager over there for delivering the alerts to the final destinations. In our case, the NUTN slack. The Thanos squareier uses the storage gateway to retrieve all data from object storage. And it's used by teams for any analysis or to create internal or external dashboards for their PMs or any other team members. And in order to get faster query results, the Thanos comparator that we talked about is used to downsample the data and store it back to the object storage. Now let's see what we had for log aggregation. So we implemented log aggregation via Fluentbit or the other word app that not a lot of us talk about. Well, Fluentbit is a well-known log aggregate tool which is a worthy successor to Fluentd. And we deployed the daemon set and started exporting our logs to OpenSearch. Fluentbit uses a plugin to add extra labels to the log line for each log stream. And for getting these extra labels, it queries the Kubernetes API for labels like pod name, namespace, annotations, and any other relevant information. So what if the Kubernetes API does not respond? What if for some reason it fails? And if that happens, it retries after some jitter. It retries and retries and eventually pushes the data to the destination. That will, this small caveat somehow managed to form a little problem in the system. And what were the expectations after this? Well, we spent a lot of time on this. It wasn't easy to configure, but we got it configured. And our expectations was effortless querying, cheap long-term metric storage, leisurely downsampling data whenever and however you wanted. And in essence, just a much cleaner metrics and logging system. And yeah, this should work straight out of the box, right? It didn't. Not really. In an IAVT, we saw a bunch of anomalies and errors popping up that we didn't make sense of initially. But sooner or later, it all came together. So the first in this series of unfortunate events, we had an issue with our network topology. So while we were building a log and metric handling infrastructure, we missed to keep in mind some basics that cost us daily. Next, I'm just playing with high cardinality. We were scraping too much data and metrics for our skill. And finally, our friendly downsampler that I've been talking about all this long, there was a mismatch in our implementation and our expectations with the Thanos downsampler, causing an increase in network and storage costs. So this sounds intense, well, it was. And I'll pass the stage now on to Ankur, who was off the deep end when this happened. Let's see what he found. Thanks, Shivam. So to drill down, this is a general network topology which looks like where you have multiple availability zone and multiple VPC. We were using that only when we were stuck with that issue. But the problem with this topology is that we were pushing, like he's mentioned that our AWS storage resources, AWS S3, so we were pushing a lot of logs data through S3. And they were actually traversing through NAT gateway, which means egress coast, and which got to a certain stage which impacted a lot to us, although this was not even a 10% or 20% of it. We'll come down to the majority part later. So how did we fix it? We actually added a VPC gateway endpoint which identifies and separates our public and private... public and private traffic to S3. So whatever traffic which needs to be go to S3 from our private servers, it will follow through VPC gateway endpoint while others still go through the NAT gateway. This is how we actually fix the problem. The other issue was high cardinality. I'll explain to you what high cardinality is. Prometheus is very much capable of actually tracking millions of metrics, but what happens is that we sometimes miss out on the cardinality issue when we are setting up these metrics alerts. For example, if you want to put a metrics just on a pod name, which is not that dynamic, it's not something which changes every second or every millisecond, then it's fine. But if you're including something which is changing every millisecond or every second, for example, you would not want to actually club your login session with timestamp, or you would not want your request to be clubbed with timestamp, or you would not want to club your request ID with your sessions also because that would lead to a very high cardinality because the values are changing very frequently. Which means Prometheus has to store multiple data for each unique metric it's getting and save it there, which adds to a lot of value. So when we were actually debugging our issue, which was not related to that, we actually identified this could be another problem, which was causing an issue and leading it to a lot of logs aggregation. So the third one was the down sampling. This was a major issue which we faced which was the main cause of the whole scenario where we are here. So if you see, if anyone has used down sampling compactor, they would know that Thanos provides a very basic down sampling compactor where it provides you retention of very specific days and down sampling up to only two levels like five minutes and one hour, which we were using it earlier. The other problem there was whenever we have to down sample any data, the main object store needs to be available. So for example in this case, if a data needs to be down sampled after 40 days to a five minute, the main data needs to be stored for 40 days for sure. And after that, if we want to actually have a down sample to one hour, it would still need the original data to down sample. So you are storing the same data thrice. Also this down sampling over a problem of NAT gateway and high cardinality leads to much bigger issue. So how did we fix it? We actually, we went back and actually wrote our own Thanos compactor which we are planning to open source it in some time. It currently works nicely with the Prometheus but we're also testing it with Victoria Matrix so it should be live soon. So what we did here is we gave freedom to developers on the down sampling fields. For example, now they can down sampling based on their own requirement, not just limited to five minutes of one hour, they can actually choose down sampling to 10 minutes, 30 minutes, even one hour. Also they don't need to keep the original object store. They can actually remove it once the first down sampling is done. For example in our case what we did was we did a 10 minute down sample which needs to be retained for 180 days. Once that is done you can actually get rid of the main object store which we're only keeping it for three days. That also means your down sampling needs to start before three days and you can choose that option in this feature that we can delete the down sampling after three days and also start the down sampling whenever we want. In this case it should start before three days and after that when we have to down sample it for 30 minutes, we can just depend on the previous down sampling data which is the 10 minutes in our case and not based on the original down sample data. This is what we did. The other problem which Shubham mentioned it was not a bigger of an issue but we detected it after we fixed the Thanos down sampling data was fluent with it. Since it's part of the same problem we thought we'll actually discuss it here only. Like most of you must be knowing that fluent with but it does it actually scrapes your application logs and actually push it to an object store. In our case we're pushing it to the cloud watch. But the problem with fluent with in our case was that once we were collecting fluent with data we were using QBAPI to append few more data. In our case it was POD and also the tracing ID which we were appending it to that. We missed the basic part that fluent with does is if QBAPI fails it keeps on trying. It keeps on trying. And as the number of requests comes in initially it will be like 10 requests which will pile up to 100 requests, 1000. So those requests are keep trying to get that data. We did not realize it because the data which we were suffering in Thanos compactor was in TVs this data was GB. That's why this came into light when the TV issue was done this was the second biggest problem for us. And the problem in our case was very simple, it was very naive also that in one of the newer services which we released recently we did not change the sub path amount sub path. So it was working fine on the local but it was not working because that sub path was not even available through QBAPI it was trying to get. So it was costing money. So with proper monitoring we were able to detect this anomalies that I would actually give credit to my team also for detecting it but still it was not as soon as we would have wanted we have done better than now this by actually putting more better alarms and more better monitoring systems but we all know that we don't actually give too much importance to logs how much would that cost we give a lot of importance to our database data that is costing us a lot but logs we always think because they don't actually we don't retain it for so long also and we don't actually cater to that the traffic the network traffic which we are using it right. So we did a calculation based on what the build we also got that what could have it cost it right. So this was a process data which was assuming from the NAT gateway 50 TV and if you actually calculate with hourly cost it will come down to 32 dollars not much but in our case it was processing 51,000 GB per month which will come to $2,304 and if you calculate with both the data it's $2,336 and this we were doing it over 6 NAT gateway since we are in a multi AZ and a multi multiple region environment. Still not a lot but the the logging cost we estimated it comes down to $1,700 based on the Cloudhose because it was in GB but assume all of that in 6,7 clusters so we were running 6,7 cluster of these which accounted it to 100K gladly we did not actually went this far and actually affected but still it cost us a bit to actually learn that lesson and that's why we are here to share it with you guys a few final words for you know we leave you the smallest mistakes hardest to fix and in turn they're just much more expensive. There's this quote from Anthony Hopkins that I really like the evolution of sentient life on this planet was created by one single tool which is a mistake so don't count out single mistakes at all. You don't know what other mistakes this is going to pile on to become second don't rush into things because it might be easy to drop in so we see a lot of chatter in the last year or two about shifting vendors adopting open source solutions and I'm all for that but make sure that you're doing your due diligence you're not using something just because it's easy to drop in and because you know there's a lot of hype around it or they claim to be a silver bullet nothing's ever going to be a silver bullet and lastly if you're a growing build fast break fast team it's often word spending on extensive monitoring especially if that's going to save you much more on saving you from errors like these and that's pretty much it one small shout out to everyone at Kubernetes, Thanos and Flo and Bit who've been tirelessly maintaining so I think there are a few Kubernetes maintainers here today thank you for everything you do and yeah that's all the time thank you everyone for listening towards the end of the day so it's nice to see a full auditorium as well we have a few socials over here if you want to get in touch with us we also have a pretty active Slack community and an SRE meet up community if you want to see more of these talks more of these investigations into site reliability and yeah that's us if you're looking for a cost effective facelift in your insert management systems check out the NUD we're still hiring and yeah we're ready for any questions which you want to share, any mistakes which were made I'm sorry Mimid from Gafana so Mimid wasn't really so this was, this is not a latest story, this was done like a couple of years back at that time it was not that mature enough as I mentioned Mimid was I'm not even sure it was launched at that time if it was it was very very nascent so we did not want to venture so our environment is pretty stable and we generally do thorough testing when we are okay apart from the mistake which we made we do rely on things which are stable we look for the most stable version but at that particular time Thanos seems to be better and still we are continuing to use that it's working really smoothly for us yeah so high cardinality problem like I said right you don't rely on very dynamic metrics so one of the examples which I gave you was not to rely on session IDs if you have to track your user behavior do it on email ID or any user ID which you have other example would stay away from timestamps that is the high volume cardinality problem which we tend to ports are fine, services are fine if you have to do it yeah so once we did the down sampling right we anyways got rid of the main object store so we didn't had to so yeah maybe in future we have to we'll see you had some questions I didn't get him I'm sorry okay so like I said with the tool which we built it's an incident management solution so for us we help companies stay reliable that's what we focus on and the reliability of our tool needs to be better than our customers tools so for that our DR and everything needs to be multi az and multi regional now we have moved to multi cloud also for that particular region we were moving that much data to different region so that if you have to switch the data is always available for monitoring so S3 is anyways a global store right we were moving data from multiple region to S3 we are on AWS yeah so we just moved our some of the workload in S3 that out and see if that is useful but in our experience I'm not trying to advocate AWS here we have been using AWS from last 10 years but we have been able to find the cost and the usage more transparent than azure azure works fine but the support which AWS providers has been more smoother that's why we have taken to AWS for now okay yeah that's a good note keep that in mind pass that feedback to my team yeah in the Thanos compacter which we have provided now you can I'm sorry I didn't get that function of metrics not the days function of metrics not the days so you want to down sample a particular metric and not critical load so in our case what we were doing so you can pick what data you need to down sample and keep it for longer version of the other in our case also we are also discarding some of the data which we don't need for 365 days or even eternity this sample which we have showed you this is only data which needs to be retained yeah and what about you have chosen solution resampling you cannot go back we have tried that in our case it has been able to work because the down sampling which we were doing is based on maximum and average so we were able to reproduce it to some level we were doing it to build some intelligence on top of it if I tell you frankly but real data is real data thank you it was a nice presentation anyone want to share a mistake with them or you can hit us later on also it's love to talk to you about if any of you have similar stories or experiences you'd like to share that we have a pretty active podcast it's called the incidentally reliable podcast we talk about talk to SREs and you know their long careers in history talk about the mistakes the volume stories so you know do check that out and yeah if any of you would like to be on that do reach out we'd love to hear from you and we are continuously hiring for back-end and infragas do hit us up on the career page and love to actually talk to you about that it's on youtube spotify I think it's there on the link as well on the Q&A you can find that over there you want to show it yeah I could share that as well currently we have a few episodes up the last one was with Manoj Sebastian who served at Flipkart and you know that was a treasure trove of stories on how Flipkart like companies like Flipkart at Lash and Yahoo manage their scale so definitely if you are into site reliability stories you know war room nightmares definitely check this out we'd love to hear from you guys as well if any of you would like to be on this do reach out to me any other questions we have a few minutes thank you guys and thank you for giving us the opportunity thank you so much


Hey there friends! How's it going? My name is Kevin and today we're playing Help Wanted. I think it would be good to maybe try and develop some actual skills in case I ever have to go into the real world and get a real job. Oh, what? What is this? Oh, look at that! They're all getting their own little intro.  I want to be part of the family. All right, now that I have been welcomed into the family, let's get started. Haha, just like my brain. There is no data. All my previous bosses have always said, You're fired. So I will just start off the job as fired. Fired. Is this okay? Well, no. But if I'm fired, I'm fired. I don't really get a say in it. But I hold the Wiimote like an 80-year-old who has never seen a television. Never mind the Wii. Why doesn't it work? Let's head to the employment office. Why don't you buy a game and then go, Oh, yes, I get to go to the employment office. And for God's sake, I just arrived at the employment office. And there's sirens and alarms going off. Yeah, they definitely know it's me. But this game is going very extreme. Some meteor just hit the earth. I just wanted to get a job in like a barber shop or something. Sir, I may have taken too much enemies of humankind and attacks on the planet Earth getting you down. We've all been there. Trust me. That's why you need the Transformer watch. What is this game? This is definitely made in Japan. This game is all over the place. There's just a meteor heading towards Earth and now we're back in the house. Okay, Grandpa's giving me an outfit and I need to go get a job to buy the thing of the telemarketing, which will help me stop a meteor which is heading towards Earth. Of course. The meteor's due in six days and I'm just trying to sleep until it gets here, but the game won't let me. Gramps needs help turning on a computer because he's a boomer. Hey there friends, how's it going? Okay, finally, jobs. Let's get to work. What do I want? I did watch a lot of Irish TikTok and I'm pretty hyped to be a farmer, if I'm honest. I'm just generally confused. Here I am to save the world. Wait, what am I doing? What am I doing? Hold on. Oh, Jesus. Oh, sweep of Jesus. Okay. No, no, no, no, no, hold on. Oh, for fuck's sake. That is not reacting in time. That is completely out of sync. Okay, this is having more success. Just freaking out. I love video games.  Yep, worth every penny. All right, out of all those carrots, I think I got five. That's a hard day's work right there. I got my paycheck. It says $8.30 and at the bottom, fired. Understandable, to be honest. That meteor is still coming. My solution to stopping this meteor hitting the earth is to go out picking carrots. All right, let's try Corrier because I don't think I'll be allowed back on that farm unless I wear a fake mustache or something. Wait a second. Before I offer you the job, didn't my farmer friend just fire you yesterday? Oh, no, no, no, that was fired. I am fared. It's European. You got the job fared.  Oh God. Whatever's in here, I hope you don't mind it being soggy. I bet it's just a package of those little dinosaurs that you soak in water and they grow in size. The box will just implode. Hi, doggy. Hi, Macquarie-er now. Oh, yeah, that means you hate me. Sorry, I forgot dogs automatically hate male people.  So that's doubling up on the male. The dog must despise me. I don't know how to get past this hedge maze that they set up for me. Can I fit? Oh, I can fit through here. Okay. Honestly, it would be easier if I just went around this entire field.  There you go. Here's your box of sausages. Oh, and I was two feet away. So sorry, you don't get your package. We're returning it to sender because I didn't make it on time. Honestly, we should just do it like the Irish postal service. Just drop kicking across the field. Or it lands, it lands, it'll be grand.  Mom, it might be time to ease up on the Botox.  Or is that tears? Okay. Well, at least someone's having a reasonable reaction to the fact we're about to die from a meteor. Not to worry. I'm going fishing to get enough money to stop it. Whoa, whoa, whoa, whoa, whoa, whoa. Wait, I'm driving the boat and fishing at the same time. I couldn't even pick carrots. This seems a bit beyond my pay grade. I caught like seven fish or something. So hopefully that's enough to buy the thing I need to stop a meteor. I have three more days to stop it. And all I have to show is like a handful of carrots, two deliveries, and about five fish. Okay. The most I've earned is from the fisher job. So I guess I'll try that again. No one else will hire me for some reason.  I think this is a shark. Let's get him. He is literally faster than my boat. How am I supposed to catch him?  It didn't work and I have ran out of time and I've caught zero fish. I have cost my employer because of all that fuel I used, just spinning around in circles. Are you working again, bro? Oh, can you call it working? He's making no money. Okay, I'm back as a courier. Your package is immediately wet. Imagine you see the courier jumping out of his van and immediately running into the sprinkler. Well, that's my penicillin gun. Come on. He can't even get to me.   I don't even blame him. Come on, come on. I need my Jim Pickens body pillow. There you go, sir. It comes pre-moistened. Okay, the meteor hits tomorrow. Why is no one else worried about this? Why is it up to me and my grandpa who has bolted a light bulb into the back of his head so it seems like he always has good ideas? What is it, Gramps? Well, it turns out you don't actually need to work every day. I mean, I've been living with that philosophy anyway. I said we can watch some TV together. Let's watch tele-shopping. Ah, yes, Japanese TV. Oh my God, what are they doing to them? Okay, so he wants me to buy this watch which will transform me into a giant and give me the ability to stop the meteor. This feels like a scam, I gotta be honest. So I need to buy the watch with points which I get from buying stuff on the tele-shopping network. This is definitely a scam. I'm just waiting for him to say send me Bitcoin and I'll double it. So the only things I can actually buy are these aprons and I don't have enough money and then you need to collect points by buying things and all of these things including the scary mask, delay the meteor. This is the worst scam ever. I'd rather send the money to some random Nigerian prince who's emailed me. That is honestly better odds. Oh, grandpa gave me some money. Great, okay. He had the money all along. Yeah, I'll order that Taylor suit. I'm gonna pick carrots out of the ground and I'm trying to be a Taylor. Okay, I'm ordering the scary mask. Me and my poor grandpa are getting scammed here. Tele-marketing preys on the old and the boomers. And I must say I'm a bit of a boomer myself. So I'll gladly buy into this. So the scary mask will arrive tomorrow and the meteor will also arrive tomorrow and we need the scary mask to delay the meteor. So it entirely depends on the courier service if we all die tomorrow and I've worked for the courier service and I will say they are not reliable. I'm gonna get a soggy piece of paper mache in the mail. Okay, there we go. Hee-hee-hee. I know, spooky, right? Yeah, you get that meteor, Gramps. Okay, it actually works somehow. The meteor got scared of my grandpa and his spooky soggy mask. All right, a new day, a new job. I will become a Taylor and as is tradition with these jobs I will not read the instructions. Just give me someone's wedding dress and I'll get to work. Wait, what? Wait, hold on. I don't know what I'm doing. Hold on, hold on. I can actually get this. I can figure this out. All right, wait. Hold on. There's only so many buttons. Okay, wait, I'm sewing the table to their shirt. Sorry about that. Wait, I don't get paid anything? Like, even if I did a poor job, I still did a job. Surely there's some labor laws here. That's it.  I'm not even attending jobs anymore. I'm just immediately going home and sleeping. I've slept for four days straight now. Perhaps this problem is bigger than me, Grandpa. Maybe I can't stop this meteor, or maybe I'm just not willing to buy into this scam anymore. Oh, there's a sale on the telemarketing network, but no, I'm still too depressed. Not even that can cheer me up. My dream of being a tailor is over before it even started. The meteor is now only seven days away, and it was 21 days away a second ago. I've just been sleeping for two weeks solid. Look, it's not my fault the meteor showed up when I was in hibernation. This is normal for me. Every year I sleep through July, August, and September. As the great philosopher Green Day once said, Wake me up when September ends. I've cooked up something that really boosts your stamina. Great, this will give me the stamina I need to sleep for another five days. Oh, that is getting mighty close. One day left on Earth. What will I spend the last day of my life doing? I think sleeping. Are you two on track to take care of that thing? Yeah, I've not been sleeping for two and a half weeks. Okay, Gramps. You really hammered home. There's a sense of urgency here. But honestly, I'm really tired. We'll just see how this whole thing plays out. You see, it looks like a beautiful day. The sun is shining. The birds are chirping. No meteor in sight. Oh, um... That's not the sun. That's the meteor. Wait, Gramps found one of those watches on the street. Okay, I think that's just a latex suit. I don't want to see my grandpa in a skin tight latex suit. Oh, and he must have got it at a half off sale. Okay, Grandpa just flew into the sky and tried punching the meteor. And yeah, that didn't work at all. He's like, I'm taking half your money and points for building repairs. I have none to begin with. Half of nothing is still nothing, Grandpa. He's extended it two weeks. All right, that's it. You know, at the start, I was thinking Grandpa was getting taken advantage of by the telemarketing people. But now I'm thinking Grandpa's in on it. That light bulb isn't lighting up because he's an idea to stop the meteor. It's lighting up because he has an idea to scam his grandchildren. I didn't look into this game much before I played it, but given that the title was Help Wanted 50 Wacky Jobs, it's very different than I expected. I just really didn't expect to see my grandpa half naked and the other half of him covered in a skin tight latex suit and me with tape wrapped all around my head. Why is this happening? I don't understand. I guess I'll just suffocate here then. I mean, it doesn't really matter. I wasn't doing anything with my life. I was just sleeping. Yeah, maybe it's best I just stay as a burden on the taxpayer because I'm not caught out to work. I think I'm just gonna sleep for the rest of my life. It's apparently gonna be two weeks until the scam gets extended, I guess. But we're gonna leave it there. I hope you enjoyed me trying out these jobs. I don't think I'm fit for the real world, so I'm gonna continue making videos. I'm gonna continue posting every day as well as streaming over on Twitch. The link for that will be in the description. I hope you enjoyed. As I said, I appreciate you watching and I hope to see you next time. Bye for now.


Recently, we have been giving Lenovo a lot of attention, and it's totally deserved. The company makes good devices at all price points and while there have been some stumbles, looking at UIDiapad gaming 3i, most of their offerings have been great. Today we have the ThinkPad L14 Gen 3, a mid-range office device that excels at its job. Once again, Thinkpads offer multiple configurations for the build, with either aluminum or plastic for the lid and plastic or carbon fiber for the bottom portion. In terms of design, the laptop is alright, looking like every other ThinkPad under the sun, while the durability with the plastic base and aluminum lid is good. In terms of weight, the laptop measures 1.39 kg and has a profile of 19.8 mm. There is a keyboard, a touchpad, and a track point, which are executed with varying levels of success. The board has long key travel, clicky feedback, a backlight, and spill resistance, making it ready for everything. The pad is smooth and accurate. The device is an input-output champ, offering two USB Type-A 3.2 Generation 1, a Thunderbolt 4 port, and a USB Type-C 3.2 Generation 1 port. There's also a LAN port, HDMI 2.1 with 8K 60Hz output, audio jack, micro SD card reader, smart card reader, and a SIM card slot. The display isn't suitable for professional work with colors, but it is great for office work. The FHD IPS panel has excellent viewing angles, peaks at 262 nits in terms of brightness, and has a very high 1470 to 1 contrast ratio. With 53% sRGB coverage and a Delta E value of 3.6 with our profile applied, it's nowhere near creator ready. Thankfully, there's no PWM usage, which is pleasant to see. There's both a 42Wh and a 57Wh battery pack, with us choosing the larger option. It lasts for 11 hours and 15 minutes of web browsing, or 10 hours and 18 minutes of video playback when paired with the Core i5-1235U. Compared to other laptops with the Core i5-1235U, the ThinkPad delivers significantly less performance. The Dell Latitude 5430 and the MSI Modern 14 come with better performing hardware, both in terms of the CPU and the GPU, as the integrated graphics also score a lot higher in benchmarks. The cooling is very simple, with two heat pipes, one fan, and a heat sink. The stress test shows that the CPU maintains high clock speeds for a short while, dropping to below 2 GHz at the 15 minute mark. As for comfort, with only one fan, the laptop works quietly, while the base won't burn your fingertips. Lenovo managed to bring great upgradeability to a very portable laptop, with two Sodom slots that can house up to 64GB of DDR4 memory. For storage, there's a single M.2 slot that supports Generation 4 drives. To learn how to access them, you can watch our teardown video, which shows a step by step process. While the laptop gets outperformed when put against its competitors, it gets so much other stuff right, like the rich input output, plenty of upgradeability, and long battery life. Business devices have been made with security and stability in mind, so even if there is a performance deficit, we're sure that many people will be able to overlook it, in order to access the rest of the pleasantries that come with the ThinkPad. Of course, we've done a lot more tests and you can check out all the results in our in-death review on laptopmedia.com. You'll find the link in the description below the video.