Hi and welcome back to program analysis. So this is part four of this lecture on operational semantics and What we'll do in this fourth part is to now look at another way to define the semantics of the language which is actually the most common way perhaps to Define the semantics of a language and these are structural operational semantics We look at two flavors of those and we'll start here in this video with the small step semantics Which is the first of these two flavors? So now if you've seen the previous video then you've seen already one way of defining the semantics of our language and this was this abstract machine So basically a hypothetical computer just built to execute simp programs Now this is one valid way of doing it But as you've also seen the abstract machine was actually pretty detailed. So a lot of the Rules were not really spent on doing actual computation But on something else and in particular there were quite a few transitions that are part of the abstract machine That are basically only about manipulating the syntax of the language so for example, there were some rules that were just about Taking a command and then decomposing it into its components in order to then reason about the meaning of these components So what we now want to do instead is to focus on a way to define the semantics where every transition is Actually a computational step and one benefit of doing this of course is that you can write down the semantics of a program with fewer transitions because you only have to write down the actual computational steps and we'll do this in two styles Where one and that's the one that we are going to look at in this video It's called small step semantics and this name implies the steps that we're taking here in each transition are relatively small Sometimes this is also called reduction semantics. So this is just in case you See this word somewhere else so you know that you now know about it and then the second style of Structural operational semantics that we look here look at here will be Dix big step semantics Which is sometimes also called evaluation semantics All right So let's get started with the small step semantics and again, we will define the semantics as a transition system So again, we need to define both the configurations that this transition system is about and the transition Relations that tell us how to get from one of those configurations to another So we here will have a transition system with a configuration And in our case this configuration now consists only of a pair So it's not the triple as before but just the pair where we have the program itself And then the so-called store which you can think of as something very similar to the memory of the abstract machine So P is the program and s is the store Which actually is a function from locations to integers so similar to the memory Now the configuration is easy to define now the slightly harder part which will take a few more minutes is to actually define the transition rules or to define this transition relation that tells us how to get from one configuration to the other and In line with the name reduction semantics that I've mentioned before Sometimes these relations are also called reduction relations and the way We'll define these relations is by defining axioms and rules that tell you when you can actually Conclude that you can get from one configuration to another Similar to what we've done for the abstract machine Let's start by defining the axioms and rules for expressions in our simple language And let's start with the most simple of all of these axioms Which is an axiom that tells us how to actually read the value of a variable From the memory So in this case, we again will use this Horizontal bar notation that we've seen in the first video of this lecture Where we do not have anything on top of the bar because this is an axiom so we do not have any hypothesis and then At the bottom We have the conclusion we can draw which is that if we have a program where we say read the location the value at location L and we have a given store then we can reduce this to a value n and the same store again if at Location L We actually have this value n and I'm going to give each of these rules a name So this is a rule called bar for variable Because this is essentially reading the value of a variable and please note that Reading a variable from the store does not really change the store itself because we're not writing anything But just reading something So there's no change of the store s itself The next axiom will be for evaluating Operations so once we've read some values from memory perhaps we want to compute something using them and To do this we have this axiom which says if our program is some value and one followed by an Operator op and then some value and two and we have some Store s then we can reduce this to some value n and the same store s if and is the result of applying this operator and to these two values and One and at two and this rule is called the op rule because it's for the Operator, so now you may wonder how do we actually get to the state where we have these values In our program well in order to do this we may have to evaluate some more complex expressions and to do this we'll have two Rules here one for evaluating the left-hand side and one for evaluating the right-hand side of a complex expression So let's say we have an expression that looks like this So it's some expression e1 op and then some expression e2 Which given a store we would like to reduce Then we can reduce e1 to e1 prime Which for example could be just a value that this expression evaluates to and we leave e2 the way it is And we may actually update Our store while doing this And now this is not a rule because we can only do this under specific conditions and in this case the condition Is that there is some way to take this expression e1? Evaluated given the store s and then obtain e1 prime and store s prime So this rule is what I call op l for left-hand side of an operation note again that the Store can be updated here So we may go from s to s prime in the simple version of simp that we are using here in the lecture This can actually not happen. So s prime is always guaranteed to be the same as s But in a slightly more general case where we may have expressions with side effects the store may actually be updated Now for the right-hand side of an expression. We can have a very similar Rule here, which now says if you already have evaluated the left-hand side So we have some value and one here followed by op and then some expression e2 Then we can reduce this to and one op e2 prime with a potentially updated store if There is a way to get from e2 and store s to e2 prime and store s prime and analogous to The op left name for the previous rule This is going to be called op r for the right-hand side of the operation So now you've seen these rules and axioms for defining the semantics of expressions in simp if you compare them to the Transition rules that we have defined for the abstract machine you'll see that they are a little bit simpler and you'll need fewer rules to actually evaluate an expression simply because we do not have these syntactic manipulations but actually always perform a computation whenever we are applying one of these transition rules and Now next we want to also look into the rules and axioms that we'll need to define the semantics of commands in simp So let's get started with an axiom that tells us how to handle assignments. So if we have An assignment that looks like this So we say that we want to write into some location the value n and we do this while having some initial store s Then we can reduce this to The following where our program afterwards just says skip. So we're basically reducing away this assignment and In turn, we are updating our store by now saying that the store is the same as before But now l maps to n So this rule is called the assignment rule and essentially what it is doing here is to update The piece of memory at location l Which is exactly what you would expect an assignment to do Now in order to get into a situation where you can actually use this assignment rule We may have to first evaluate the expression that is on the right-hand side of this assignment So we will have a situation that looks like this where our program is an Assignment into location l and the right-hand side is some complex expression e Now what should we do here? So this is what I'm going to show in this rule so we can conclude that this can be reduced to L gets assigned e prime with a possibly updated store s prime if There is a transition that goes from e and store s to e prime and the updated store s prime and this rule is called the right-hand side of assignment rule in the first rule you've seen that Evaluating this or executing this assignment that leads to the skip command So let's just have a look at what we can do with this skip command in order to get rid of it We are defining this axiom, which is basically saying if you have skip followed by some other command C and We see this using some store s then we can just remove the skip and keep the Remaining command C without really changing the store s so this rule is called skip and then another kind of Boring rule is for chaining two commands. So if you have Something that looks like this some command C1 semicolon C2 and we are now executing this given some store s then if we can say that C1 executed in s leads to C1 prime which updates the store s to s prime then we can execute This entire sequence of commands by replacing C1 by C1 prime followed by C2 and Updating this store s so essentially that gives us a way to execute the first out of two commands for example using the assignment Rule that we've seen above and then eventually once this is reduced to skip we can get rid of it using the skip rule and This last rule here is called a sequence rule because it's handling sequences of commands So now that two commands left in our simple language that we still need to define the semantics for and these are if and While commands so let's get started with if and then we'll actually see a quite elegant trick to define while commands The very little effort. So if you have an if There are again these cases where we have true and false in our condition. So let's start with the true case where We have something like if true Then C1 else C2 and we want to Reduce this given some stores Then we can reduce this by simply saying oh now we execute C1 and forget about all the rest and Distort stays the way it is. So this is the If rule for the true case and Then very similarly we have also the same for the false case So if you have an if and we already know that the boolean condition evaluates to false Then this is reduced to just taking the false branch. I'm sorry the DL's branch C2 and And then in the next step afterwards you can evaluate that command C2 So this is the if rule for the false branch So now you may wonder how do we actually get to this if true or if false and The answer is that we of course first have to evaluate the boolean expression. So let's have a look at how we do this So if you have an arbitrary if statement that does not yet have its boolean expression evaluated It's something like this if B then C1 else C2 And let's say we are Reasoning about this given an initial store s that we need to have Some way to reduce this boolean expression where we take the boolean expression and the initial store and if we are able to reduce it to some B prime which for example could be true or false and In general this may have an effect on the store. So afterwards it may be S prime then we can also Replace this boolean expression in our entire if command So instead of B we now here have B prime for example true or false and the rest stays the same And of course because we have changed potentially changed the store into S prime We also need to change it here So this rule is called the if rule And now finally we also need to define the semantics of wild statements And we'll do this in a way that reuses what we have said about if statements above as follows so if you have a wild command like this wild some boolean expression B do C And this is going to be evaluated with store S Then we actually can rewrite this into an if statement which contains a while loop inside where we just say if B so if this boolean expression is true then First execute the body of the loop and then we sequence this with another command Namely a wild command again where we have the same while B do C loop So we're essentially saying Well, what the semantics of a while loop actually is so if the boolean condition is true then evaluate The body of the loop once and then go again to the beginning of the loop and see what happens next And if the boolean expression is not true, then we are not going to do anything So we just replace or we just put here as dl branch our skip command and this is in a sense just a syntactic trick But it saves us the effort of writing down rules that most specifically say what a while loops are about because we can express it Very simple in a very simple way just like this Let's make all of this a bit more concrete by looking at an example And this example is a program that we've seen earlier. It's a program that basically swaps two variables the values of two variables using some temporary variable. So we have this variable z Which will get the value of x And then we have two more commands here one that writes the value of y into x And then one that takes the temporary the value of the temporary variable z and writes It's yeah, this value into y and now let's say we want to evaluate this program Given an initial store where z is zero x is one And y is two Then in order to do this we need to find a sequence of transitions where we start with p and s and then Apply multiple transitions go through multiple configurations and eventually want to arrive at only the skip command And then we'll also see it updated store and because you Um understand what this program is about you'll also understand that in the updated store z will have the value one And the values of x and y are swapped. So x will have the value two And y will have the value one So now you may wonder well, what's actually in this dot dot dot because that in a sense is the most interesting part of here so each transition each step that we are taking here needs to be um Yeah shown to be valid using either an axiom that we are using Or by using some rule and in order to um Use one of these rules we'll actually Need to show the proof tree Which basically shows that the preconditions the hypothesis of the rule are also true for which we may need to use other rules Or maybe need to use an accept. Yes, sorry an axiom Now let's look at an Accept and that's where the word already came from Um of this proof tree And this will be the very first Um step that we are going to take here. So on this very first step Um, we look at the entire old program. So this will just be p as we have it given initially and s And this can be reduced into a program Where we are reading the value of x And we'll see that this is one And then the rest of the program Is just the same as before And the store doesn't change here. So this is the sequence Rule that we are using here But now in order to use this rule, we of course have to so if you go back Um to where this was defined. So if you want to use this rule, um, we also need to show something So we need to show that we can actually take the first command and reduce it the way we want to do it here And in order to do this for this example, what we'll need to do is to basically Reason about the right hand side of this assignment. So we somehow want to show that if we have z Getting the value of x With store s, then you can reduce this to z Getting the value one with store s And in order to do this, we are using the Rule for right hand sides of an assignment and now Going back to that rule So this is this one You see that in order to do this, we need to have something that tells us How to evaluate the expression on the right hand side to an expression e prime Which in this case means we need to have a rule that tells us that if you look up x in the store We'll get the value one and actually we do have a rule for this Which is saying well if you have a lookup of x given This store s that you see above Then this actually yields one and the store doesn't change and the rule that we can use here Or to be more precise the axiom is the one that we have looked at at the very beginning Which was the one for looking up the value of a variable Now this is just the very first step in this sequence of transitions and for each of these Transitions, we either need to just use an axiom Which is the simple case or need to show that we can actually use a rule and if we have To use a rule that we need to provide a proof tree like this to show that we can actually Take this transition So this sequence of transitions is called the evaluation sequence and let's have a look at What properties this evaluation sequence could have and what this actually means for the program that we are reasoning about here, so given a configuration ps We say that the evaluation sequence Is a uniquely defined sequence of transitions So uniquely defined here means that there's one only one way of actually Showing this transition or the sequence and this sequence of transition starts with this configuration p comma s and it has maximal length and What this means Is that we've applied all transitions that are possibly applicable using the axioms and rules that we've just defined, so we do not stop at some point When we still could continue, but instead we add transitions and perform transitions as long as we can And now if we do this there are three possible outcomes of the program the first possible outcome is that this Evaluation sequence turns out to be infinite And what this essentially means is that this program has for example in an infinite loop Or maybe if we had a language that also has functions We could also have a recursive function call that continues forever but in any case the term we'll use here is that the program is divergent Which is just um, yeah another way of saying that it's going to run forever Then the second possible outcome is that we have a finite sequence that Eventually reaches One of the good states that tell us about the final configuration or the result of this program So in this case, we have a finite sequence that reaches either a configuration n comma s so where we have some integer value or configuration b comma s where we have some boolean value or the configuration Skip comma s where we do not have any value, but we also don't have anything left to do And in this case we say that the program Terminates And we say that the final configuration that we get so one of these three configurations that you've just seen Is the result of the program And then there's a third case Which is that we do have a finite sequence But the sequence Stops at some other configuration. So it's not n comma s b comma s or skip comma s But some other configuration for example one where we try to still execute some command But for some reason we can't And in this case we are saying that the program is blocked Which essentially means that the program just can't continue to execute and we can't really say What the result of this program is because it's in some sense or another an invalid program So let's have a look at some examples for these three cases so that these three concepts of divergent Terminating and blocked programs become a bit more clear so An example for the first case where the program is diverging would be the following Let's say we have a program that says while true Do skip And now let's say we have some store s and want to Evaluate this program and reason about its semantics. Well, then what we'll do is we'll first use our rule for For while which says We replace this with an if followed by the boolean condition in this case true Then We take the loop body once so skip followed by the entire loop again so again while true Do skip And then we also an else branch where we put skip And now if we reduce this further in this case now by using our if true rule then we can reduce this to the if branch of this Yeah, the then branch of this if statement, which in this case would be skip followed by Our loop again. So while true Do skip And then as you can see this goes on and on and on and on so we can basically Continue forever applying our transition rules, but we will never terminate simply because this program has this infinite loop Because it says while true do something So this is an example of a program that we can Show to be divergent As a second example, let's look at this Case where the program is terminating. So that's the Typical behavior of a program that you would like to see because this means the program Finishes at some point and it has a result. So for example, let's say we have a sim program That says if four equals zero Then skip else skip So not a particularly useful program, but one that is short enough to just show here as an example So in this case, um, we will use one of our rules to actually show that this boolean expression evaluates to false So we will have if false Then skip at skip Which doesn't change the store and now we can use another rule which Is about the if false case Actually, that's an axiom Which means we can replace this if statement with the um else branch which in this case is skip Which leads us to skip and s and this was one of the possible um Result states that tell us that our program has actually terminated. So this second example Is an example of a program that is terminating And then finally, let's have a look at the third case. So a program that is blocked So there are different reasons why this could happen But one is what i want to show here where we have And if that says if the value of x is larger than zero Then perform some command c which for the example doesn't really matter. So let's just call it c as skip And now we are evaluating this given some store s Where actually this value x is not in the domain of our store s Which means the store doesn't really know what the value of x is because this label is not mapped to anything And then if you have um a program like this where we basically try to read some memory that is not defined in our store Then we can't really evaluate this program any further and this means this means our semantics Here are blocked and we can't really tell what the meaning of this program is All right, and that's already the end of Video number four in this lecture on operational semantics. So you've now seen Small step semantics, which is one of the two styles of structural operational semantics that we're looking at here And as you've seen you can use axioms and rules that tell you when to transition from one configuration to another To determine what the semantics of a program are and whether the program terminates or maybe is blocked or maybe Is divergent. Thank you very much for listening and see you next time


with Transitional Justice. I'm Jay Fidel. This is TIG Tech on a given Monday afternoon and we're talking today about violence in Colombia with Nicholas Sussman, a regular contributor who lives there in Bogota. So Nicholas, welcome back to the show. Thank you Jay. Good night to all the people who is watching us and it's always a pleasure to be with you. It's great to have you. So a lot of people weren't aware but there has been violence and sort of an unraveling in Colombia which as you reported earlier was doing pretty well but maybe not now. What is happening in Colombia these days? Right. So right now we've been after more than a month of a national strike that has turned very violent. Again as demonstrations against what people consider a very unfair tax reform and it was answered the government responded with disproportionate police violence with lack of dispossession to dialogue and so on. So it then became a national strike against everything that the people think is wrong about the government and it has been met again with more police violence and a lot of situations where not only the police is participating in violence against the demonstrators but also we're seeing a private individual civilians that are arming themselves to attack the demonstrators who have been mostly specific and never as violent as the violence they have received from governmental forces. Gee, so it sounds like the right and the left. The right being the government and the left being the protesters and of course there will be people who are aligned with the right and the government and so you do have two sides in the streets. How did it start? You're talking about a labor movement of some kind and how has it gone since then in terms of the escalation of violence? Right. So originally this was a strike like more than a strike demonstrations called by something that has named itself like the national strike committee. This national strike committee had been created for some years ago and led the previous national strike that took place in 2019. This committee has some labor unions, some labor movements, some educators, students and so on and they called for those demonstrations against a very unfair tax reform. So they are the ones who organized it at first but now it became a whole country, whole population thing. Even there are parts and big parts and important parts of the population who say that this committee does not represent them and they are calling more for popular assemblies and the regions where the people can gather the ideas that they need to take to the government to solve it. So these were the people who organized it at first but it has become bigger and I would say that at this point this national strike committee does not represent the whole of the demonstrations and just the precision even if this national strike committee could be aligned with the left. Nowadays I wouldn't describe the demonstrations or the strike as a right left thing to accept that the government is a far right government but the population at this point is just against some of the measures regardless of their political opinions because some of the measures are just going against the people. Even many of the people that voted for the government or who do not feel aligned to the left and actually that is one of the strategies that government has used to delegitimize the protest saying that it is the left or that they are being used and guided by a left candidate or even by the dissonances of the guerrillas which is not the case. It sounds like chaos Nicholas. You don't know who is for what. This is not a good thing and it sounds like it also started out with one issue and gathered other issues and before you notice everything. Do you agree with that and can you explain it? Yeah, I agree. Originally it was about a tax reform which was very aggressive which was rather unfair that wanted to tax things that should not be taxed because people livelihoods depend on that so food, basic food, public utilities and so on that is the first thing. This was met with unjustified government expenses like the purchase of military equipment which is more fit for state-to-state conflict instead of law enforcement operations which should be what the government is facing right now because our biggest armed conflict ended with a peace agreement. So this became about this and then when the government responded with violence it became about everything else starting with the police violence during the demonstrations but going back to protests in the past and then about everything else because the government where a lot of people hasn't done anything right in a lot of aspects and keeps blaming the previous government even after three years in office. So it's just a lot of disappointment and anger against many of the policies of the government. Oh that sounds like chaos. So how this is happening in various cities, the article I pulled up is reporting that there's violence in Kali, C-A-L-I, but is it happening in Bogota also? Is it happening all over the country? Right so I must say that right now the intensity of the situation has decreased a bit and then you have still some focuses of violence, some places where you have violence, one of them is Kali, Kali has has not decreased at all and some parts in Bogota and some parts in the rest of the country but at the beginning it was everywhere everywhere and right now I would say that Bogota in some places is still an important point but the main point is Kali. Kali is a city in the southwest of Colombia going to the Pacific and it has a lot of social complexities that make it prone to this violence and that makes it prone to division as well. When you say social complexities and division, you mean you know diverse population, do you mean disparity in income? Do you mean racial strife? What do you mean by that? Yeah basically those two things income and racial that are actually intertwined. Kali originally since the beginning of of Colombia even the colonies was a place where a lot of sugar plantations were there, you had a lot of slavery and you also had land owners, landowners were wide coming directly from the Spanish, conquerors and colonizers and you have black people who come from slaves and then you have a mixture of people and this division has gone has gone on and on and on and the other complexities that Kali has also been strongly affected by drug dealing businesses. In the worst time of drug dealing in Colombia you had two cartels, the cartel of Medellin which is the most renowned one and you had the cartel from Kali which was the second one and Kali has been the center of violence because of its closeness to the Pacific which is one of the favorite routes to take out drugs of the country so you have a mixture of race conflicts, class conflicts, drug dealing, violence from all groups because it also had a strong presence of guerrilla so it's has been very complicated and you have a deeply divided society because of all that. Okay so yeah let's talk about COVID and the pandemic, how has that affected this? Has it been an exacerbating factor? Has it launched additional violence somehow directly or indirectly? I said that it added up to the increase in violence and to the dissatisfaction of people. As I was saying in early before COVID there was another strike in 2019 it was a very strong strike where we saw a lot of the things that we are seeing nowadays not as strong, not as long but still you could see like a bit of foreshadowing in those demonstrations in that strike and then you had Christmas 2019 this helped the demonstrations didn't continue, people went to Christmas to the end of the year and so on and then in the beginning of 2020 COVID started and then you had a pandemic but the issues were still there. Actually the plan from the National Strike Committee was to continue the strike in January but then COVID came, lockdowns came, COVID hit everyone differently because remote work is possible for some people but it's not possible for other people who work in factories or who work in services and usually the people who have more challenges doing remote work are people with low income so that was a factor to the satisfaction and the other factor was that there was no strong social support from the government to these people so a lot of people lost their jobs, a lot of people had to start using their savings to survive or to go out even breaking lockdowns and whatever because they needed to eat. So this added to the dissatisfaction also it was the satisfaction that could be expressed during more than a year because people were in lockdowns and also lockdowns helped the government to be more authoritarian because if people cannot go to the streets, the government is using emergency powers or whatever related figure, they do all of things that would have more control, the Congress was not sessioning in person so there's another lack of control you have so of course COVID helped to these and I think that they thought that they were going to get away with their tax reform because of that but it was too much for the people and they didn't care they just went to the streets even in the middle of a COVID peak because we have an outbreak since early year and they just went out. So you're in an outbreak now? Sorry? You're in an outbreak now? We are in an outbreak yeah we are in an outbreak but the people actually what many of the demonstrators said was either we're going to die from COVID or we're going to die from hunger. This is the situation so we have to go to the streets which is very strong but is the reality of a lot of people's health? Well you know there was an article in the paper this morning I think about how some people in the U.S. where they've been locked down for a year and now people were supposed to go back to work in certain jobs and they didn't want to go back and what you know what that tells you is that on the way in COVID disrupts societies and on the way out COVID disrupts societies whatever is wrong will be you know emphasized and exacerbate other effects so I think there's a parallel to be drawn somehow between what has happened in Colombia and what has happened maybe in different ways but nevertheless a process that has happened elsewhere. So how how deep is the violence I mean are people being what weapons are being used are people being injured killed taken to the hospital what level of violence do we have in Colombia? Well during the demonstrations and I must say it mostly by the government without acknowledging that there has been some violence and serious violence from the from some of the demonstrators and I wouldn't speak about the demonstrations as the demonstrators as a whole because that's where there's a key difference that the government is using in their favor not telling the people about that but the people respond individually government responses are collective and the violence is terrible you have people killed you have people tortured you have sexual violence you have injury you have non-conventional weapons or you have crowd control weapons used in an inappropriate manner so it's terrible it's terrible the things that we are seeing are for example live rounds being fired or gas canisters or tear gas canisters being fired but not to the to the sky but rather in a direct way or even used gas canisters which that have already expired so you cannot guarantee the quality and the and the security of their of their of their of the material you're thrown to them they have even found I don't know shrapnel of all sorts being used just regular violence but the beatings have been terrible you have forced disappearances as well of people who are not even in the demonstrations were just walking by and the police just grabbed them because they are around and takes them and no one knows where you have violence against human rights defenders against government officials from from the local level we're trying to to de-escalate the situation so the violence is really really bad right now the numbers are very high there has been reports from Human Rights Watch from Amnesty International the UN has called several times for intervention the Inter-American Commission did a visit like it has been really really bad in the forms of violence that are taking place right now so where does the government stand on this they do they want to quell the violence what position did they take what are what are their options and which options are they are they choosing right so that's been I don't know it's strange because now that you know the government you know that this is their modus operandi but it has not been very coherent right so you have them speaking publicly about they they went wanting to dialogue but also condemning only the vandals no mention about police brutality human rights violations accountability and so on and something very strange that happened is that during today you had dialogues you had peaceful demonstrations uh you had one enforcement mostly compliant with the law and then at night after night full abuses took place in a big scale and all the poor disappearances the key links the paramilitary groups taking uh collaborating with the police everything took place here and there has not been a strong condemnation with the government the government has been evasive to accountability they even denied at first the entrance of the Inter-American Commission but not in a straightforward way because that would put them close to Cuba and Venezuela which is what they tried the most not to be in their speech but they were like we will let you in after this and this happens we will let you in after some months we will let you in under certain conditions which is know how human how human right organs work also that has been the situation there has been some dialogues but the dialogues have not been very fruitful uh they still condemn the vandals and speak about the vandals all the time and that just creates more and more violence and they also buck up all the time the police uh in this place so so it's it's very complicated and that's where they stand right now very very troubling so what I get out of this is that once you you pull the plug on the rule of law once you pull the plug on social order and the social compact if you will where people sort of have an under underlying agreement not to violate the space and and the and the rights of others once you pull the plug on that it's chaos and it goes in any direction it wants it's hard to predict where it'll go do you agree with me and can you predict where this is going to go yeah I totally agree Jay and if we go back to the past two shows that we did you could see things like that coming because you see the police being indoors you saw a whole situation of fragmented and widespread human rights violations taking place and like they were pushing the boundary every time a bit more further and further and further and further and now when when the time bomb blows which is the situation we're facing right now they feel encouraged they feel strong and now the institutions that do not work as efficiently as you as you would consider um as where is it going I don't know I don't know I don't think we will we're heading into a dictatorship or things like that even if that is the opinion of many people I talked to my Venezuelan friends who live abroad now a day and they just feel saddened about what's going on because they say you never think this is going to happen to you and when we saw this in our country we thought that what happened afterwards didn't happen so that that is something we should be careful about this is a pre-election year next year we're going to have president congress elections and I really hope that they're going to go right there's nothing that indicates that they want to stay in power or they're going to subvert the rule of law to that point but something needs to change and and and with needs to change I don't say that they should like the people in Colombia should choose a left government I think they should just choose someone who respects the rule of law and who actually will comply with the constitution if it is right center right even far right you can have far right governments that respect the rule of law but what we need to ensure is that anything that is what is coming the elections and I didn't think that the alternatives are very clear right now there's a lot of division in the opposition and there's a lot of division in the right as well because everyone wants to chip in with the government but doesn't want to be with the government because of this it's not very popular so that is the situation and I don't know where it's going right now well what about you where are you going it strikes me that if you walked out on the street you couldn't be nearly as sure of your own security as you might have been earlier um it strikes me that um you know if you had the opportunity to leave because it got intolerable you wouldn't have a plan and you would leave uh are people leaving right uh so that is a complexity that the Colombian situation has always had that even if you have phases of the country where you have full outbreaks of conflict they don't hit as hard uh those of us who fortunately are privileged so I cannot say that while I was there right now I'm in the States traveling but I just traveling I'll go I'm going back in a couple of weeks but I wouldn't say that my safety was as affected when I when the strike was taking place I just didn't go to the streets because I know that if I was caught in the middle of a demonstration it would be gasped or something like that so I don't to do it but it's not that they're going to come into my house and take me right but that is the thing but the situation is very saddening there's not a lot of future uh I didn't know what will happen after next year but of course it is it is complicated and for me as a human rights lawyer it is also concerning how this is going to go uh after four years but I'm not even close to the to the level of threat and danger that social leaders are or who actual human rights defenders who are on the ground are risky but but but I speak publicly about what we do why I speak about the the human rights abuses and well today I am safe but if they keep uh in the government and if they stay there um and they don't have a lot of pressure of international pressure what really makes them compliant uh it can be dangerous uh so maybe living just in terms of life quality would be an option but if things keep getting worse uh maybe leaving us a necessity just to guarantee and to anticipate uh to the situations might might be an option as it happened to actually the people in Venezuela those who left when they could and are safe and then others couldn't leave or had to leave everything behind in order to to guarantee your safety so so you don't know but it is scary. Nicholas how's the press doing how's freedom of speech doing uh are the press uh free to discuss these things? I think the press is free the press is free uh you have abuses on the ground against some journalists but but not as a as a systematic persecution against the press well just something of the moment right uh and you have a lot of alternative media covering the situation of course they feel afraid of course um they have been subject to some violence uh but I wouldn't say that there is a attack that concerning again freedom of press and that has been one of the good things actually that the whole documentation of of the strike has been done by by alternative press and they have been a very hey they have done a very thorough job that they could hand into the inter-american commission actually so that is good having having freedom of press and having the judiciary safe I think are two things that we can rescue from the situation and that will guarantee in the future that the situation keeps within certain boundaries even if it's escalated as it is right now. Yeah guardrails so to speak so here's a big question we should spend a little time on you know you can you're familiar with the way things go in the United States in general and of course you get the media from the United States and you have some training in the United States so what can the United States learn I mean we we had an insurrection on January 6th and we have people that elected office who deny the fact of that insurrection or who lie about it we have people who refuse to investigate that insurrection I mean we know from project expedite justice that you have to know the true facts if you're going to deal with human rights violations and atrocities and violations of the rule of law you have to know and you can't know without an investigation or a commission anyway so my you must have reactions as to what the United States can learn from how this is devolving how it has devolved in Colombia. Yeah I think the main lesson is that when you're talking about the rule of law there is no threat too small and the timely action is everything right uh when this president that we have nowadays began his his candidacy to the presidency he had like four percent right and no one knew who he was we have discussed this he held some positions of international level and then he was elected to the congress not by his own votes but in a closed list system where the head of the list just gained a lot of votes that he then shared with the rest of the list and and got him some representative so this is how he got to congress and no one thought he was going to be elected because he went so low but then his mentor had a lot of importance it is a former president who was liked by a lot of people they have the media on their side they have corporate groups on their side and they're very good at propaganda including fake news and so on and conspiracy theories and so on and when we noticed that these was a problem because I would say that any policy against rule of law is a problem I think it was too late it was too late there was not enough time to counter the effects of fake news there was no enough time to build a strong coalition and candidacy against this type of project and now three years later we are seeing the consequences so we think there is no threat to small you should take them seriously you should investigate and you should bring accountability because accountability also brings a series of consequences that not only impact the individuals that participated in the crimes but also affects the movements but it is not the same to have a leader who is investigated to have a leader that is convicted and can no longer participate in politics for example so we think that is that is very important and to have strong strong speeches and strong rule of law information going around to the people because if the people receive fake information and they don't have true information to compare it with they are just going to stay with fake confirmation that makes them feel safe or scared with the promise of safety so that is very important I would say that. Now the other thing is you know we have had street scenes we had street scenes last year and you could say it stemmed out of Black Lives Matter but it went beyond that in a number of American cities and it runs a kind of parallel where you start a protest on one issue and before you know it people take that as license to protest against everything and the police take the protest as license to use violence and force including illegal force on the protesters and before you know it you have a melee in the streets and I gather that you know to some extent that's what you've been talking about here in Colombia so it seems to me that we've already had a smattering of that in the United States and we have the elements of what you've been talking about not all exactly the same but nevertheless elements there's parallels that we can draw from a social psychological point of view so my question is you know you you do know both societies and I wonder what your your thought is about whether what has happened in Colombia and happened in the United States? Well I didn't know I didn't know but I I think there are a lot of things one can learn from each other the first one is that the society as a whole anything that is one believe that both societies need to overcome need to understand that peaceful demonstration assemblies and even protests even if they are disruptive because a protest needs to be disruptive to be effective are okay and are part of democracy if we win that battle as a society and start as a society surrounding the protesters and the demonstrations because many of us will not go to the streets because a series of reasons I think we've won that that's the first thing if you protect the demonstrators and the society condemns violence against the demonstration that is the first thing the second thing I need I consider we need to learn is what comes after the demonstrations because you're yelling for the government to listen to you and if you're disruptive or effective enough they will listen to you but when these things get get out of hand is when you come to the table and then you don't know what you're going to say because you know you're not in favor of what's going on but there needs to be complete organization and coordination beforehand or maybe during the demonstrations to bring strong points to this table I think that's that's the second thing that is very important uh and the third one is that you need to teach your law enforcement about human rights and their limits and I know it flows because I know people from the military and as most of the people in the military in my country in the state they're good people people who actually want to serve their country I don't doubt that but what they're being said is not right and they're being put in danger and then when accountability procedures come the ones who end up in jail or convicted or the men and women serve or the institution taught them and told them that some things that should not be done would be done so that this key a strong training human rights proportionality use of force and so on to the law enforcement is a protection to the community to the rule law and also to law enforcement because these people think they do what they can do but but they don't and then there's a problem from for them and the publications are not going to stand behind them when they're convicted actually an easy way to to get out of accountability as a politician is just ordering investigations on the individuals instead of addressing the whole problem with the force that is not a problem of individuals oh yeah that we have that problem I mean we have we have systemic issues that have been brewing a long time and you know you can talk about individuals but you really have to think bigger than that you have to think about fixing it fixing the system but one of the things that I get out of our discussion in your remarks Nicholas is this that if you have an angry protest out there which has the possibility of expanding the issues and expanding their fervor so to speak in their street presence one way to deal with it is to simply say and mean that you are going to listen that you respect their views and you take them seriously and you're going to do something about it rather than just beat them up on the street has this has this been tried has it been tried well or badly do you think it would work if tried correctly I think it would although I believe that also social anger needs to be expressed in some way it's not like okay heard you get out of the streets let's sit on a table that that's not the way of doing it people have the right and societies need the need to express their emotions and anger is a valid emotion and if it doesn't reach to the level of violence it's valid and it's necessary the government and the people need to know and need to be allowed to express their anger but then comes as you say a moment of sitting down and listening I think that was done partially at the local level at Columbia and to some extent it worked and some agreements have come from that but it needs a lot of political will and it needs a lot of follow-up both from the government and from the demonstrators because otherwise you're just going to add another element to the to the insatisfaction that is the lack of compliance with previous agreements so that is key but absolutely dialogue is the way like a demonstration is never a solution of itself it's just I mean to one express anger that social emotions are very important and that are healthy for our society and to to make the government listen but once they listen you need to do something about it you know it strikes me also that at the end of the day in order to maintain the social compact because without the social compact we are we are in Lord of the Flies without the social compact we're in we're in continuing chaos devolving chaos you have to give people some confidence you have to create public real public confidence not not fake public confidence not you know propaganda public confidence so I guess if you're if you are the government you're burdened in a situation like that is to find ways to make people confident not only of the existing administration but the system is this do you think this is doable is it doable in in Colombia is it doable other places I think it's doable the thing is if they want to do it and they're willing to do what is necessary because you need to be transparent you need to integrate a government with people from the other side and and these governments who tend to be very authoritarian they're very closed on themselves they name their friends they name people that think like them and that is no way to have a healthy government and a healthy public administration actually you need people from all sides and you need experts from all sides to tell you what's necessary and even to have people from your from the other side sitting beside you telling you you are not right within differently and there you find a consensus so I think that transparency and inclusion is the solution of course but that is not a point of if it's possible or not but if they're willing to do it or not because that requires a lot of I don't know moral fiber in the first sense and a lot of skills in leadership as well because you have your government plan and you need to to to make it happen but you need to govern for all the population yeah you have to go back to the fundamentals and remember the government is there to serve the people and the people are there to be part of the government it's all it's a two-way street well thank you so much Nicholas it's it's always fabulous to talk to you I so enjoy your comments and your thought and I hope we can do this again soon and I wish you well in all particular thank you Jay thank you for the invitation and we'll keep you updated about what's going on at home please thanks


All righty, why am I here? Great question, it's not because I'm awesome at AV, it's because I like putting out dumpster fires. So, let's talk a little bit, just to kind of frame where I'm coming here. Last 10 years, I have been working pretty much exclusively with Apache Cassandra, throwing some other databases here and there, but I've done this a lot. I worked at DataStacks for a little while with Patrick McFadden. His team had a great time over there. Went to The Last Pickle, we were a small kind of boutique consulting company. Was there for about three years, I was added as a committer to the project at that time. I moved on to Apple for a little bit, worked on the Cassandra 4.0 upgrade, went to Netflix, did a lot of the performance tuning for the Netflix fleet, including all the GC tuning for the entire thing, put out a lot of dumpster fires there, and did the performance tuning for some of the most high profile clusters in the company. Some of the ones that were 600 plus nodes, so I did a lot of work there. We do have to ask ourselves, why do we care about performance? I think this is one of those things that it's easy to kind of misunderstand the reason why we care. Throughput is typically always affected by performance, latency makes a big difference. We really care about performance because first, it affects our end users.  Reason number two, and this is a really big one, cost. Especially when you're running something like Cassandra and a fairly large cluster, it's not cost effective to have to double or triple the size of your cluster when instead you can do performance optimizations and achieve the same result. Because at the end of the day, what we really want to do is get our latency down as low as we can while keeping our throughput as high as we can. And that's really what I've been focused on for a really long time. Now, when I was at The Last Pickle, I have to read. When I was at The Last Pickle, we really kind of focused on having this expert mindset and understanding really all aspects of a problem before we try and solve it. And so it really helps to have a methodology when you're gonna be doing this that you follow rigorously every time you approach solving a problem. So today I'm gonna be talking about a couple different methodologies that we can use for our problem solving. It also really helps to have a good amount of observability. And by that I mean we need the right tools in place and we need to be able to ask the right questions as a result of the methodology that we're gonna develop and we need the right tools to get the right answers. There's a lot of really bad tools out there and there's a lot of really good tools out there. Some tools don't give us great information, other tools do. So I'm just gonna talk a little bit about that a bit. And lastly, much like playing a musical instrument, you need to practice. It doesn't matter if you sit here in this talk and you don't actually take the things that I say and walk away and actually try them on your own. You have to try these on your own. Otherwise when it comes time to do performance tuning or there's an incident, you won't be ready to do it because you hadn't practiced. Just like you couldn't read a book about playing the guitar, and then go up on stage and play, you'd never do that, it makes no sense. You have to practice. So this is like any other skill. So speaking a minute about methodology. Without a methodology, without approaching a problem in some sort of sane way, we're literally just doing the equivalent of closing our eyes and turning dials. I've seen a lot of people try to do performance tuning by making arbitrary changes to configuration files, to GC settings in hopes that they're gonna do something correctly without understanding the impact of what they're doing. But in the case of Cassandra or any database system in general, there's literally hundreds of changes that you could possibly make. And if you need to make two changes and there's thousands of different parameters that you could change, whether it be schema, whether it be something at the OS level, Cassandra configuration, it becomes very, very difficult to guess and get the right thing, especially when there's multiple. So what I'm gonna teach you is the right approach to achieve an understanding that allows you to tune the right parameters. We're gonna start with something called the Udalloop. This is something that we used heavily at the last pickle. It was developed by the Air Force for fighter pilots, but it turns out that that kind of problem solving and that sort of methodology is the exact same type of thing that we need when we are trying to put out a dumpster fire or we're trying to just improve performance in general. So the Udalloop stands for observe, orient, decide, and act. Observe means that we just need to take in information. Orient means that we need to build a mental model to understand what's actually happening in the system that we're looking at and then decide and act or pretty much self-explanatory. If you think about the example I just gave where we just changed arbitrary configs, that's just act. We didn't really make a decision based on our observations and a mental model. We just guess at things. It doesn't work. We're gonna stop doing that. What we're gonna do, whenever we wanna do some sort of performance tuning, is implement the use method. Now one thing I should have noted, but I got a little bit caught off guard by my original, by my laptop issue here. There are QR codes on these slides. I don't know if they're gonna work when you take a picture of them, but they do link to other helpful resources, whether it be blog posts, YouTube videos, things I've written, stuff like that. So maybe it works, if someone can try, hopefully it does. Yeah, it does, awesome, cool. So these QR codes are gonna be helpful. So I'm not gonna be able to go into the normal amount of detail that I would love to, because I don't have six hours to speak to you today, and you probably don't wanna do that. When you're ready, lots of really good information here. The use method is a way of understanding what's happening on a system by examining all the different resources. And this was invented by Brendan Gregg. And basically what we do is we say for each resource, we're gonna check its utilization, saturation, and errors. Now, utilization is something that we might be familiar with if you think about CPU utilization. This is fairly common terminology, but the definition is the average time that the resource was busy servicing work. So for example, our CPU is at 90% utilization. When we talk about saturation, we wanna know the degree to which the resource has extra work, which it can't service. Okay, so when you have any of your components of your system all have queues. So for instance, when Cassandra sends a request to the underlying IO device, it first goes into the queue and then it becomes part of the, and then it gets serviced. So if we end up at a point where we've sent too many requests to an underlying device, we end up with a queue and the saturation is the measure of the length of that queue, right? For example, we might have an IO queue of 100. Using a tool like IOstat, you can see this is gonna show up as the average service queue. And errors, we definitely need to know how many errors a particular subsystem is throwing. This can be at the network level. For example, if we see TCP retransmit, we know that we have packet loss along the way. So we need to be able to monitor each of the different components of the system, understand their utilization, saturation, and error rate. And we do that with awesome observability tools. So from a high level, you're definitely gonna wanna make sure that you have high level dashboards that you can use to figure out when you have nodes that are anomalies or performing outside of the normal expected behavior or if the entire cluster has a problem. There's a few things that you don't wanna do. I'm just gonna go over a handful of anti-patterns because I see these a lot and I wanna make sure that folks walk away and have something that they can work on here. Using averages is really terrible. I've seen a lot of dashboards where people plot average latency and they use that as their indicator whether or not their performance of their cluster is doing okay. Average latency is worse than nothing because it will actually mislead you into thinking that the cluster is okay and you'll be sleeping just fine while your customers are experiencing pretty terrible performance. The reason for that is because a lot of your requests, it turns out, fall outside of the average. And if we're not doing a good job of identifying when those requests are actually happening, we get lulled into a false sense of security. We don't wanna hide the outliers.  So you need to know what your max latency is at any given time and your P99 latency and anything under that, I would just ignore. I would not bother doing those graphs. Another common mistake is averaging summary data and only showing that. For instance, if we were to take the P99 latency from each node in a cluster and average that together and just plot that on a graph, we're not getting useful information because now we might have an outlier that's experiencing significantly worse latency and we've hidden it by averaging that number. And if we take a deep dive into statistics, which we're not going to do right now, you'll discover that averaging a P99 latency does not give you any useful information at all. It's completely worthless.  So whatever you think your average latency is for the cluster, you also wanna make sure that you understand what your worst latency is. And another anti-pattern is not understanding what your tools are actually telling you. It's really common, especially with a tool like IOSTAT, to look at some of the output that it is giving you, like the average service time and think that that number is telling you something good or bad, all it's doing is giving, once again, an average that doesn't mean anything. So I really recommend taking a look at this link that I have here with the QR code. We're actually gonna learn quite a bit more. There's a guy named Gil Tenne who talks about, he has a talk from Strange Loop in 2016. And the talk was called, I think it was everything you know about latency is wrong. Really, really interesting talk he is the CTO and co-founder of Azul and an incredibly bright guy. So I highly recommend looking into that and watching it. It's entire talk on this topic. So once we understand what's going on at a high level, I really like to use system tools. I think observability tools like Grafana, Prometheus, AxonOps are fantastic. Sometimes you need to get in and get more granular data and you need to get some data that might not be available in those tools. So you definitely wanna be familiar with this. Now at a high level, I think a lot of people are familiar with Sysstat. Sysstat comes with useful tools like IOstat, which I've mentioned a couple of times, MPstat and a handful of other utilities. And these are really nice for getting a high level first cursory look into a node, but it doesn't actually give you enough detailed information to help you narrow down problems quickly. That's where I really, really like EBPF and BCC tools. So EBPF is a technology that originally was called Extended Berkeley Packet Filter. Now it's just its own name and it's no longer that. This gives you the ability to tap into the Linux kernel underlying probes and you can actually get a lot of detailed information about what's going on in the various subsystems of your server. So let's suppose that we wanna answer a question, right? This is kind of a common activity. What happens if we give Cassandra a bigger heap? There's a few things that can happen as a result of this. One of those things is we can improve our GC pause frequency because a larger heap gets collected less often. But one of the other side effects of this is we can take away memory from our page cache. If we only have 32 gigs of RAM and we give Cassandra 30 gigs of RAM, we're not gonna have a lot of memory left over for our page cache. So one of the things that people are very cautious about is making changes here, especially when it can take away. Your page cache is gonna serve requests a lot faster than your underlying drive can. So making sure that you have high cache utilization is really important. But how to measure that is a big problem.  Again, this is available in BCC Tools. So CacheStat can tell you your page cache hit ratio and it can do so at any interval that you want. So I typically will run this at a one second interval over a long period of time and I'll watch to get an idea of what kind of page cache hit rate I'm getting. That way, if I've made a change to the amount of memory that I've allocated to Cassandra, I know whether or not that change now has a side effect of taking away memory from page cache. And then as a result of that, you would be expecting that you'd see more IO. So then you have to ask yourself, well, how can I tell what's going on at the IO level of my file system? So that's where we're gonna use tools like EXTFS slower and EXT for DIST. The same tools exist for XFS, EFS, BTRFS and a whole slew of file systems. EXT for DIST is really nice because it will break out by the type of operation that you're doing, how many of those operations there were and what the latency distribution is. So you're gonna, these render power of two histograms, which is really, really nice because whenever you do encounter a performance problem, the first thing that, I think one of the first things that people blame is the file system where they blame the underlying drive. So it really helps to be able to get numbers out of there like this. Now, one of the things that is a little bit tricky about this is these numbers are gonna be a bimodal distribution.  So you need to then ask yourself, well, how do I get the information out of my block device? Well, there's another great tool in BCC tools called biolatency. Biolatency will show you the same type of histogram, but it will take a look at the underlying block device, clock every request and give you a histogram of that latency distribution. I was able to use this at several organizations, one of which was Netflix to prove that EBS was not a performance problem and another client of mine who had a SAN and they were convinced that the SAN was the problem and we were able to use this tool to determine that the SAN was actually not the underlying problem for their performance issues. Now that we've taken a look at the underlying hardware, we know how to look at IO, I think one of the next logical questions that we can ask is, how can we tell what Cassandra itself is doing and spending time on? Because I think a lot of the time it's easy to think of software as a black box that we can't peek into and understand. But the reality is, is that modern profilers, especially sample-based profilers are safe to run in production and can give you incredible insight into your application. So that's where I rely on flame graphs. Flame graphs are probably one of the most useful tools I've ever encountered in my entire life and I've spent the last several years using them to solve some of the most complex problems that otherwise would have taken a lot longer to solve. So this, I believe this QR code goes to an article in my blog where I've kind of go into some of the detail, really, really go into detail about how this is used in some different ways that you can use it to profile, to do CPU profiling and heat profiling. So you can actually see the root cause of allocations in addition to CPU time and numerous other things that you can take a look at. But let's take a look at one example. So this is a flame graph that I took when I was running my Cassandra benchmarking tool TLP stress against a single node. And one of the things that you can see here is that there's kind of a spike up in the middle with a wide line. Now the levels of the flame graph represents the stack of the program. So as functions are called, it gets higher and higher and higher and the width represents the amount of time spent. So one of the things that you probably can't read is that at the top of that graph where there's that larger red line is it says LZ4 decompress. Now that corresponds to the LZ4 decompression algorithm that's used when we're reading SS tables off disk. Now if we ask ourselves, okay, cool, how does this help me? How does this help me? One of the things that we should naturally ask ourselves is okay, if we're spending a lot of time in LZ4 decompression, is there a way that we can make it faster? Is there a way that we can do less of it? So compression is one of the table parameters that exists on every table. Whether or not you specify it. In versions before Cassandra 4.0, this is gonna be that question mark there that's kind of tough to read is gonna correspond to, it's gonna say 64. And on newer versions, it's gonna be 16. So compression works by having, so the compression chunk length in KB value is used to create a buffer. When you're writing to an SS table, boom on disk, that buffer gets filled up, it gets compressed and it gets written to disk. Whenever you wanna do a read, the opposite flow happens, it reads the compressed buffer off disk, decompresses it and returns it to Cassandra. So the interesting thing here that we can ask ourselves is okay, what does that mean, what's the impact? Well, if we have really, really small values, like small rows that we're reading like I have right here, where you can see that the partition size is only 35 bytes, because there's hardly anything in it, you start to wonder what's actually taking place here. And what we can logically deduce is that Cassandra is reading a whole bunch of data off disk, it's decompressing it, and it's only using a tiny bit of that data to return to the client. So what we actually have here is an example of massive read amplification, where we're reading so much data off disk and we're turning hardly any of it. That results in a higher level of disk utilization and under enough load, we'll eventually read to full saturation where we end up with queuing on the disk and then our requests take a long time because they're slowed down. So one of the things that's important here is to be able to have the knowledge, to be able to look at a tool like no-tool table histograms and figure out from the partition size that it's okay to resize your compression chunk length. And when you do that, in this case, you would wanna resize it to four because that's the smallest possible value that you can use and your partitions are so small that what we wanna do is minimize the amount of data that needs to be read off disk. So now when we use four, we're actually gonna read the block that was originally 4K and now it's compressed. The trade-off here is that you're gonna use a little bit more memory for compression metadata. So you might wanna increase your heap size because that's the trade-off and you may get a slightly worse compression rate. However, if your problem is IO and your problem is related to GC pauses, this can help a lot because read implication on small values is one of the problems that I've seen in almost every single cluster that I've looked at. So this doesn't just affect read-heavy workloads. It actually affects several other workloads. So it affects anything else that's doing a read before a write. For example, if you're doing a lightweight transaction, when you do lightweight transactions, we actually have to read the original data off disk because you say if not exists or if some value equals something. So as a result of reading data off disk, now we actually have to think about how to tune our system and how to tune Cassandra to handle reads as performant as possible. And so typically with lightweight transactions, this might be very, very small records. In fact, I can only think of instances where you would be doing something with one row. And so you're pretty much always going to want to resize your compression chunk length to four KB in instances where you're using lightweight transactions because the effect of read amplification is so significant. And it's the same thing with counters. Counters read the current value off disk and bring it back into memory. There is a countercache, but countercache shouldn't be thought of of something that just patches over a slow problem. We need to fully optimize our system so that we can, even in the case of writing to counters which are not in the countercache, we still get high performance. And that's why in Cassandra 4.0, we made this change, lowering the default chunk length in KB from 64 to 16 because for so many workloads, it actually makes a significant impact to make this very, very simple change. And on the topic of read amplification, if any of you have seen any of the things I've written or read about or spoken about at conferences before, I really wanna try and make sure that people understand this. One of the worst things that you can have on in Cassandra is one of the system defaults, which is read ahead. Read ahead is one of the most tragically broken features in modern hardware. And it should basically be disabled in almost every single circumstance outside of, if you had like a really, really slow sand where there was very, very high latency. Read ahead works by, when you do a request for a particular block, read ahead or a particular file, read ahead will get extra data. And it puts it in the page cache. And the intent is to make up for the slow latency of spinning disks and it's supposed to help you keep more things in page cache. But what it really does is it puts more pressure on the CPU, more pressure on your disk and results in overall just poor performance for Cassandra. So I worked in several situations, both in my own benchmarking and in real environments where I've gotten a 10X improvement to workloads like counters and lightweight transactions. I did the performance tuning for one of the largest gaming services on the planet. And this was the most important change that they made that helped them go from outages during major releases to everything humming along super silky smooth. So if you're curious about how to figure this out, if you're impacted by this, if you haven't made a change, you probably are. By using block dev report, we can take a look at the read ahead by going to this RA column. And the RA column is gonna say generally 256. Now this corresponds to 512 byte sectors. So read ahead, whatever you're doing here, divide it by two and that's the size in kilobytes. You can also read from, there's a system variable that you can read out of, it's like sysq, sysblockq, something. I forgot what it is exactly. To change it, you can do block dev, set RA, pass your new read ahead. I like to put a fairly small value here. It's okay, so this is eight, which means the value is 4K. The reason why I do that is because you have to read at least 4K if you're reading off disk anyway, so it's okay to do this. So the last thing I wanna touch on was practice. Practice is really important. As you can probably tell, I've spent a lot of time just looking at this stuff in detail and thinking about it and trying to wrap my head around how it works. It's helped me get a really, really good understanding of Cassandra at a level that I think most people aren't willing to commit to because I've spent so many hours doing it. The easiest way to get started doing this is to set up a lab environment. Now I wrote a tool when I was back at the last pickle that Datasnacks now maintains custody of. It's called TLP Cluster. You can use it to spin up lab environments in AWS in just a few minutes. It's actually, I designed it to be pretty easy to use. I think you have to maybe type in about three commands and you'll end up with running Cassandra clusters that already have some additional software installed like the Flamegraph tooling that I already described. I'm in the process of rewriting some of this tool so that it actually builds an AMI with all the tools on it already, including all versions of Cassandra so that we can actually get that start time down to about a minute. And I'm really, really excited to be working on that but it's totally broken right now so don't use my version yet. The other alternative that you can do is set up AxonOps. So AxonOps recently, I think, but today, yesterday announced your provisioning process. So AxonOps is, you can talk to John back there. They have a really nice provisioning tool that can set up clusters for you and set up monitoring for you. So if you're trying to just tinker to something and get it done, I know that they're offering, you can use a small environment for free. So pretty cool stuff but you just definitely need to have something that you can mess with. If you've already got all your own software, awesome, keep using that. You don't have to use anything else but if you're looking to get started with something easy, one of these tools is probably gonna be your best bet. And the second thing that you need to do is benchmark. So you need to break things. I can't say enough how helpful it's been for me to have worked on so many clusters that were broken. I know that might seem kind of crazy but it means that I had to fix problems. And when you have to fix problems and you use something like the Oodaloop, you get put in a position where you ask the right questions and then you can kind of build that mental model to understand how things work. And you get the best, in my opinion, understanding of how a system works by breaking it and looking inside of it. So I wrote TLP Stress back when I was at the last pickle. This is one of the most widely used benchmarking, one of the most pieces of widely used benchmarking software for Cassandra I know of several hundred teams that are using it. It's used to benchmark some of the largest environments in the world. There's also NoSQL Bench, which is from Datastacks and Indy Bench, which the folks at Netflix have provided. So in summary, it's really important that we have a good methodology and we follow it whenever trying to do any sort of problem solving. It doesn't just apply to fixing, improving performance and fixing a problem. It applies to dog fighting and anything else that you can possibly think of. Next, we wanna make sure that we understand the full state of observability. That doesn't mean just your dashboards, but also the tools that you wanna use when you wanna look at an individual machine and answer very, very specific questions. There's a lot more to the observability side that I didn't talk about. There was a slide on Brenning Greg's slide that I went by kind of quickly. There was a QR code there, I hope you got it. And the last thing is practice. Practice is super important. If you don't practice, you won't know. So one of the things that I'm really, really excited about is to announce that I'm now not just doing Cassandra Consulting, but I'm also doing training. So I am doing hands-on training sessions starting Q2 of next year where we're gonna be doing Cassandra fundamentals and a lot of this type of thing, figuring out what's going on. So if you are looking to learn pretty much everything about running Cassandra in a production environment at really, really big environments, lots, lots of requests, millions of requests a second on some of the most complex systems in the world, I would love to share the information that I have and help you become experts as well. Thank you. Did I go over time-wise? Am I okay? Oh, it looks like one question. Average latency is not useful for performance investigations whatsoever because it doesn't tell you, you lose, you're throwing away the worst part of the information. So if you look and you have an average latency of five milliseconds and a P99 latency of 200 milliseconds, those numbers are so far off that you can't possibly use average for anything meaningful. And so the thing that you wanna make sure is that you, whenever you're looking at latency you're kind of thinking about both P99 latency, which is gonna be what most customers are experiencing and your max latency, which is gonna be your outliers. A little slow. In the case of, again, I still say that average latency is not useful. So the QR code that I posted on that slide leads to that talk by Gil Tenne. The reason why average latency is not useful at the database level is because of the birthday problem. So if you're looking at 50%, for example, if you look at P50 instead, your P50 time as soon as you do two requests by one user to the database, you only have a 25% likelihood of achieving that P50 latency. So as soon as you start plotting your averages, you start looking in this fictional world where your customers aren't actually experiencing it. So what you really wanna do is focus on the P99. Yes? Yeah. Yeah, you're not gonna wanna go past 31 because as soon as you lose your compressed object pointers, it'll be, I think you have to go into the around 45 gigs or close to 50 gigs before you actually see the effective memory be about the same. So I never go above it. And the only thing that you really wanna do is make sure that you have enough space for the compression metadata. And you can limit that in size as well. So yeah, I wouldn't go above 30. But to your original question, yes, I want to see the effectiveness of my page cache. And whenever I make changes to my heap, I wanna make sure that I didn't do so with the detriment to page cache. I think I'm out of time. Thank you, everyone.


In the year 2000 at the age of 86, Hedy Lamar passed away in Florida. She hadn't been seen in public for years. The actress was completely glamorous on the silver screen, but as age crept up on her, she shilled away from public life and Hedy would have faded into legend with the rest of the actors of the silver screen if it wasn't for one thing. In 1941, she invented one of the greatest technologies of the 20th century, an invention the US government never credited her for until the 1990s, a lost patent that was stolen. Hedy Lamar is not remembered as a pioneer, but that is exactly what she was. She championed wireless communications, invented Bluetooth, and many other forms of wireless technologies that became known as spread spectrum technology. The international beauty from Austria invented the secret communication system to stop the advance of the Nazi party by manipulating radio frequencies at irregular intervals between transmissions and reception. This invention formed an unbreakable code to prevent classified messages from being intercepted by enemy personnel. An invention so important, so advanced that it was kept secret until 1997 when her son pushed for her recognition. She received the Electronic Frontier Foundation Pioneer Award in 1997 and though she was not present at the event, she phoned her son to see how it went. Funny enough, her son was giving the speech she prepared for him to read out on her behalf just as she called. He took the phone call to his mom and spontaneous ovation broke out in the crowd. Hedy remarked, it's about time. During her heyday, Lamar was considered the most beautiful woman in the world. Her face was the inspiration for Disney Snow White and for Catwoman. She was the most famous Hollywood star who would finish performing on set with Clark Gable and then go back to her trailer and work on her inventions. In rare long lost cassette tapes from the 1990s, Lamar describes her contribution to aerospace engineering when she says, I thought the airplanes were too slow. I decided that's not right. They shouldn't be square the wings. So I bought a book on fish and I bought a book on birds and then use the fastest bird connected it with the fastest fish and I drew it together and showed it to Howard Hughes.  The concept of bringing animals of this earth together to replicate their motion is something no one had thought about until Hedy said this. This was a quantum leap forward in our understanding of the technological things and where will we be without her influence? A quiet influence that affects us all in everyday life. Let us remember the great thinker for what she deserves to be remembered for. Hedy once said that people's brains are more interesting than their looks. A message of her frustration in a world that could not see past her intense beauty gifted in many ways like a goddess to behold. She describes ideas as coming to her naturally. Maybe she knew the secrets of the Acacic realm, but what do you guys make of this? Comments below and as always, thank you for watching.


Now that we've shattered some classic money myths, it's time for us to begin really rewiring our consciousness and putting ourselves into a place where we can start opening to making money and getting to that place where you never have to worry about it again. Let us begin with a psychological experiment that describes how we view money in relationship with value in the world today. In this particular experiment, people were asked to judge five different glasses of wine presented at various prices from $5 to $90. Before drinking, participants were told how much each bottle of wine cost. After tasting them all, the participants generally agreed that the most expensive wine was their favorite and the cheaper ones were their least. Now, maybe you saw this coming, but it was later revealed that the wine presented as a $90 bottle and the wine presented as a $10 bottle were actually the same drink. That means that in the mind of the taste testers, the price tag itself changed their perception and gave them a better or worse experience. Not only did participants report enjoying the same wine with a higher price tag more, they even had increased activation in regions of their brain hypothesized to be associated with pleasure. This study teaches us a few valuable lessons. One, it would seem, is that the more you pay for something, the more you tend to value it. This is commonly reflected in an analogy about books. If someone just gives you a book randomly, you're less likely to read it and you're personally invested in buying it. We can also see this with wedding rings in society today. You might be familiar with the fact that diamonds aren't actually very rare, but they have a long history of being marketed as if they were, so they could be sold for very outlandish prices. Because of this perception of value around them though, a wedding ring that costs $10,000 versus one that costs only $100 says a lot to the young couple due to the value that's been attached to it. Now, I'm not saying that it's a good thing that big diamonds are sold for thousands, but rather, if we can recognize the relationship between price and personal value, we can make better choices about what we spend our money on. Actually, let's stay on the wedding rings for just one more minute because this is relevant. The first recorded evidence of a wedding ring concept actually appears to have come from ancient Egypt over 3,000 years ago. The writings depict couples presenting each other with braided rings fashioned from hemp or reeds, which would eventually be replaced with bands of either leather, bone, or ivory. The more expensive the material, the more love was believed to be shown. The hieroglyphs show that Egyptians believed that the rings symbolized undying commitment and eternal love between the couple because the circle has no beginning or end with the center opening symbolizing a door to the future. The reason I'm sharing this is that we have this curious tradition that would take an individual's love and demonstrate it with the symbolism of both the ring and its material, physical value and worth in a monetary sense. However, we must always remember that the money doesn't buy love and that even in the Egyptian tradition the love came first and the ring was a reflection of the love. While the material and its value played a part it was more of a psychological trick, if anything, giving couples a sense of feeling that their love was indicated by the worth of the material of the ring. This is significant because it's not so much the dollar value that's necessarily important but how much something is worth to you personally. For some people a thousand dollars is a lot of money an astronomical amount. For others that same thousand dollars is not worth very much at all. So we each have a different identification with our perception of value and this is important to identify if we want to shift that at all. On that subject let's look at value in another way. From the perception of the service provider in the case of someone who is offering a product or service having a higher price tag is actually the best possible choice because it will allow for a higher quality of service. For example let's compare going out for a premium dinner versus the drive-thru at McDonald's. There's an entirely different kind of experience here and the environment itself will most certainly play a part in how you feel personally about the experience not to mention the quality of the food the cleanliness of the environment the atmosphere, everything. I don't want to say that it's always best to go with the most expensive route especially if it's unnecessary in that particular situation. In truth, sometimes premium is better and other times it's better to just let all that go. For example, it's like the difference between a fancy trip on a cruise around the Caribbean or a camping trip to a beautiful place a few hour drive from home. Now in this thought experiment notice what happens in your mind when we look at these two options. For people who can't afford a luxury cruise experience to immediately start judging it as negative because it's out of one's reach. And if you're someone who is for lack of a better word fully loaded how do you feel about camping? And I don't mean glamping. I mean a tent and a fire and just being in the rawness of nature. Does the experience feel beneath you? These two potentials could both be wonderful but our perception and relationship with money indicate how we approach the options we have in our lives and more importantly how we feel about them. In my life, I once had a friend who absolutely hated Apple computers. Apple was the worst he would declare and list off all of the ways that Max sucked and why Windows was superior in every way. He was very passionate about this subject however after several long conversations eventually the truth came out. He hated Apple products because he couldn't afford them because of the MacBook. Well he'd probably love it. The hate and disdain that he felt towards this physical product actually had nothing to do with the product at all. It was all about his own sense of value and self worth and the negative relationship that he had with money. So the question then becomes how do we shift this? How do we transform our relationship evolve our mindset when it comes to money? The first stage in wealth transformation is to recognize our limitations and if we have fixed beliefs about who we are and what we're capable of or not capable of we'll have to break these down if we are to evolve beyond them. We can't break free without calling out our blockages. True transformation begins within and that's where we must start. Speaking to this further the most widespread disease of our time may very well be icantitis. It is contracted by many of us early in life from our elders and peers. In fact society has made a song of it and I know reason but it's heard everywhere. I can't because I'm poor I can't because I'm sick I can't because I do not have ability I can't because there's no opportunity or perhaps I'm just very, very old and so on and so forth. Few people use more than a small part of the tremendous God power within but you can alter the pattern of the icant tune and in fact you must. No progress in the realization of prosperity unless you do so. This calls for knowing, really knowing that you are a spiritual being living in a limitless universe endowed with the whole potential energy flow of the cosmos.  Let go of icant and begin to identify yourself as the living enterprise of the infinite one. You are not an average person you are you a unique individualization of the universal creative process affirm for yourself now I can because I am.  Or perhaps you need a bit more time affirming the cosmic power within you. Well if so, that's great because it looks like we're out of time for this episode. I'll wrap it up with a fun quote and I'll see you next week to go much deeper. You can only become truly accomplished with something you love. Don't make money your goal.