Whenever, we are designing a water resources project, we must decide on what should be the design values. For example, if we are designing against flood control, then we should know what is the maximum flood which can occur. Now, these are all probabilistic or depending on chance. For example, the floods we do not know what will be the flood which will be occurring next year or next to next year and so on. But, based on the previous data, we can estimate some of the values. Similarly, if we are making a storage reservoir to protect against drought, we should know what is the minimum flow which can occur in that area. So, all these are, since they are all based on probability and chance, we should now look at some of the things which depend on a probability. And these are known as risk, reliability and safety factor. All these factors, they partly will depend on what is the probability of occurrence of an event. Most of the hydrological events, for example, rainfall or floods, they would be dependent on chance and nobody can predict what will happen next year or next to next year. So, that is why we have to analyze them in a probabilistic frame, so that we know at least what is the probability that such a flood will occur next year or what is the probability that minimum flow in the river would be so much. So, for this, we should look at some probability analysis and that is what we are going to do in this lecture. We will look at some of the probability theories and based on that, we can assign a risk and choose a design value. .. So, let us write the probability as probability of occurrence of an event. Let us say x is an event which may be rainfall, it may be flood in a river, it may be water level in the river and we want typically probability that this x would be greater than or equal to some specified value x. Now, if we are dealing with floods, typically we will be using greater than or equal to. If we are concerned with drought, then typically we want the probability that x is less than or equal to certain given value x which may be the minimum desired flow in a river. If we use this greater than or equal to x means probability of accidents, then typically this would be the flood in a river and naturally this probability would depend on what time period we are talking about. For example, what is the probability of occurring it in next 10 years, next 100 years and so on. So, this we would denote by P and we would write it as probability x greater than or equal to x or in most of our discussions, we will just use the symbol P to denote the probability that the variable x which may be let us say the maximum annual flow in the river would be greater than or equal to some specified value x. Now, based on the probability, we can define a return period T and it would be simply equal to 1 over P or we can also call it recurrence interval means what is the average time lag or time gap between two occurrences of an event. So, typically it would be given as 1 over P where P is the probability. So, if probability of an event occurring in 1 year is let us say 1 by 2, then the return period will be 2 that means on average the event will occur once in 2 years, but we should be careful with the term return period it does not mean that the event will occur every T years does not mean event occurs every T years. So, this is important because an event may have a probability of 0.5 it does not mean that it will occur once in 2 years, we can take the example of a coin if we toss a coin we know that the probability of heads or tails is half. So, this means that on an average out of every 2. So, the return period in this case would be 2 1 over the probability. So, this is the probability of heads or tails and therefore, the recurrence interval would be 2. So, we can say that on an average in every 2 tosses of the coin one head and one tail will occur, but it does not mean that if we toss the coin twice or let us say 4 times there will be 2 heads, but if we take a very large number of trials. So, the key word here is that large number. So, if we toss the coin let us say 1000 times then we would expect there would be 500 heads and 500 tails. So, it is not that every 2 occurrences will have one head and one tail, but on an average every 2 occurrences will have a head. So, that is why we should be careful in using this return period that it is not that if we have a return period of 10 years the event will occur every 10 years or once in every 10 years it may occur twice in a period of 10 years then it may not occur even once in the next 10 years. The other term which we use or we can based on this value of p suppose we have n years of record and typically these will be consecutive years. So, suppose p is known to us and we want to predict how many times the event will occur in the next n years. So, we can define a probability of x being greater than equal to x this event occurring r times in let us say next n years. So, in theory of probability there are equations which give this probability and we can write it as p r n which is the probability that x will be greater than capital X or equal to capital X r times and n is the number of years typically we would deal in terms of years. So, if annual flood we will say that n is the number of years in which the event is occurring r times and this can be given in the probability theory there is an equation which relates this is nothing but n c r factorial n over factorial n minus r factorial r then we have probability of occurrence and probability of non occurrence. So, if we look at this equation this is the combination parameter n c r p is the probability of occurrence of the event and 1 minus p is the probability of non occurrence. So, this equation gives us the probability that an event will occur r times in next n years. Now, based on this we can decide what is the probability that the event will not occur at all in the next n years. So, if r equal to 0 this indicates that the event does not occur in next n years and this is a term in which we are interested quite a bit because if we design some project with some value of let us say design flood and its design life is n years we would like to know what is the reliability. So, we will call this term reliability because this indicates that if we design some structure for n years of useful life then in those n years the design flood will not be exceeded because r equal to 0 means that the event is not occurring even once. So, we can write the reliability r e by putting r equal to 0 in this equation and therefore, the equation which we get is 1 minus p to the power n.  So, reliability is given by this simple equation if we know the probability of occurrence of the event in a single year then in n years the probability that the event does not occur even once is the reliability of the structure or that project. So, we say that the reliability of the project is the probability that the event will not occur during the lifetime of the project which is let us say n years and therefore, the structure will be safe because it has been designed for that design flood. Similar to reliability there is another concept which we associate with it which is the risk and risk let us call it r i is nothing but 1 minus reliability. So, which is logical and this risk is the probability that design value will be exceeded in this case we will talk mostly about the floods. So, we will use the exceeded term here in route it will be less than that. So, probability that design value will be exceeded at least once in n years where n will be the design life or the useful life of this structure and at least once means it may occur more than once, but at least once if it occurs then we will assume that if the flood is more than a design flood then the structure is likely to fail. So, our risk associated with the project will be this probability and therefore, we can also write it as 1 minus power n or since p is 1 over t that is the return period of the particular design flood we can put that value here or here and get the reliability and the risk in terms of the return period. So, we can write the risk as 1 minus where t is the return period of the design let us call it design flood or design discharge. So, if we use the design flood for a particular return period we can get what is the risk associated with that design flood and risk is something which we have to choose. So, we choose how much risk we can accept for a particular structure if the structure is very important for example, if there is a dam failure of the dam may cause a lot of loss of life and property then we would try to have minimum risk or very small risk, but if the structure is such that its failure will not lead to loss of life or property it may be just inconvenience to people for example, there may be a culvert on a road. So, the flooding of that road will not cause typically loss of life or property it may be just inconvenience to the travelling people that the road may be blocked for some time, but the loss will be minimum in that case we can go for a higher risk and to reduce the cost. As is clear if we use a low risk then the cost becomes large because low risk means that the return period has to be high. So, high return period would mean low risk, but costly because q will be larger the design q obviously, once in 2 years flood or once in 100 years flood they will be very different, once in 100 year flood will be very high. So, if we design for that the structure will be very expensive and risk will be very small. So, we have to balance the cost and the risk based on our engineering judgment as to how important the structure is. Now, similar to the concept of risk and reliability there is a concept of safety which we can described in terms of a factor of safety or sometimes we use the term safety margin. They are little different factor of safety is a ratio as the name implies and it is defined as the actual value used for some parameter p divided by its theoretical value of the same parameter p where this parameter may be rainfall, it may be flood, it may be water level. So, A p is the actual value of p used in design and T p is the theoretical value of the same parameter. So, this p may be it may be flood design flood, it may be water level for example, in a river or it may be rainfall. So, this may be any parameter p and T p is the theoretical value of p which is based on the hydrologic analysis. So, what the factor of safety means that once we find out T p using theoretical considerations we would further increase it by some factor to take care of other uncertainties. So, we perform a theoretical analysis, but this analysis also is based on some assumptions and therefore, to be on the safer side we would increase the value of T p by some factor this may be for example, 1.1 or may be 2. So, this factor of safety also will depend on it may have any range and it will depend on what is the importance of this structure. More important structure we will probably use a higher factor of safety and safety margin is nothing, but the difference of A p and T p. So, for example, if theoretical flood comes out to be 10,000 meter cube per second the actual value which is used is may be 11,000 meter cube per second with a factor of safety of 1.1 then our safety margin will be a 1000 meter cube per second. So, this means that if our design flood is let us say 11,000 meter cube per second means we have a margin of safety there of about 1000 meter cube per second. The actual maximum which we expect is only 10,000 meter cube per second, but in some case if there is a higher flood due to unforeseen circumstances up to 1000 meter cube per second higher flood can be accounted for. So, this gives us a margin which there is further safety in our design. So, risk reliability and factor of safety all these are very important concepts whenever we design something which depends on chance. So, all these hydrological variables for example, flood rainfall etcetera they would depend on chance and therefore, these three must be accounted for the risk reliability and the factor of safety. Now, let us see how we can obtain the probability and how we can analyze a given data this is typically done using frequency analysis. The data which we have typically will have values of parameters given like time may be q suppose we are dealing with the flood in a river then this q may be the discharge in the river. The time may be daily it may be hourly it may even be weekly or 10 days. So, that depends on what frequency of measurement we are using, but the discharge values in the river they may be shown in a continuous curve it may look like this will respond to rainfall in the catchment area or flow to ground water evaporation. So, all these factors will decide what shape of the discharge curve we will get. Now, based on this data this can be thought of as a time series. So, q is a variable which is a function of time and we can think of this as a time series and then there are lots of methods commonly called this frequency analysis methods which can be used to obtain the frequency distribution or the distribution with time also we can say that suppose this q data is available to us for 10 years or 30 years and we want to design our structure for let us say 100 year return period. So, one aim of the frequency analysis will be to extrapolate the available data. If the data available is only for 10 years or 30 years and we want 100 year flood then we would have to do some kind of extrapolation and those techniques frequency analysis techniques can be used to do that. The way we do it is suppose we are interested in the flood. So, we can find out suppose this time is in years 1, 2, 3 and suppose we have this 50 year of data. Here we can say annual maximum flood or maximum annual flood. So, for the first year what is the maximum flood in the river for the second year what is the maximum value for the third year what is the maximum value and so on. So, we would have 50 of these values available to us and based on these 50 values we have to decide on the probability of a certain event. We may have to extrapolate it to get 100 year flood or 200 years flood. When we choose this maximum annual flood generally what we do is for a single year we take the maximum value and we ignore the values which are smaller than that maximum value. There is one other option in which we can consider all the floods and then choose let us say 50 largest floods. In that case in 1 year itself there may be 2 occurrences of floods, but in hydrology typically we use maximum annual flood such that only one which is the largest flood in a year is accounted for. So, we consider only the largest flood in a particular year. If the second largest flood in a year is larger than any other largest flood of any other year then we do not consider that. So, the maximum annual flood is for a particular year the maximum value. Once we get these values suppose there are n records we will arrange them in order. So, let us say we have here the rank and their queue. So, these n records are arranged in order like m will be going from 1 to and suppose n is 50 or we can use some number n here. The largest value would be written here and the smallest here. So, these would be decreasing order of magnitude. For example, if we look at these 50 values suppose the largest value occurs in year 40 that largest value will go as rank 1 and the smallest of these floods may be it will occur in let us say year 30 then that value will be at the end the smallest. So, once we arrange them in the order. So, we have a table like this where we have m and q we define a new term which is the plotting position. So, m goes from 1 to up to n q largest and smallest and the plotting position is generally taken as the plotting position m minus some constant c 1 over n plus some constant c 2. These constants vary for different techniques. It is most commonly used technique is known as the viable plotting techniques and in the viable technique we have m over n plus 1. So, the plotting position here which is used is 1 over m over n plus 1. So, for example, for this case it will be 1 over n plus 1 and for the last value. So, this tells the probability of exceedance of the particular event. So, for this discharge this would be the probability for this discharge this would be the probability and the recurrence interval is just 1 over probability. So, if we do that we can plot either probability versus q or we can plot recurrence interval t versus q. So, t or p since p is 1 over t we can plot either 1 they are essentially the same. So, the time period of the recurrence interval would let us say go from 1.01 and we may reach a recurrence interval of about 50, 100 and so on. So, data which we will plot from this we can see that the largest value will have a high recurrence interval then the smallest value will have. So, there will be some kind of plot here which will tell us about recurrence interval versus discharge relationship and then we can extrapolate. So, we can extend following that same trend and we can extrapolate based on extension of this curve what will be the 100 year flood in the river. So, this probability plotting is an essential component of extrapolation. Weibull is just one method there are some other methods also for example, you can use both c 1, c 2 as 0, 0 or you can sometimes use half and 0 sometimes 3 by 8, 1 by 4. So, these are various combinations, but we would not be using these we will just stick with c 1 equal to 0 and c 2 equal to 1. So, 0 and 1 which gives us the weibull plotting position. All these frequency distributions are typically written in terms of there is some mean value of the variable x bar and k sigma x. So, this is the mean of x. So, x may be the annual maximum flood and this is the standard deviation. This x is t here written period of t. So, x. So, we want to find out the value which has written period of t it will depend on the mean the standard deviation and a frequency factor. A frequency factor which is a function of written period and type of distribution.  So, depending on what kind of distribution we have for the variable x in this case q, we will have some equation which will relate the frequency factor with the written period and the type of distribution. And once we know the value of k, we can find out the x for any written period t and use that in our design. The some of the commonly used distributions we know that we mostly use the normal distribution for most random variables. So, normal distribution or which is also known as the Gaussian distribution is quite common for most of the variables, but here we should realize that the events which we are talking about are extreme events. So, q may be normally distributed, but the maximum q in a year these are all extreme events. So, if we look at let us say discharge in a river it goes now in one year they will be let us say this maximum then the other year there is some other maximum and so on. So, very simplistic approach in which we are saying that most of the times it is almost constant and once there is a big flood sometime in the monsoon and then like this. So, the actual distribution of q may be closer to normal, but distribution of the extreme events which are the floods will not be normal. And therefore, we do not typically use the normal distribution for extreme values. There are some distributions which are known as extreme value distributions which are commonly used for analyzing the maximum floods in a river. We will look at two or three common distributions for example, gumbel distribution then there is a Pearson type 3 distribution and we take normally the log of the variable. So, that we call it log of log Pearson type 3 distribution and sometimes we take log normal distribution where we say that the values are not normally distributed, but their logarithms are normally distributed. So, let us start with the gumbel distribution and the gumbel distribution the probability is given in terms of a variable and the again the probability that x is greater than or equal to some x is given by this equation where y is a new variable we can call it a reduced variable which depends on the mean and the variance of x. So, 0.577 plus 1.2825 x minus its mean. So, this term represents the deviation from the mean and then we divided by the standard deviation sigma x to get reduced variable y and based on this y we can obtain the probability of accidents of x from that value x. So, this probability distribution for any given pi p we can also obtain. So, if we say that we want the reduced variable y for a given probability p then we can use this expression which is nothing but transformation of this equation to y p. So, based on return period also suppose we know return period t then simply using p equal to 1 over t we can write this in terms of y t as minus 7 and of t over t minus 1. So, if we know the return period we can estimate what is y t we can put that y t here and get x corresponding to that return period or if we know the probability same thing we can find out y p from this which are identical equations except that we are using this probability in one case and return period in other case and then we can obtain the x corresponding to that probability or return period using this equation. So, the equation which we use is x equal to x bar. So, for any given time period we can find out y t from this equation if t is known we get y t from this and knowing y t we can get x corresponding to that return period t using this equation. This equation is valid when we have a very long period of recurred. So, if the data is limited we will not be able to use this equation let us say n years. So, this equation which is based on the Gumbels distribution is valid for a long period of record. If it is limited to n years data then we will have to modify this equation a little bit by comparing that Gumbels equation with this we see that this factor k which is the frequency factor in Gumbels distribution is equal to. So, this is k. So, this k is valid very long period of now if as we say data is limited to n years of record then this k has to be modified and the equation which we use for k becomes y t y n bar. So, now this value and this value which are some kind of mean n is termed division they would depend on n that is why the subscript n is put there and y n bar is generally the function of n of course, y n bar is reduced mean for n years of data and for example if n is 10 then y n bar is about 0.5 if n is about 50 then y n bar is about 0.55 and as we know if as an infinity then we get the old value 577. So, there is a table of values given in various books and the references which provide the value of y n bar. So, you can have a table of values in which n will be given and corresponding y n bar values of the given. So, this table can be seen in various books. Similarly, this S n is a reduced standard deviation this also is a function of n and again for different n values the value of S n is given in books and other references. So, a table of n and S d or S n will be given for different n values you have different S d values or S n values for infinity of course, you get 1.2825. Some other values are for example, n equal to 10 the value of n is about 0.95 for n equal to 50 it is about 1.16. So, these values again can be looked up from the tables you can put the value here and get a value of k from this equation. So, for smaller number of data points n we can modify the k value and then use that k value here to obtain the value of x. Now, once we have the data we should check whether the data really follows Gumbel's distribution or not. So, for that plotting of the data is required and one of the plots which is commonly used is known as the Gumbel's probability paper plot and in this as we know the time period and corresponding probability which is given by. So, this is the time period and this is the corresponding probability. So, return period of t corresponds to a probability of this. Now, what we can do is we can plot this value, but do not write the values here we write the corresponding values of t and that way we prepare a probability paper which let us say has time return period here 1.01 to 5, 10, 50, 100 and so on. On the simple scale these values show the time period, but they really indicate the value of this term. For example, if you take 1.01, 1.01 corresponding to 1.01 from this equation we will get a value which is around minus 1. something. Similarly, when you take a value of 10 you will get minus ln. So, natural log of 10 over 9. So, you can compute this value and that will be the value corresponding to 10 similarly, 400 and so on. So, on this one you can plot the time period here and then the values of let us say discharge which will be meter cube per second and sometimes when we are talking about floods this will typically be in 1000 meter cube per second because the discharges are generally of the order of a few 1000 meter cube per second. So, that value may go from let us 0 to 4, 6 and on this curve if we plot the observed data and it shows a straight line that means our data follows the Gumbel's probability closely and from this suppose we have data up to 50 year return period then we can extend this line and estimate the 100 year return period flow or the 100 year return period flood using this distribution. You can note down in Gumbel's distribution there is a point which is if you say t equal to 2.33 that is the flood which has written period of 2.33 this will correspond to the mean of x or q in this case and that we can see from this equation when the x is equal to mean. So, when x equal to the mean y is equal to 0.577 by putting y equal to 0.577 and this equation we will get a return period of 2.33 using this equation or using this equation we can get for y equal to 0.577 t equal to 2.33. So, in the Gumbel's plot the value corresponding to 2.33 would be the mean of all the values. So, this is one distribution which can be used and as we have already discussed the data is has some uncertainty. So, the 100 year flood which we predict will also have some uncertainty and therefore, one other thing which becomes important to know is what is known as the confidence interval. So, let us say that we say our 100 year let us say this x is at t return period flood. Now, what is the confidence level which we attach to this x or let us say what would be the range. So, this x is a single value, but what we would now like to have is what is the range of values which will have let us say 90 percent or 95 percent confidence interval. So, what we say is that the x value because there will be some error in our estimates. So, we call this standard error the FPC is a factor which is dependent on the confidence or confidence level. So, our confidence level may be let us say 90 percent or it may be 95 percent or 99 percent. So, depending on that this factor will change some values for example, if we have a 50 percent confidence probability the value is about 0.674. So, this is the confidence level or confidence probability and this will be the function which we use here. Similarly, if we have about 90 percent probability then the so x plus minus means from the expected value which we present for the design flood we will go plus minus this value and what we say is these will be two values. So, one would be x is in the middle then here we have x plus this thing and then the third value is x minus same thing. So, within this range there is 90 percent chance that our actual value will be within this range. So, if we want 90 percent probability we use 1.645 factor here to get the two values of x. The standard error is typically given by some constant c standard deviation of the data points which are n data points. So, sigma n minus 1 standard deviation divided by square root of n multiplied by some constant c and there are empirical equations available to express this c as a function of k and one of the equations which is commonly used is 1 plus 1.3 k plus 1.1 k square. So, this equation can be used to obtain c and then we can obtain the standard error depending on the probability. For example, if we have 99 percent probability we have to use a large multiplying factor 2.58 and therefore, this range will become higher. So, if we have this expected value x the range will be here for let us say 90 percent for 99 percent the range would go even higher. So, we say that with 99 percent confidence we can say that 100 year flood will be within this range or with 90 percent confidence we can say that the 100 year flood will be within this range. So, that is the confidence interval which is also an important concept because there is really so much uncertainty in the data that we cannot say that there is a single value of the 100 year flood. So, this confidence interval gives us a possible range of values and we can choose a higher or lower values depending on the importance of this structure. The other type of distribution which we use commonly for extreme value is known as a log Pearson type 3 distribution. There is a distribution which is Pearson type 3 and when we say log Pearson type 3 that means instead of the variable let us say x we will be using log of x and say that that log x follows a Pearson type 3 distribution. The equation for this is first we transform x to z where z is nothing but log of x and then z will follow a Pearson type 3 distribution. So, first we do log of x and then z will follow Pearson type 3 distribution which is given as. So, now we will be doing in terms of z rather than x, but we use the equation which is similar to the previous one. We have this frequency factor mean standard deviation and this is written period t value. K z again is a function of the written period and there is a coefficient which is known as the skewness coefficient which depends on the particular distribution. We can change the values of C s and we can get this different distributions within Pearson type 3 and these skewness coefficient is defined as n number of records z minus z bar cubed. The skewness in a sense represents how far away from a symmetric distribution we are. So, if we have a symmetric distribution skewness will be 0. We may have sometimes a positive skew or sometimes we may have a negative skew. So, depending on that we will get a skewness coefficient which may be positive or negative or it may be 0 if it is completely symmetric. There are tables available for this K z as a function of C s. So, there are C s values here. There is a time period here. So, it may be 2 years, 10 years, 20 years, 100 years. C s let us say minus 3, minus 2, 0, 1, 2, 3. So, for these C s values the value of K will be given in the tables. So, K for this combination would be given like this. The values which correspond to 0 are known as log normal distribution because they have 0 skewness means they are symmetric and therefore, it is like a normal distribution. So, depending on this we can find out K z. Then we can find out z from here and once we know z then x can be easily obtained as 10 to power z and therefore, capital X which is our design value would be 10 to power z. So, these distributions help us in obtaining the value of the variate for any return period. So, if we want 100 year flood we will go here. Suppose we want to use log normal distribution we will go to C s equal to 0, find out the value of K z. From the mean of z and standard deviation of z we can find out capital Z and then find out. So, we have looked at some methods of extrapolating available data. That means suppose we have a record for 30 years or 50 years and then we want to extrapolate and find out what is the 100 year flood. We can use some of these standard extreme value distributions and extrapolate the values. We have also looked at the concept of risk reliability and safety factors and in the next few lectures we would look at how to obtain the design values for different kind of structures.


It's time to talk about not good critters, which we also played, but La Cosa Nostra, a 2014 board game that is relatively new to us. I played it a couple of times. I played it one time. Then we got Scott to play it. It's pretty good. Yeah, we got a friend who really likes mafia themed games and mafia stuff in general. I like mafia thing, but I don't really like mafia things. Right. I just I regular like mafia things hiding in the stove. But yeah, so this game, the way the fundamental concept is that you're a mob boss and there are other mob bosses. And you got people in your mob because that's what a mob is. You get some drug dealers. You got some fences. You got some, I don't know. Oh, no, it's working down just as I'm trying to do the show. You got some hit man. You know, you got all the kinds of no, you got all the kinds of like the no do gooders that you would have in your mafia crime squad. Right. But you can only have so many bad dudes, right? Working for you. Everyone else is bad dudes working for them too. You can hire more dudes with money. Sure. But you got jobs you need to do to get money. And each job requires a set of dudes. So it's like, oh, to do this job. I need a nightclub. I need a pimp. And I need a shitload of drugs. Right. So it's like, I only have a nightclub. Hey, rim, can I have your pimp? I need his help to do some work. Hey, Joe, can I borrow your night your your drug dealer? And my response might be, yes, Scott, you can use my pimp at some point in the future. If you will let me at an undetermined point use your loan shark. Yep.  In fact, give me four thousand dollars. I'm going to kill all the other pimps on the table. I will let you use my nightclub. But I understand that you are using my nightclub for a very lucrative party. You will have to give me half the revenues of the party that occurs in my nightclub. How about, how about I'll just give you a six grand cash right now? That tells me that the party you are going to do in my nightclub is going to make you more than twelve thousand dollars. So now I wanted him even more. Yep. So we say we make our deal and we're like, our deal is fine. We're both going to make a lot of money. Then it's our friend, Joey Jujer's turn.   Yep. And then the job won't happen. And the game is pretty like we're not going to go through the rules because the game is actually pretty complex and has a lot of phases. But it's not complex to play. The complexity just makes the game work because everything happens. You just keep going clockwise sequentially. And first you're taking your mobsters and some of them are smart and some of them are kind of shitty. Yeah. And they make money for you. Yeah.   And you put the mobster face up and the job face down. And then it goes around again. You say, Tommy two toes is doing this other job. You put the job face down. And some of them are better than others. Yeah. So one job might be like somebody. So the job of the like five gun strength super mobster, right? The guy in the white suit, he's doing the most important job. Yeah. But the job that the one gun guy is doing is probably like some lame old job. Some of the jobs are public. Like I might put Joey Jojo Shabadu down as one gun mobster and be like, he's just buying that nightclub legit. He is buying this for I sent him out to go buy a nightclub. That's all he's doing. He's Johnny.  He's going to buy he's he's buying a pimp. Like there's a new pimp in town. We're going to buy him. Get him. Get him into first. But he's going to buy a caravan from some but the jobs might be things like this job will pay out 20 grand, but you need a nightclub and a senator and a shitload of cocaine. Yeah, you need like the jobs that pay out a lot. The requirements are big. So the way to win this game is to get lots and lots of people to agree to do things with you. Right. Like if I make a deal with Ram and I make a deal with every other player, like one deal with each other player. I'm now doing four deals. I'm making bank, right? You want to get as many things to execute as possible and you got to look at who's winning and you you can just direct screwing right in terms of murdering people.  So it doesn't happen and make sure that you're the person who successfully executes the most deals. But everything is all that matters at the end of the game is how much money all you want at the end is money. Everything is transactual. Just mobster wins. Well, you do get some bonuses for if you still have all your mobster, they turn into money though. Yeah, everything turns into money. And for mobsters, you have killed. Yep.  It's also a side effect. Like if you've done if you're if you're in the position where you've killed more and have your guys killed less, you've probably completed more jobs. Oh, yeah. So done more stuff. So like some jobs are like whack, like literally just kill some some other mobster. Some of them are like bust somebody up, like bust up someone else's pimp. Yes. So they're like that. They're out of commission, but then they come back. Yeah. So here's a specific job. Senators party. This is what I was thinking of. It'll pay out if you get two successes, 24 grand. Yeah, there's a little bit of randomness going on. If you get one, but you can mitigate that randomness by putting the better mobsters on something or like putting a henchman down. Yeah, you roll more dice. Yeah. If it if it's only one success, 18 grand. But you need a politician, a nightclub, a pimp and a drug dealer. All the things you need for the senator's party. It's very thematic. Like the theme of this game is a plus. Yeah.  So I might I might have a job that when I trigger it, thought I need a nightclub and a pimp. What if I don't trigger that job right away on Scott's turn? He just murders the pimp. Uh oh. Now my job is going to fail if I trigger the mobster. Now I'm going to make a deal to get another pimp in a hurry. So if you're going to do the senator's party, which is worth a lot of money, you're going to want to do that when you're going to go first. So you can do it before anyone kills the senator. So a lot of games that have this, like, a sidereal confluence, let's say, where you can just make binding deals with everybody all the time forever. They tend to bog down in the dealmaking phase. Yep. This game avoids that because every player takes a short, simple action and you just keep going around and going around and going around and going around. That's a nice modern design there. No weight, no weight in constant doing. Game doesn't feel long. But it also means that negotiations will happen when they're relevant. I might make a deal in advance. I might make deals. You're usually only making a deal relevant to if it's relevant to the one simple action you're about to take. Yep. Or I have a deal in my hand and during the before it's time to put out the deals. That's what I'm like. Well, I want to make this deal happen this turn. I'm missing these two things. Let's see who has those two things. I'm going to try to deal with those two things. Yep. So the player whose turn it is to do their action is likely to be making the more urgent deals and capturing the attention to whoever they need. But simultaneously, if Joey Jojo has taken a long time on his turn because it's a complicated turn or he's negotiating a big deal, me and Scott on the side might just talking about start talking about like, hey, let's trade pimp for pimp for the future. Like things like that. Mm hmm. The mechanic of the deals is really elegant too. The if you want to use someone's thing, you get the right to put a deal marker on that thing. The other player can't renege on that deal. You know, nor like I can use that pimp for whatever I want. Right. It's like rim says, OK, you can put your marker on my pimp. I put the marker on his pimp. Now that means I can use it once and remove the token. But now everyone else when they're like killing, they look around the board and they see like a pimp with like three tokens on it. They're like, I guess I'm killing that pimp. Yep. Unless everybody pays me.  Yep. So it's it's on the one hand. It's like a good mechanic, but it's also a great indicator and really mess with people's heuristics. You don't want to like, do I really want to ask Grim if I can borrow his pimp if it already has a marker from someone else on it? No. Right. That's going to get that pimp killed. There's no point. I'll do something else. No, no, no. I want to use maybe I'll buy my own pimp from the middle, even though it costs me money. No, no, no. I want to use your other pimp, Scott, not the one you're loading up that other pimp. That's right. Give me your good pimp. That's right. And then when I go around and I say, oh, which pimp do I want to kill? Well, Rims losing the game. I don't really need to kill any of his stuff. Oh, but what's that? I see Joey Jojo's got a marker on Rims pimp and Jojo does win in the game. I can hurt Joey Jojo and Rims by attacking that dude. Yep. So despite being highly political, the politics is deeply constrained by the realities of the game and the deal making ends up being the right level of complexity. Like the kinds of deals I've seen having played this game a few times are things like, here, Scott, if you let me use your pimp, I'll cut you in 50% on the deal that that pimp facilitates. That's a perfectly fine deal, especially if you make such deals frequently with a lot of players and most of them execute, you will end up on the plus side because everyone else will have executed one deal and you'll have executed three or four. Yep. Or I might make a deal like, listen, Scott, let me use your pimp. I'll give you five grand in cash right now, because if you have a deal that pays out until you trigger the deal as one of your actions, you don't have that money. So getting cash up front might be very valuable to be able to afford the business.  Yeah, you can bash your own dude, I think. Right. So maybe not. I don't remember. Yeah, I don't remember if you could. I don't think I did. No one did. I don't think I would. I think I might have asked, but it's like, I could make a deal.  But that generally like the other thing I've seen happen is because this thing is just keeping the pimp. He does spit out money all the time and because it's a real transactional negotiating game, I've found reliably that players who act in good faith tend to get ignored by the other players or just attacked outright and revenge is a powerful mechanic in the context of this game in a way that I haven't seen play out in other games that have similar themes and mechanics. This game is just a very well crafted, perfect storm of the right mechanics, the right turn length, the right level of interaction to give you all that fun feeling of a kind of serious and realistic mafia game without getting too bogged down in details and without being just advanced werewolf. Yeah. The thing about the other thing about this game is that it's just cards. Yeah. It's just all cards. So it's real cash. It's all the cash is a little is papers. It was not the greatest. I think I might replay it wasn't. See, usually when I see paper money, I'm like, er, but actually I thought that when I saw the game set up on the table, but actually when I was playing, it was never a problem. So it feels good when you like I asked someone to kill it. I think I would still replace it with chips just because, you know, right? But I don't know. I didn't upset me like bad paper cash in other games. Well, because something felt good or leave it. Something felt good about getting a fat stack of cash and then doing the one for you, one for me. Yeah. A one to two for me. One. It's two for you. One two for me. But anyway, but the if you made chips or replace the money, you could really pack this game small. Yeah. Size. The game's already really small. Right. I'm saying size per game. Really good ratio. It also ain't that expensive. Like I forget what I'm saying. It's a lot of game in a small package. And it doesn't take that long to play, but you get a lot when you do play it. All right. Oh, yeah. And there's an expansion that adds more mobsters that are powers, which also it's very easy to expand a game. There's just a stack of cards. You just make more cards. You want to just buy this on Amazon. It's like 40 bucks. That this game is worth 40 bucks. Yeah. But it is a lot for a stack of cards, though. It's a it's a heavy stack. There's a lot of cars. It is true. Yeah. Also, I'm pretty sure at a game store, you'll find this at a much significant discount. Oh, OK. At PAX, it was on pretty big sale. In fact, I was going to go buy it. It's not. It's not a big hit game. So it gives us from 2014. Oh, so maybe the prices up is hard to find now. There's 19 in stock on Amazon, Amazon from Capstone Games right now. Expansion, there's only one left in stock. Maybe I'll just buy this right after the before I put the episode up. So yeah, this game is worth playing if you want like this kind of mafia. If you want a game that involves deal making and threatening. Direct attacking other players. Yes.   Right. It's a voter wins game, but also has significant like, you know, mechanic stuff going on. If you can make good deals like structured deals, you can use this in the game. I've seen people make pretty complex deals. Right. I'll let you borrow this guy and this guy. But then you're going to give me half the money from that deal and I get half the money from that. In one game, I sold insurance to people where if they gave me five grand, then I would give immunity from certain kinds of negative activities and protection rackets work in this game. And sometimes it's worth it to pay the protection racket. Do what mafia would do. Yeah. This has been Geek Nights with Rima and Scott special thanks to.


Hi, my name is Minda Camphouse and I wanted to say thank you to the School of Information Scholarship Committee for the Special Sessions Award for new students. I am so grateful and overjoyed and I can't wait to start the fall semester at San Jose State University as a parent of four working a full-time job and having not been in school for the last 20 years. I was really nervous and apprehensive about starting school and this scholarship means so much it will ease so much tension and stress and worry and I can't be more grateful or thankful so thank you so much and I look forward to doing new things with libraries and connecting with communities and learning all about the wonderful world of librarianship so thank you so much.


Hello, CCC. I'm Tonimir, and I'll be talking about Unhash, which are methods for better password cracking. So the core of my talk is my research into how to automate boring stuff for checking passwords on online systems and improve password cracking attacks against offline passwords. The idea is to contribute and create elements that are reusable that can be simply integrated into other tools like Metasploit or any other tool with openness and reusability in mind. So let's start first with the psychology of passwords. What kind of passwords exist? First, there are some default passwords or default backdoors that somebody was smart enough to integrate into a system like root support admin and so on. They are real passwords because people need a system to remember a password. Each of you has a system how he creates a password, how he remembers the password, right? So people use combinations, mutations, they add words, they have their own ways and they're pseudo random passwords because you cannot say that a password is truly random in case, in some special cases maybe, but something like this, if somebody can memorize this one, that's awesome, but usually I can't. And that's why we have systems to remember passwords. So the talk is split into two problems. The first problem is the boring one. So we put the interesting thing in the back. The first problem is how to check systems for default passwords on network devices like routers and switches. How to check for embedded backdoors? Backdoors, not in the case of programmatic backdoors, but backdoor passwords that somebody left in the account. And the third one is how to obtain data from automated attacks. The boring problem here is default passwords on devices. Each of you has a router at home, some other type of device. Usually they come pre-configured for a default password. That problem usually is manifested in production via some kind of a forgotten device. In a situation, we need wireless right now, just leave a device and let's get it working. In testing systems, like somebody, there's a database, there's one username and password combination that is for the database admin, other one for the system admin, and the second one never gets changed, or with sloppy configurations. That's a boring problem. The mildly interesting problems are backdoor passwords and collection of data of attacks in the wild. So if you want to collect default user and password pairs, one of the ideas I did was just simply scrape all default password lists. So I would like to first thank all the nice guys from Fino Elite, Certnet, Liquid Matrix, DexNet, Security Override, that left their password lists of default devices on the net. They're crawled and of course organized and weighted by occurrence, which is useful to test if a simple device has a forgotten password. Ordering collected data by occurrence is the only useful metric we have for analysis of default standard passwords. If you have an idea for a better one, I would like to hear it because if a username and password pair is more common, like admin-admin or admin without a password, it will simply float to the top and it will be tested sooner, so this saves time. Other metrics can be by vendor or by device, but that usually doesn't cover the use case we want to have. For instance, testing for forgotten devices, testing for weak devices, and somebody will say, why would you want to test online devices? What if somebody has failed to ban or SSH guard? Well, certainly then they didn't leave the default passwords in place. The code for this simple thing is available on GitHub. You can use it and also it is available on Metasploit framework as default user password services on hash if you trust my maintenance of this part. The mildly interesting problem are hard-coded backdoors. There is an interesting story. I obtained a HP storage device, which is a Sun storage device that after some time there was this interesting post on the net that somebody found embedded backdoors in HP storage devices. And guess what? I tried the default username HP support and the password badgers in LIT and you can, of course, access the administration part. Billy Ryoset, this year's black hat, posted that almost all airport security controls like access control devices that enable opening of doors or who is registered on the floor from the staff or like things like the morpho itemizer, which is a device that scans for narcotics or explosives when they swab you, that they have integrated backdoors. Those are the usernames and passwords and, of course, all of those devices are networked. Unfortunately, the trend of embedding usernames and passwords as backdoors is starting to take more and more more and more trends. For instance, how many of you know about the last year's leak of a popular VPN firewall anti-spam vendor, which embedded multiple backdoor accounts into their devices? For instance, the best part was my SQL database with the default user product and the password, which was blank, with a remote SSH key with remote access capability. Does anybody remember that? Well, the problem is that this is starting to be a bigger and bigger problem. For instance, those kind of devices cannot enjoy any privilege of being secure. I cannot believe that somebody would want to use them. I would like to say at least people start auditing stuff that you own. But when I say own, I don't mean own as in a bad stuff, but please first consult your lawyers, your local laws, because that can bring you all sorts of trouble. The third one for online attacks is how to collect data sets from attackers. If you want to collect data that is used on online attacks, for instance, what botnets use. One interesting part of this is you can use SSH POT, which collects data from SSH honeypots. So if you run Kip or some other honeypot technology, please ship data to them. It will be more than useful. And that enables you to collect information on public-facing brute-force attacks. And when you download all the username and password combinations that are used, you start to see some interesting patterns. For instance, very popular with a high number of hits is root and let's say, and let it be anonymous, a root or vendor name plus 46. When you Google it, it's not a default password. You cannot find it as a default password. The second one is URL of a popular ISP plus some random string. Third one is twice the research cooperation's name and some other combinations that you cannot find as default starting passwords of some systems. Now, there is an interesting question. Are those backdoors or did somebody have an inside info on how to break some systems and just simply tries everything? Let's not jump to conclusions. But I would ask all of you that are interested in such things just to take a look at the public-facing attack data and you might find some interesting stuff in there. Why are we losing on this front? Because let's be simple. We can't have one centralized repository that will collect all known backdoor data, all known backdoor passwords, unknown starting passwords. When you take a look at any tool that has some password lists for testing, you'll see that you cannot find from where that list came. Who created it? For what use case? Why should you use it? There is no exploratory data that will say from how this password list was created and there is no way you can integrate it mostly with other tools and there is no updates and so on. Reading pull requests of GitHub is not an idea when you want to see how something ended up in the one tool. This is a manual process and as you see, this part is boring. Nobody can be bothered to add passwords or maintain those lists. So I really have highest regard for people that maintain those lists because somebody has to do it and I would like to just say one thing. We need a centralized repository of that stuff. Let's maintain one repository, not six repositories of those data that are not connected. Let's enable it to be easily integrated in tools and let's enable the data to be vetted. So if you want to work on this, just contact me and we'll figure something out. From the boring part to the interesting part, how to speed up cracking for non pseudo random passwords. Let's first find out how people create passwords. That's the question. If you can see how somebody creates his password, then you can probably create a better attack. Why should you want to do that? Because that's also connected to another thing. You want to create word lists or other data that is targeted for specific class of users. For instance, if somebody wants to attack people here, he'll use security, activism, 31c3 and other keywords and data derived from that to enable his attack. So what choices do we have when we want to crack a lot of hashes or when we want to crack passwords? First is always the simple one and always the most fun one. Build a faster cracking machine, like multiple GPUs, multiple processors, building clusters and so on. The second way is to simply attack the algorithm. For instance, MD4 algorithm was known to be that you can break it with a pen on a piece of paper. Third one is bypassing, which means use something like pass the hash attacks, differential power analysis or do all the things that NSA does to simply obtain the password somehow from the machine without any interaction on cracking the real password. The third one is just on brute force the password. Try a smarter attack. Then people use word lists, then they use rule files, then they use time memory tradeoff attacks, like rainbow tables, then they use Markov attacks, mask attacks, fingerprint attacks or something else. So with a mandatory XKCD reference, people create passwords like this, write pass phrases. How can you crack it with a normal tool? It's a little bit more difficult, right? So I wanted to create something that you could write something like this, which is an actual rule file that will say, oh, just permute all four words from a word list and let's see what comes out of it, right? The idea behind that was a data-driven approach. So how do you learn how people create passwords? First, you need data and data is, believe it or not, hard to find on the net, at least quality data. Some of the nice people on the internet contributed a lot of good research data, which means clean password dumps with occurrences. There should be raw hashes or all plain text, not just a snippet or some fished parts, which means that your choices are really limited. The learning set was 32 million passwords, that means 14 million unique passwords, which all Cyrillic, Greek and other lettering was dropped out because they didn't want to include those languages in the learning process because they don't know them real well. And the three biggest lists that were available was Rocku, PHPBB and Yahoo lists. Anybody that does password research knows those lists as popular public lists. The verification data was two sets of lists. The first one was LinkedIn, which is the second biggest leak after Rocku, and the third one was from EliteHackers, CarderCC, Hack5 and MySpace pages. The idea was first, let's use a machine learning algorithm and figure something out, which failed miserably. And after that, I created something called sieving, which is the following idea. Just create a classifier for each identified password class. You cannot classify all passwords. You can't figure out how all passwords are created. Let's just see a subset. Create a classifier that can identify a subset. And then for each of the classes, create one classifier, decide which ones are probably the most correct ones, which ones were right, and then dump everything in a format that enables analysis. These are some password classes that I identified. Weak patterns like just repeating some letters on the keyboard, keyboard patterns. Some really passwords that I thought that were really strong were really keyboard patterns. And the pseudo random password you saw on the first three or four slides was actually a keyboard pattern. It looks like a really strong password, but it's created by just following keys in a sequential way on the keyboard. There are also, you can see all kinds of stuff, mutation, that's anything that changes a word, change O into zero, change I into one, and so on, or combination. Just combine a few words, combine a few elements, or just simply use elements. When you see each of those classifiers can be iterated and you can tear passwords apart. The second problem is how to identify languages. For instance, if you say security, you know that it's popular in all languages, but how can you say if you say something like plaza, which is what is that? How can you identify that word? So, if you want a good word list, you simply can't rely on things that are available. First off, Google ngram corpuses are too big, same thing as with hunspell and aspell, which fail on one thing. People don't create passwords by using grammatically correct sentences. They use all kinds of stuff. For instance, info about things from a popular series, like names of characters, names of places from popular books, or so on. So, one ideal source is Wikipedia, because Wikipedia has all kinds of articles that are not really bound by language, right? So, I dumped English, German, Croatian, French, Spanish, Italian, and Dutch Wikipedia, and that created a big password list that can be used to identify characters. And to improve on Wikipedia, Google has, Wikipedia has about 84 main categories that I identified, and I used those categories to scrape 51st results from Google to improve on the Wikipedia dump. To enable you to create password lists from Google searches, I created a small script called gword list, which will scrape top and Google results based on keywords or Google dorks or Google hacks, how you want to call them, which you can enter. This will also create a word list based on occurrence that you can just simply pipe into a password cracking tool or anything else that you want. So, why do you need this? Because usually, if you use that, it will be more effective than if you use a generic word list. If you're attacking security researchers, you can use, for instance, that data, which will have specific information or specific keywords that they might use, right? And of course, you can recurse. You can take the resultant list and then recurse on that and create an even bigger list. That's an iterative approach. First, you ID known elements with the sieving algorithm. You create a rule that's the composition and you collect all the data, how a person created the password, what words did he use, what numbers, what replacements, what mutations, everything. If you have something that is unknown, store it, analyze it later and create a new classifier. And when you're done, you have a set of classifiers, an algorithm that can classify it. In practice, this looks like this. So, you put something that looks like a strong password and it will tell you it's a keyboard pattern. If you look at your keyboard, you'll see that somebody just typed sequential keys on the keyboard and pressed shift. For instance, other weak patterns can be identified and you can see how people create them. Or a complicated password, something like this, will be disassembled and you will identify all the elements of the password, which can be written down as a rule for unhash like this. So, this here that you see is a genuine rule for the unhash tool that can be executed and it will result in using, for instance, numlen4 is a dictionary of numbers length of four, dictionary length 11 is all words that have length 11. R is replacement, replace i with one, replace o with zero. And strlen4 is a dictionary with all strings of length four. In the data mining process, when the sieving algorithm classified the passwords, I collected all the data. For instance, all numbers that were four in length were collected and after that, you have a word list that is based on occurrence of the top used numbers. What's the probability that somebody will use the number? Rules are supposed to be expressive in unhash and you can use any combination of word lists, strings, generators, any combination that you want to generate candidate passwords. You can add replacements, you can add uppercasing with or without permutations. For instance, if you want to have a password and if you want to change s to five, will you change all? Ss, will you change none? Will you change only the first one or only the second one? If you want to permute it, you don't have to write multiple rules to describe such process, you just have to say one flag and it will permute it to all possible permutations of replacements. And also you can write your own, like Python functions or expressions. The benefit of unhash rules is that you can use any word list you like. You have ones that are shipped with unhash that are generated by data mining on the 32 million password set. You can use Google word list to create additional passwords that are made by scraping some keywords from Google or you can use your own. I suggest you combine the first two methods, scraping results from Google with G word list and using the ones that are shipped with unhash. The use case here is pretty simple. The tool was created to be able to execute rule files and you can pipe it into any popular password cracking program like John the Ripper or hash cat or something like that. The interesting part here is if you look at GitHub, which I think some of you are already doing, you'll see that probably it's the most hideous code you have ever seen. But if there are some Python ingests here that know how to make it faster, I'm open for suggestions. The interesting part is that I used PyPy, which thanks to the creators of PyPy, that actually enabled pretty much faster runtime for the algorithm. Some results are, don't get it, I'll tell a few words later about comparison of results. If you use a normal brute forcing program, it will usually uncover some amount of passwords in some amount of time. I used 24 hours as a measuring element. And what did I do? I used unhash with Google word list and used Google word list to generate a word list that is scraped top 10 pages based on these keywords like LinkedIn, like business recruiting, networking, job contacts. And used that list in combination with the data mind list from. And you can see that you can get at least 20% better results if you add that approach. Why? Because brute forcing enables you to uncover a lot more short passwords that are complex. So this tool will fail if you have a password that looks pseudo random. But if you have a password that is based on combinations, mutations, like the ones I showed before, then that kind of approach really helps. If you see here, longer passwords are pretty much better uncovered by this approach. First off, don't think that unhash is better than John the Ripper, Hashcat or Hashkill. Those tools are awesome and Solar Designer, Hashcat team and Gateway from Hashkill do an extremely awesome job. Just use the right tool for the right job. This here sounds maybe a bit complicated, but you can find on GitHub the example files and all the rule files and you can just simply execute them and it will do its magic. Those are two different use cases. Still, if you have a GPU cluster, going GPU is the king. Or if you have a FPGA border or something like that, if you want to test for slow hashes like Bcrypt, Scrypt, PBKDF2 and you can't afford a large amount of comparisons because the cracking process is slow, then try this approach and maybe it will help. This also helps with passphrases. You know a part of a passphrase, you know how the passphrase was generated, how many words does it have, you can pretty much simply brute force it then. Or in custom testing or in custom tools. One thing I see that data-centric approaches are really interesting. Try to experiment and build something interesting. When I mean build, please don't be evil. It's available on GitHub. It's simple to use. Try it out. And of course contributors and researchers are welcome. And of course I was a bit shorter so you can have more time for Q&A because if somebody is really interested in this topic, I think he has some questions to ask. So you have my Twitter account here, you have my email, you have the GitHub link and I'm open for questions. Thank you very much for the talk. Four questions, please line up in front of the microphones in the room. Are there any questions from the internet? No, there are no questions from the internet. Okay. Questions on comments in the room? On a five. Yeah, sure. You said that you had trouble using a machine learning algorithm. I'm just wondering if you could explain or elaborate why that was. That was because machine learning algorithms were things I tested weren't really well suited because there are really many classes of passwords. So it's the classic problem of overtraining. You train your machine learning algorithm to figure out one part of the story, like one class of learned passwords and then it will fail on the other one. The one thing you can do is try something like random force but they didn't give enough good classification. Or the second thing you can do is something like committee of the machines. Saving algorithms is pretty much inspired by committee of the machines and random force approach because it is much simpler to build a classifier for a type of password because you can then also get contributions on types of how people create passwords. You can get pretty much more complex ideas into how to classify passwords. For instance, what if somebody creates a passphrase and then takes every first letter from the passphrase? How will you classify that? But with the saving algorithm, you can classify that pretty much more easily because you can write a classifier that will give a result for that and give you a probability score. Is it better or is it not better than something else? But does your saving algorithm assume that the classes of passwords are independent? Yeah, the password classes are independent because somebody just combines elements, somebody uses keyboard patterns, somebody uses combines and replaces characters, somebody just uses a word and replaces a couple of words. Those are all independent classes. That's why it's pretty much simpler to do it. Microphone 1. Hi. I was wondering if Unhash also takes into account specific target attributes like a native language and other things. You could also use a tool to scrape the target's website or something like that to add it to the word list and make it more target-specific. And is it capable of doing that? The first part is the machine learning part, which I did, was based on these dictionaries. English, German, Croatian, French, Spanish, Italian, and Dutch. So the rules are derived by those main classes. The scraping you can do with the G-word list tool that is enabled. You can write Google Dorks. For instance, you can write N keywords and add site column something like NL. And you can add multiple entries of keywords with site NL. And it will just tell him to scrape top 100 sites. And you will get a word list that is only Dutch based for those keywords. It works in UTF-8, so you won't have any problems. You can use the resultant word list from G-word list with the Unhash tool. So that's the point that we want to take here. You can scrape the results from the net. You can scrape it with G-word list that uses Google. You can use all Google Dorks that you want. For instance, reducing search to a certain country, reducing it to a set of keywords, telling it, okay, scrape top 500 sites, and then you can use it on the rules that are freely available. And one more thing about implementing password policies, you can see on the site, like they have this policy, like you need to use one capital, one special character, or things like that. This is also implemented or should we just write regex to filter out the passwords in the list?  It is pretty simple to see which rules are not covered by the password classes or by the password policy. You see by the length, you can see if it has a replacement or not. If it requires one special character, you can reverse-grab only those rules that don't have a specific character, right? So you can pretty much custom design it for any kind of attack you want. That was the point behind Unhash, that you can adapt it to any kind of use case that you want. It's, as I said before, it's not a replacement for John, the Ripper, Hashcat, or anything else. Those are awesome tools for brute-forcing and for their right. But this is more interesting if you want to do an attack on a passphrase, on a slow hash, if you want to just test a specific set of words, like you have a word list that is medical in nature, and you want to, you have a pen test on a, for instance, on a medical facility, and you want to use medical-specific words. You can just use g-word lists to scrape them and then use it with Unhash rules, right? Yes, it would be nice to see a web-based implementation of a word list generator. So you're telling me I should make a cloud cracking service, right? Or a cloud word list generation? A word list generator. Okay, good idea. People if you would like to leave the room, please do that more quietly. It's not that hard. Microphone 4, last question. Thank you for the talk. A quick question you showed for one instance, a performance gain of 23%, have you run any other instances and do you have any metrics for possible performance gains? Okay, the possible performance gains are really hard to measure correctly. That's why I said don't think this is better than John or Hashcat or anything else, because it depends on the password list you gain. If some, for instance, those were general password lists, and as I said, the research data is hard to obtain because there are not really large password lists with occurrences that have an equal number of good users that are, how should I say, conscious security conscious that use hard passwords and a normal type of users. So I use 24 hours as symmetric because you need to do a lot of experiments, each experiment lasts for a day and that's probably the time somebody pen testing without a cluster will just try to without all weak passwords, right? That's why I used big password lists in testing. For instance, LinkedIn, for instance, for instance, Yahoo password lists, elite hackers and so on. Thank you. Thanks a lot, Tony. Please give them, give, give a lot of applause for this talk because I actually thought that was pretty great.


Hello everybody so great that so many of you are joining already for the boring opening ceremony. I hope I can make it less boring by telling you a bit about how this is all going to work in this online setup which we certainly didn't expect when we were planning for this conference. And I will also introduce what do we hope to get out of the Solidity Summit as well as who's on the team and everything you need to know to survive the next two days with us. So I think we are good to go. That means I will share my screen with you now. And we can get started. The first life hack for everybody who will present after me is that you click share your screen and then you click on your camera again so that you can see both the speaker as well as the slides. All right. Hello everybody. I'm super happy that you are all here. My name is Francisca and I will be your host and moderator for the next two days except for the open discussion sessions where we will have expert moderators leading you through the sessions. We're super happy to have you all here. This is our very first Solidity Summit. It was expected to be an in person event in Berlin but due to the corona crisis which is happening right now obviously we are not hosting in person conference but we switched everything to be an online conference. And yeah, we hope that this will work out fine. This is certainly new to run for us to be hosting an online conference, but we are super happy to do this together with the interspace team and we hope that all of you will have a smooth experience. In order to have such a smooth experience I already sent you some instructions, but I will also walk you through how everything will work again so that everybody knows how to get the best out of this conference. So firstly welcome to everybody and why are we even hosting this event. So obviously we want to get people involved and interested in the Solidity language together and want them to be able to exchange about the topics that they are interested in. We certainly want to exchange experiences and discuss language design and tooling together because the developers of the Solidity language certainly have some questions for you who are using the Solidity language on how the language design should be moving forward. So I think that those discussions will actually then result in improvement proposals, which will definitely impact the actual implementation of the Solidity language and you will. And we don't only want to foster the communication between the Solidity team so us and you guys, but we also want to foster the communication between teams that are working on, maybe the same or similar topics. So that you are aware of each other and maybe you can collaborate moving forward. But ultimately also we want to obviously identify needs for the smart contract ecosystem for Ethereum and derive steps forward together. So yeah, first of all meet the Solidity team. Those are all the people who are officially working on the Solidity language at the moment but obviously this is an un exhaustive list we have so many open source contributors that are either contributing by reporting bugs or by requesting features or by just joining the conversations with us so this is just the official team that I'm listing here. And if there are some people from the official team in the in the jitzy room right now, feel free to turn on your camera really briefly so that people can see you. Yeah, hey guys. Also there's all of the guitar handles from the team there so in case you have questions, you can just add them in the guitar channel and then you can discuss with them over there. So now how to Solidity Summit. This conference is as I already mentioned powered by interspace, which is a new open source initiative to basically bring online conferencing to the blockchain space and maybe even beyond let's see. We've produced some great events in the past, for example the noncon, which was I think the first conference where interspace was used and then also a virtual hackathon Eastern Turin and I believe there will be much more events coming up in the future. So what does this mean this conference is facilitated by a couple of tools, which I hope will make the collaboration and interaction between all of us during the conference. It's easy and easy and will almost feel as if we are all in the same room together. It's powered by jitzy mostly jitzy is this open source conferencing tool and video conferencing tool which we are using right now. So there's one main conference room. This is a single track event so everything will happen in this main room. There's the lobby room which you can use to hang out for example in breaks if you want to try out jitzy if you want to just chat casually with people who are hanging around. You can also create your own breakout rooms. So basically if you want to take a discussion further and it's not part of the agenda anymore, you can create your own breakout room. And then you can distribute the link of this breakout room to the people who are interested, for example, using the guitar chat. We are using YouTube for the live stream so if you are watching via the live stream right now already you might know that we're using the Ethereum Foundation YouTube account and you can find the live stream link also on the interspace border. So we're using the guitar chat, solidity guitar chat for the Q&A sessions and for coordination. So in case you have any question for a speaker during an open discussion session or whatever, feel free to just put your question there and the moderators will relay the questions. We have agendas in a Google sheet as well as also in a Google calendar so if you want to create your own, oops, if you want to create your own individual itinerary you can do that by going to the Google calendar entry and just copying the agenda points that you would like to see. In the open discussion sessions we will use HackMDs to take collaborative notes and those HackMDs are also always listed in the agenda points. If you're participating today or tomorrow or even both days of the event you are also able to claim a proof of attendance protocol token, which will verify that you attended this event, and for very ad hoc coordination in case all other people have just listed above fails us. We also have a telegram group for emergency coordination in case this room breaks down or whatever needs to be coordinated. I will be sure to put it in the telegram group as well as in the guitar channel. So that's basically that and you might ask yourself, but where do I find all of those links crazy right. So, don't you worry, just go to interspace.solidity.summit.etherum.org if you aren't there already. And there you will really find everything you need in one place. And as I was outlining before you for example find the collaborative notes in the description of the open discussion session, if you click on agenda. So how does this jitzy thing work. I have made a screenshot when I was doing a conference, all on my own. So if in case you open the jitzy room in another browser tab which I think is the recommended option to do so if you are for example a speaker. It might look similar to you. And this jitzy is very easy to use actually super straight forward. There's a few features you can share your screen sharing your screen is on the left in the corner and on the bottom. You can raise your hand which might be a yeah excite surface raising his hand for example you can do that in case you want to speak so that in an open discussion session or whatever moderator can see you want to say something. So if you have the microphone settings in the middle where you can also choose which mic you would want to be using. You can quit the meeting in the middle and you can also change your camera settings there. And then on the right hand side on the bottom you have other settings. You can change the view, and you can click on the I to know which is the room you are else but you can also know that just by seeing your browser tab. Yeah, and you can also set yourself a name and as I see almost everybody who's in the room already did so. So basically by clicking on your video or by clicking on the settings settings you can give yourself a username. But this sounds like everybody already did so. I wanted to mention that you can click on the help button in the sidebar of the inner space, and there's a description of the jitsie interface if you didn't catch everything that that Francie said. Absolutely exactly. And then what's the recommended setup. So jitsie has this maximum participants of 75 participants per room limit. So you will reach this limit, but to ensure a smooth experience for everybody. In case you don't plan on actively being involved, aka speaking or discussing actively in a session. I would always recommend you to go for the live stream, because it also saves your CPU. Obviously, you can switch between live stream and participating in the jitsie room directly depending on the session. If you want to listen in, you go watch the live stream. If you want to really actively participate, you join the room. So feel free to, you know, jump around and switch around between those two. Talk to us in Gitter for Q&A. And yeah, just FYI, the live stream by nature has a little delay. So if you really actively want to discuss in such an open discussion session, I would recommend you to join the jitsie meeting because otherwise you will be one minute late more or less to the discussion. Yeah, then how do you survive 9 hours online per day? Obviously, you don't need to participate in every single session, but I would love you to do so because I think it's all really relevant and interesting content. So yeah, just a few quick reminders, which you might already be aware of, but I just wanted to let you know again, please don't forget over all of the school conferencing to drink something depending on your time of the day, coffee, tea, club matter, beer, water, whatever you prefer.  Don't forget to get up and stretch your legs sometimes and also have a screen break for your eyes. So just watch out of the window or go watch your kids if they are playing around at home at the moment during Corona. And yeah, you can also open the YouTube live stream on your phone. So in case you need to go somewhere or just want to wander around a little bit. That's also an option. Okay, last but not least, let's have a quick look at what is going to happen today together at the agenda. So this is the agenda you might be familiar already with it. We are currently here and doing the opening right after that Chris will present us the Solidity 2020 roadmap. After this, we have a nice talk coming up from Richard and Shang Wei about the key framework and case Solidity. After this, we will look at a source level formal verification tool together with Akash. Then we will have a presentation on set Torah. Then we will learn something about the type and chain lens, which are two tools developed by Loredana. Then we will hear about some thoughts on language design and fragmentation from Alex. And then in the afternoon, we start with the open discussion sessions. They always start in the afternoon because we want to give also the Americas a chance to participate in those. So we will start with the first discussion on safe math by default moderated by Chris. Then we will have a break. Then we will have a discussion on libraries 2.0 moderated by Chris again. Then we will receive a nice introduction on Yule plus from Nick Dodson. And after that we will also have a discussion on new food features for you moderated by Alex. Then we will learn something about the sole compiler. Then there will be another open discussion on upgradable contracts. Another short break. Another discussion on fixed point types. Then tracking mapping keys with a travel debugger by Harry and consecutively also travel debugger demo and followed by a debugging data discussion. You might notice that one talk is missing today. Unfortunately, one of our speakers fell sick. So we will not hear about curiosities and solidity development from Nick. And that's why the fixed point types discussion has been moved up a little bit. And that's almost it from my side so long. I hope you will have fun and enjoy the upcoming talks and discussions. And please feel free to ping me with any feedback you might have. I'm underscore francy high on Twitter and francy high everywhere else. So have fun and enjoy and with that, without further ado, I hand the mic over to Chris.