We have been talking about the permeability and flow of groundwater in porous media and we have been discussing about this problem that there is a porous media encased in a glass tube or a control volume and then we were trying to analyze this problem as far as total head is concerned, pressure head is concerned and the elevation head is concerned. So, as far as the elevation head is concerned that is easy to obtain. So, if I say point elevation head this is in meters, the pressure head is also in meters and the total head this is also in meters. So, point 1 was quite straightforward, we have already proven that between point 0 and 1 there is not going to be any flow because the total heads are same. Point 1 if I assume the elevation to be matching with the free water table and suppose if I assume this as x I am sure you must be realizing that the most of the confusion is because of selecting the datum. So, you have to add this x to this, a better way of doing this would be if I shift the datum at point 2 alright. So, the moment I shift datum to point 2 life becomes simple at point number 2 the elevation head is 0 and what about the pressure head, this is exposed to the atmosphere.  I will tell you what is the fallacy which we were discussing last time. At point number 1 the elevation head is L, the pressure head is L plus H no sorry H and hence the total head is L plus H. So, the hydraulic gradient would be if I compute the hydraulic gradient this will be delta H upon L which is H 1 minus H 2 over L this is nothing but L plus H over L alright. The porous media is described by these parameters that is porosity where they show gamma D is known, SR is known, specific gravity is known and so on. Now the moment you shift point number 2 to let us say 2 prime what is going to happen. So, this is where the trick is I think you should be careful when you are writing these heads if I shift to the point 2 what is going to happen the elevation head at point number 2. So, this is second situation when I have shifted when the data means that let us say A, A at point number 2 the elevation head is X. Now what you have to understand is that this is directly exposed to the atmosphere because of its connection with the free water table. So, if you put a piezometer over here it is not going to get you know rise up to this point is not correct why it is not correct it cannot get lifted up to this point what is the reason because you have a situation where the boundary condition says that this line is exposed to the atmosphere and hence the total head at this point is going to be 0 clear. Now those of you who are confusing this with the situation which we have been dealing with in the past when we are considering different type of layered system and I had asked you to find out what is the state of stress at this point and find out the UW clear and suppose the water table is up here. Now in this case I can install a piezometric tube over here and this is what is going to rise up to this point and hence the head is going to be this much. So, what is the difference between this situation and this situation anybody in the class and that is what you have to understand here the boundary conditions are different as compared to this is a continuum I hope you are getting this point which is not exposed to the atmosphere. So, if you install a piezometric tube over here this will go and meet the priatic surface but in this case at this point and this point the atmospheric conditions prevail. So, it is a good idea to shift the datum to the bottom most point of the sample it becomes easy to go ahead with the analysis this is part clear this is the discrepancy which I wanted to discuss in the last lecture this is part clear. Another way of doing this would be if you say the elevation head here is x if I install a piezometer here what is going to happen this is going to drop to minus x and hence the total head will be equal to 0 I think the better way would be to shift the datum and do the analysis is this part ok. Now what I will do is I will go ahead with the another situation yeah. So, see what we have done is we have established here that the pressure gradient across this sample is delta H upon L and delta H upon L is nothing but H 1 minus H 2 upon L H 2 is 0 once you shift the datum over here H 1 is equal to L plus H divided by L is the hydraulic gradient. Now this gradient is going to cause the flow through the porous media. So, this yellow arrow shows the flow through the porous media where point number one elevation head see if you fix the datum over here. So, this is the elevation head at point 1 this is L if you put the piezometric tube over here it goes meets up to here so L H plus clear. So, pressure head is H at this point elevation head is L total head at this point is L plus H fine have you understood the discrepancy ok. Have you understood this part and this part the difference between these two there is no boundary condition which is enforcing the atmospheric condition. Now if I do engineering how do you do surgery of the system suppose if I include a layer of a porous media over here nowadays directional drilling is possible you can drill laterally alright a good example is our Santa Cruz airstrip where the lateral drilling is done to take all the pipelines under the active runway and then this I can connect to the atmosphere clear. So, what I have done I have created the same situation as this over here now at each and every point here this is what is known as a filter I have included a sand layer in case of your gas hydrates you are doing hydro fracturing and you are making the system more permeable this is connected to the atmosphere and hence the pore-water pressure at this point is going to be 0 then this situation identical to this situation it is ok. Any questions is this clear? So, this is what the engineered system is I hope this point is clear now let us do a bit more of extrapolation of the situation with which we are dealing here. So, what we have done is we have established the flow condition over here and why flow is there because of delta H by L hydraulic gradient whatever drop in column between 0.0 to 1 is going to take place this is the discharge clear whatever flow is going to take place from point 1 to 2 because of the presence of porous media would be seepage or flow through porous media now when the discharge or the flow is taking place through or the seepage is taking place through porous media what we do is we introduce the component Vs this is what is known as the seepage velocity I hope you understand the difference between this. So, this is your Q upon area of cross section which is nothing but the capital V and what is going to enter into this system is because of seepage this will be the seepage velocity now it so happens the seepage velocity is equal to discharge velocity upon porosity how suppose if I consider a control volume and this is in the third dimension like this these are the pores and rest of the portion is solid soil mass. So, these are the pores suppose if I say area of the pores is Av air voids and area of the solids is As so the total area will be equal to As plus Av is this okay now imagine if the discharge is taking place in this direction which is equal to Q what is going to happen the certain amount of water is going to enter through the pores not the entire flux because the solid portion is going to put lot of resistance in the flow and that is what we have defined as the hydraulic conductivity k coefficient of permeability is this okay. So, if I say that the Q is equal to As and As multiplied by V this is okay and this will be equal to area of pores multiplied by seepage velocity this is the continuity. So, that means what I am saying is Vs equal to As upon Av multiplied by velocity V sorry which one on this side yeah there is the yes you are right. So, this is no I can deal with a solids also no okay okay you are right. So, let it be like this now what it indicates is that the seepage velocity is always greater than the discharge velocity which one is faster the water molecule has to travel from one point to another point either in the microscopic lens or in the macroscopic lens. So, L will be the macroscopic lens to maintain the continuity now this path is going to be tortuous that means this path is not going to be straight line this path is going to be something of this sort alright. So, this is 1 and 2 which I have represented here as 1 and 2. However, the linear dimension is straight so imagine if the discharge is taking place through 0.1 and 2 the seepage is going to take place through 1 and 2 but to maintain the equity or the continuity the time is same the discharge in the form of seepage is going to be same as the discharge in the form of the discharge and hence this is what comes out to be from this analysis see when the water is entering from 1 to 2 there are 2 possibilities either the water moves through the macroscopic lens which is the physical dimension of the sample or through the interstitial of the soil sample which are quite tortuous in nature that means in the same time in the same distance is to be covered the velocity through this is going to be more than velocity through this what it indicates is that the velocity of discharge sorry flow of water is going to be lesser when you have macroscopic length but when the discharge is taking place through the microscopic lens which is the tortuosity the seepage velocity is going to be higher than the discharge velocity. Normally how would you maintain the continuity suppose if I give you a cylindrical sample and if I say that the Q is moving from here and Q is coming out there is no loss of fluid clear imagine like this. So, this is what actually we are trying to analyze this is another attribute of the porous media which we are going to use over here. So, when I say the flow has initiated because of the hydraulic gradient there is going to be a velocity component V s. So, V is known capital V a porosity is known V s is known and you can show how V s is changing with respect to V. Now, because of the virtue of this porosity as I said V s is going to be more than capital V the discharge. So, this is what we have to include over here is this part ok. Now, if you go back to this equation where the Darcy's law says Darcy's law is valid for discharge velocity. So, we say that discharge velocity is proportional to hydraulic gradient correct or this is equal to K into I what is K coefficient of permeability what is coefficient of permeability this is the resistance offered by the porous media to the flow of fluid is this ok. The units are in centimeter per second. So, there are 2 definitions which are prevalent in this direction and this context not direction the one definition is how do you define permeability. The permeability is defined as the rate of flow of water under laminar conditions through a unit cross section cross sectional area of a porous medium under a unit hydraulic gradient hydraulic gradient is I and standard temperature conditions. So, basically permeability is the rate of flow of water under laminar flow conditions through a unit cross sectional area of a porous media under a unit hydraulic gradient and standard temperature conditions. That means if I put I equal to 1 the discharge velocity is equal to K. Now, most of the time what is done is that there is a relationship between coefficient of hydraulic conductivity and I think last time I told you to stick to small k because I wanted to preserve capital K for some other notation. So, this is this capital K is known as intrinsic permeability just concentrate on the board for a minute try to understand what is this concept which I am introducing over here. The hydraulic conductivity K is let us say two variables A multiplied by B capital K depicts the porous media. This is the fundamental characteristic of the porous media and that is why we call this as intrinsic permeability. Intrinsic means fundamental nature fundamental behaviour it is not going to change. However, B term so this is the PM and what is B? B is the fluid clear. In other words hydraulic conductivity or coefficient of permeability is a sort of an interaction of porous media with a fluid mathematical definition would be this is how you define the interaction between solid porous media and the fluid clear. So, the mapping is between the material porous media and the fluid what is rho into g density unit weight clear divided by mu viscosity of the fluid. So, attributes of the fluid attribute of the porous media conductivity clear. I hope now you can understand easily what I have to determine not capital K I should be determining sorry not small k I should be determining capital K because this is what is corresponding to a porous system fluid can be anything it could be water it could be contaminated water it could be hydrocarbon it could be gases it could be bacterial suspension whatever clear. So, this is what please remember outside mean okay yeah so you are talking about this portion that means there is no you are right so there is no porous media over here this is the discharge this is the seepage this is again the discharge you are right in this portion in this portion in this portion viscosity is not so important because we are not talking about the seepage taking place to the porous media this is a discharge only. So, suppose if you remember last time we had derived some expression following head test so as the time passes by what is going to happen this column is going to decrease is it not. So, h is decreasing and then we had taken in minus del H power middle t. So, this is a free discharge this is the seepage that is the difference and seepage is one minute seepage is attributed to all these characteristics this is a glass tube and I am filling up the sample over here this is the soil sample porous media this is the point number 1 this is the point number 2 clear and this is kept in the water bath come out of all confusions simplify things treat your this is H water table this is the water level here and the length of the sample is L shift the datum to this point number 1 to get rid of all this confusions.  So, that means the piezometric head at point 1 is H the piezometric head at point 1 is going to be rising up to the height H to equilibrate with the atmospheric conditions. At this point now the elevation head has become 0 what is the pressure head because this itself is atmosphere is 0 finished total head at this point is 0 total head at this point is going to be L plus H coming back to your question head across this and this is L plus H the gradient is L plus H upon L which is going to cause the flow into the sample otherwise this is free discharge this is also free discharge agreed this is part clear by virtue of being a porous media the void ratio density soil property specific gravity all are defined what you are ensuring is this is a steady state and hence the sample is fully saturated if the sample is not saturated then the problems will occur. We always talk in terms of meters of the pressure head if you remember the pressure head at this point is the height of the piezometric rise in water so this is going to be H clear. Now suppose now I am going to play with the boundary conditions so please have a close look at what I am going to discuss this is my tube and this is the porous media fine. Now what I will do is I will connect it to a water bath is this ok so all those who are asking this question of the surface being exposed to the atmosphere I think now you are realizing something what I have done have I done something is substantially different or is the same thing what I have been discussing until now then you have understood what I am going to talk about it does not matter where I connect this point to. So what I am basically doing is this is the head difference between the two water levels and the logic says that the hydraulic gradient is the one between the difference of the energy state between the two surfaces and that is what is going to cause the flow suppose if I lower it down a bit what is going to happen the discharge is going to place from take place from 1 to 2 the moment I have lifted it up what is going to happen discharge is going to take place from 2 to 1 is this point clear simple mechanics have you understood this I can change the flow direction through the porous media just by manipulating the H H could be positive H could be negative so if I bring it down and if I keep this system or what you call it as a reservoir somewhere here what is going to happen now the delta of the free water surface and this free water surface is going to cause the discharge to take place let us prove it you can fix the datum somewhere work it out now when you go back today please do this with different different datums you know the total energy here to do not total energy here and you can prove that the discharge will take place from 2 to 1 is this part okay have you understood this now suppose if I ask you to draw the free body diagram of this element there is another manipulation first understand the manipulations different type of manipulation which I can do I can replace this porous media by a multi layered system agreed that means I can say this system has permeability k1 this has k2 in earlier case you must have realized that the dissipation of the hydraulic gradient is going to take place between 1 and 2 and that was linear now because of introduction of k2 and k1 what I have done I manipulated the system much more I am tending towards more natural processes and let me complicate it further k1 with porosity even porosity e2 e1 e2 gamma d1 gamma d2 g1 g2 what does nature is nature offers you same thing what I have created I have created a multi layered system 2 layered system where now you are going to be in a difficulty but it is simple now what I can do is this 1 and 2 I can replace with point number 3 and I can still go ahead with the same thing what I require is one equation of continuity that means this q is going to be the same as this q is going to be the same as the q which comes out that is it in other words if my q remains same I will say k1 multiplied by h1 minus h2 over l12 this will be equal to k2 h2 minus h3 is this correct l32 and area of cross section so area of cross section you multiply fortunately the area of cross section remains same so that does not come to picture another manipulation what I can do I can reduce the area of cross sections so I can create a sample like this the first layer and then second sample is sitting like this ok so this is your first sample this is your second sample or a multi layered sample whatever is the concept clear as far as the total head pressure heads are concerned they remain same only thing is now you are introducing a point at which the state of energy is not known which can be solved by using the continuity principle yeah so this q is nothing but your a into v alright so this v is here and this v is here also that is it this is ok concept clear look at the pressure gradient that is important so at this point and this point the pressure gradient was straight line earlier now what has happened the slope has got changed why because you introduce another point another property here so earlier this was like this now who knows it might have become like this and you simply create a datum over here now you will not be confused at all I hope all this is gone is the same thing in the previous case what did I do I kept me how does it matter whether you are keeping it in a tube connected to this or whether this whole thing is inside a water bath it does not matter if you are putting vacuum over here so this it does not matter keep the datum as it is pressure at this point will become minus of your value of vacuum which you are applying that is it is this ok so if this point is being sucked with a suction pump what will be the pressure at this point minus that value so what happens to the hydraulic gradient this is a good question so if you suck something what happens now he is getting realization of what we are discussing the discharge increases so minus of this thing and this minus of minus of this thing that means the hydraulic gradient is going to become higher clear so v is proportional to i so that means the movement of the fluid through this porous system will get enhanced now let us manipulate this a bit more direction of flow is clear to all of you this head is causing everything to happen now suppose if I draw the free body diagram draw it how many forces are acting ok you let us do it together so is there something acting from the top how much is that this is a sample what is the total head at point one H total head this H is not directly connected to point one there is a dampener in between which is soil mass clear so that means at this point elevation head is known L if you are putting the data mat over here at this point the pressure head is Z so total is L plus Z into gamma W into A this is part clear what is happening here this point what is the elevation head 0 what is the pressure head if you connect this to the piezometric column what is going to happen L plus Z plus H this surface is connected to the atmosphere here so this is L plus Z plus H gamma W into A what is the other force which is acting on this element very good excellent so what is the weight component sorry yes so weight will be equal to L very good L what gamma what type of gamma sorry gamma dry whole thing is put in a steady state condition how can this be dry yes do not do these mistakes these are basics alright so gamma saturated and A nice are you happy with this what is the other force which is going to act on this element I said something if I raise this and if I bring it down I can modulate something yes you are right not gradient force so that means yes he is right have you understood have you seen a dashpot ok if you go to the hospitals you must have seen these salines so what do they do they put it on a stand if you are lying on a bed over here here so what you do normally they put a saline here correct this is the same situation what what not normally sisters will do they will modulate the height of the saline so somewhere they will fix it a small kid they cannot put it 1 meter high why imagine the hydraulic head now we are going to see what is going to happen we are going to model that thing this is ok so if I lower it down and bring it up I can create a force which is going to act on the system so this is your seepage force where the seepage force is acting what is the plane on which the seepage force is acting you have to look at the figure wherever your data means at this point why because the entire pressure head and elevation head is acting at this point clear so truly speaking what is happening you have 1, 2, 3 forces and they are being balanced by f and this what is known as seepage force now so this is the seepage force that video clipping was regarding what quick sand we are going to create quick sand condition how so little bit more of understanding for all of you you know let us put it in components let us filter out the seepage force in the form of hydraulic gradient so what I can do is I can say l plus z into gamma w a of course weight is acting and this becomes l plus z gamma w a plus if I write like this is there any problem what I have done I have filtered out the effect of the hydraulic gradient h h was the driving force which is causing seepage to occur from point number 2 to 1 because this is how the flow is going to take place so the flow is now taking place from 2 to 1 positive gradient is here less gradient is here alright so suppose if I write this expression f is equal to now this is what is known as seepage force normally we define seepage force per unit volume now if I do per unit volume what is going to happen this will become a into l and this can be written as I into gamma w so I into gamma w is the seepage force which is acting on the system clear the whole situation could be something like you know this is a dam made up of concrete or earth and for me there is no difference between this system this system and this system is this okay the moment I have created h over here rains might do it in the due course of time what has happened there is a h which got created over here look at the base what is going to happen I into gamma w per unit volume is acting here have you understood the consequence of this I will come to that do not worry first let us talk about the seepage there is a more big culprit know seepage is listen but what I should be worrying about is the seepage forces so this is what I am trying to explain right now we will take care of that part do not worry so what I am trying to do is this situation is a simplified model to do the analysis this is out of the way I discuss something and this is the real life situation which forces us to study what is the influence of hydraulic gradient on the stability of the structures is this part clear and what we have done systematically is we have shown that this is the hydraulic gradient which is the seepage force per unit volume now rest is all simple mechanics you can solve this I will skip few steps and try to solve this function if you can write it quickly it saves some of my time so this will be equal to L gamma submerged minus h gamma w into a try to show this this is the seepage force what is f upon a stress clear and suppose if I say that this is effective stress why h gamma w is what pore pressure so L gamma submerged minus pore pressure is nothing but the effective stress and if I put a condition that this tends to 0 this is nothing but your sand boiling now if you simplify this what you will be getting can you do it quickly I comes out to be what is I h upon L we define this as critical this will be equal to g minus 1 over 1 plus try to prove this is this correct jasmine is this equation correct you must remember is this okay meaning so you try to prove that I h upon L critical will be g minus 1 over 1 plus what it indicates is a soil is going to boil under hydraulic gradients and at that time the hydraulic gradient is this which is equal to the specific gravity minus 1 over 1 plus void ratio have you understood the whole khani or not so you are talking about the weight what you have to do is f will be f will be equal to what f plus this equal to this plus this a better way of defining this would be this minus this minus this is equal to f mathematically they are same is same as the h correct h is causing the entire story so what is h this h this is the incremental pressure now do an exercise suppose h tends to 0 what is the meaning of this this water column matches with this water column and try to see what happens the words are intuitive feeling if h tends to 0 what is going to happen hydraulic gradient is going to be 0 that is it so in other words this is the response of the porous media to the external gradients hydraulic gradients.


I thank you to the UK Parliament Open Lecture series for inviting me to talk to you, and also thank you to the European Parliament office, which is also represented here. I'm sure you all know, but this year marks 40 years of the European Union and Britain's membership of the EU. We first joined in 1973, so it's a significant anniversary, and one that I'm pleased to be able to give this talk on, Felly, mae erioed wedi atlent cwm wedi'i gofyn â'r UK.  Mae gennym��ionalau, a dw i'r Llekwm Be Awlodau, famu Llenchwyddo, mae'r unrhywion yw'r unrhywion efo'r eu ddechrau a'r unrhywion yn ymdannu'n eu ddechrau. Mae'n ddweud bod yn unrhywion o'i ddweud o'r unrhywion ddweud, a rydyn ni'n ddweud iddyn nhw'n ddweud o'n ddweud o'r unionu efo efo eu ddweud o'r unionu europei. Mae'r unrhywion yn cael ei ddweud i gael eich beth, ar y syniadwch gynnig hefyd oedd, a'r unrhywion yn cael eich beth, a 13 yr yw yw 2000. Mae'r ddigon i gyd wedi'u i gyd yn unrhyw iawn yn bwysig, ac mae'n ddod yn rhan o'r ffordd, ond nid oedd amgylchedd y cyflawn ar y cyfrannu a bydd y dyfodol yn cyflawn ar beth mae'n ddysgrifennu'r hun, ond oedd yw'r cyfrannu arall yn cyfrannu.  Felly mae'n таr ar dwi'n mynd i fi ffrifwyr. Mae'n ddod o'r bwysig yma, a wedi sach yng Nghymru felly yw eonsi yma yn yn fan mwy gwyllfa. Mae ar y cyfrifol 27 oes. Mae 23 rhan o yng Nghymru, dwi'n cyfnod Amol gallwn mynd i'r cyfrannu gwyd-igol, yn ganos weld yn biasio i gael gyrfa'n gydrygiadau o'i cyfrannu ei wneud. Mae ein bod ydydd gan y gweithio y Llefi ydych chi'n ddysgu, ac efallai yn y gweithio ystod y ddiw yn gweithio. Mae oedd yn ni wedi'u gweld bod yna'r ystod fel ystod yn ddiw yn gweithio ystod, a mae wedi gweithio'r ystod y ledes ystod oedd sefydliadol yn gallu gwirionedd. ac ond y cyfraseid wedi'i frydydd i'r ystyried yn ddod. Ond y yw'r yw'r yw yn ymwysig ar y cwrs. Mynd i'n mynd i'n ymdangos o'r cyfraseid o'r yw'r yw'r yw'r yw'r cyfraseid yn cyflawniaeth ar y cyfraseid a union europei. the European economy is not doing to well either. We have had a double dipp recession in which the Euro has avoided. So we are in the midst of a global economic crisis and I think we should remember that when we talk about the Euro. I think it is unfortunate in this country when we have media news press reports on Europe Mae'r meddlidau sy'n cyfryd o'r eu rai. Mae'n dweud o'n dweud. Mae'n dweud o'r meddlidau yn gofyn, ac mae'n dweud o'r meddlidau. Mae'n dweud o'r meddlidau yn gofyn yn unig yng Nghymru ac ynghylch yn parlymen. A'u ddweud o'r pethau tynnu ar y penerau is put the other side of the case. The positive case, no one would say at all that everything in the EU is perfect and we don't need to change anything. No institution anywhere can claim that and I would be one of the first people to point out that there are things which need changing and I think everybody in the pro-European camp would agree with that. But what we don't hear enough of is the advantages, the positive things about the EU and I'm hoping that you will go away with a slightly better understanding, at least as good understanding as I can give you of the way that we in the European Parliament see it and the good sides of the EU. I must say though, before I start that in any great detail, that the EU is not and has no pretensions to ever be a super state. It's something that we hear a lot in this country that oh the EU is taking over everything, it's going to somehow or other become this massive United States of Europe. That is not the intention that's never been what the EU was set up to do and we are absolutely not on that route. The EU has a very clear set of competencies which it carries out and it's delineated clearly and the principle of subsidiarity which I'm sure you all know is that member states devolve power down to the lowest possible level and decisions are taken at the lowest possible level is very strong indeed. So we are not an EU super state and we're not going anywhere near that. I will also say at the beginning that those who call for a referendum on EU membership at the moment certainly really don't I think understand the full implications of such a referendum. We have been as I said at the beginning in the EU for 40 years. That is a whole generation and I suspect a lot of you weren't even born in 1973 when Britain joined the EU. So it's a lot of history, it's a lot of shared values, it's a lot of shared legislation and it would take an enormous amount of unpicking. Not to mention the economic instability and the difficulties that may be caused by any attempt or any thoughts even of withdrawing from the EU. As far as I'm concerned it's an absolute non-starter and I think it's clear that that is now, that's my position and I wanted to make that clear to you before I started because that's very much where I'm coming from, the Labour MPs are coming from. But what I really want to talk to you about today is the EU itself in more detail both the history in the background and what is going on at the moment and a bit about what I do there and my own experiences of working in the parliament. So I want to cover why and how the EU was set up, what the EU stands for, certain milestones in the history of the EU, how the institutions work, what the EU has achieved and benefits of EU membership, some of the current issues that we're facing and what the future might be. The EU, as I'm sure you all know, was created at the end of the Second World War out of the ruins of Europe and the real collapse of Europe at the end of the Second World War when certain leaders of certain main countries came together and decided that they wanted to deal with this once and for all. And I think it was actually a very positive step not only to build and recreate from the ruins of Europe but also I think what the EU did at that time was represent a new humanitarianism after the awful happenings of the Second World War. You see emerging in Europe at the time and continuing much more of a respect for life, which is why the EU has, as some of its fundamental principles, equality and respect for individuals, gender equality, anti-racism and all of that came out of the horrors of the Second World War. It was also set up to combat nationalism, which obviously was a major factor in the causes and the duration of the Second World War and had been in the First World War when then and before the response to trade difficulties and disputes and territorial disputes was actually to go to war. And that had always been the history of Europe since the fall of the Roman Empire so we're talking for very, very, very long time. And the EU sought to overcome those negative attitudes and to build something much more positive in its place. Of course there was also communism and I would be wrong if I claimed the EU was completely idealistic because it wasn't. And part of the founding of the EU was to be at bulwark against the communist countries to the east and also of course importantly to provide a European voice along with the American voice in dealing with the Soviet threat and Soviet battleground. And there was also the other aspect, practical aspect coming out of the Second World War was martial aid and the martial plan. And it was felt that the countries who were benefiting from the martial plan would be well advised to get together to actually have a common view on that. So the EU was set up I think mainly for idealistic reasons although obviously practical elements as well. But certainly the founding fathers as they're still known and they sadly were all fathers had a very idealistic view of where they wanted to go. And I think we still hold on to that in many ways across the EU. And the EU itself when it was formed did learn from the lessons of the past. The interwar League of Nations had largely failed because it was just that. It was a collection of member states of national states who really didn't have an awful lot in common and couldn't quite work together for common goals. So those who founded the EU were determined that that wasn't going to happen again. And it also I think reflected and grew out of the internationalism and the desire to cooperate which was again a legacy of the horrors of the Second World War. There were quite a lot of bodies which were set up at a similar time. The United Nations itself of course, NATO, the International Monetary Fund, the World Bank, the GAT, the General Agreement on Trade and Tariffs, the Council of Europe as well. They all happened at a similar time during the 1950s when it was felt that war had just really just been too awful to contemplate again. An international cooperation was the way forward and the EU I think was the shining example of that if you like because it was and still is obviously much more than just an international organisation working together for common goals. There are those who believe that the EU should be further integrated and political integration is a worthwhile goal. And there are others who think that we should go wider and not have such a level of political integration but we should continue to work as we are now and very much on the same sort of level as we are now. That has always been quite a debate in the EU. My own view on that is that now we are 27 member states, Croatia is due to join quite soon and so we will be 28 in the not too distant future. I don't think closer political integration in the sense that it was envisaged 30, 40 years ago is actually something that can be achieved very easily when you've got such a large European Union. So I don't think that's going to go very much further. We will see the EU working on the basis that it is now, I believe, a common currency which is changing and evolving and that may lead to some further political union but I think it's very difficult to see that happening with such a large EU with the size it is now to any meaningful political extent. But it will be interesting to watch how that debate goes. And of course we are to a large extent in the UK on the sidelines of that because we aren't in the euro. We are not involved in a lot of those debates and discussions within the euro zone and we would in any event I think resist further political integration. Having set up in the way that it was it will come as no surprise to you to learn that the EU actually has a very strong and fundamental value base. And certain principles which the EU is founded on and which a lot of MEPs certainly talk about very eloquently and they are basically democracy, the rule of law, human rights, respect and protection of minorities and a functioning market economy. And we're also looking at the capacity to cut the competitive pressures and market forces within the EU adherence to the aims of the economic and monetary union and all members to enact the body of European law which is known as the ackey community as I'm sure you know. So the values are very important and something which we talk about quite often and talk about quite often also in the context of legislation which I think is slightly unusual for a British audience because it isn't viewed in the same way here and we don't discuss things in those kind of abstract concepts in the same sort of way. But I think it's worth emphasising this because those values are very important to the way the EU functions and very important to Member States and actually very important to those of us who are representatives in the European Parliament. The milestones. Well the EU was or as it was the European community grew out of the European coal and steel community which was set up in 1951 really quite early on and the founding members were the six founding members of the what came to be the EEC, Belgium, France, West Germany, Italy, Luxembourg and the Netherlands. They believed in integration, they thought that integration then might be something that they would wish to see projected into the future and they wanted that rather than just cooperation. So from the very beginning the EU was set up as an organisation which was not just going to be about international cooperation or European cooperation, it was going to be about more than that and it was going to involve even at that point some pooling of sovereignty. Now British people and the certain members of the House of Commons get very worked up about this idea of sovereignty. I think it's quite an emotionally charged word but basically what it means is that you will hand over some of your power on the basis that you will get something back in return. So it's not really a giving up of anything, it's actually an exchange and I think that's actually what the EU's always been about. It's a deal if you like. We will do that with you and in return we will get that out of the organisation. And it's something that started right at the very beginning. 1957 was the Treaty of Rome which set up the European Economic Community, the common market it was known. And also your atom because of course the other threat that was very much hanging over the world at this time was atomic weapons which I think did as well as the good things that shaped the post Second World War settlement. I think the fear of nuclear weapons was perhaps the negative side of it. In 1960 the European Free Trade Association was established which included Britain. 1968 was the European Customs Union which was the EEC plus one or two others including Turkey rather interestingly. Turkey's association with what is now the EU goes back that long and in fact the customs union was exactly what it said. It removed all internal customs duties and quotas and established a common external tariff. 1973 was the first enlargement of the European Union which Denmark and Ireland joined and of course the UK. That's 40 years ago when Britain became a member of the EEC. 1974 the European Council was created which is the member state government and is one of the three institutions which make up the European Union. I'm sure you all know there's the European Parliament, the Council of Ministers and the European Commission and those are the three institutions which from this time onwards were to form the EU and were to be the basis of the legislative process in the EU. 1979 was the first year that there were direct elections to the European Parliament. Before that time member state governments had sent representatives to the European Parliament but 1979 was the first direct elections. 1981 rather interestingly Greece during the EU so Greece has been a member of the EU for a very long time as a matter of interest. What I think we saw then after direct elections to the European Parliament, after the enlargement of the 1970s, I think what we saw was an increasing confidence in the EU and an idea that maybe it was becoming an accepted institution and that there were things that could be taken forward. I think what's also interesting during this period is that in this country the attitudes of the two political parties were exactly the opposite from what they are now. The Labour Party was very opposed to the EU and the common market and membership of the common market then whereas the Conservative Party was much more in favour of it. I think one of the interesting things about politics in this country in relation to the EUC and the EU is how that has really changed to be completely the other way around. Of course in 1986 one of the really big issues, one of the really defining moments of the EU took place and that was the setting up of the single market, the Single European Act interestingly done by Margaret Thatcher. It was intended to do exactly that to provide an internal market for goods and services across the whole of the EU without barriers and there were four things that it set up to do. Free movement of people, goods, services and money and it remains to this day the one thing that people who are opposed to EU don't even like membership of the EU want to maintain. I have yet to meet a politician on any side of any political fence except maybe some of the more extreme UKIP members who want Britain to leave the single market and an awful lot of subsequent EU legislation was actually linked to the single market and was intended to make the single market more effective. So I think it's important to hang on to this because after the mid 1980s the single market really was to a very large extent what the EU was about. It had become a trading area and it's a very important one these days for us. Almost half of our exports from the UK go to the European Union single market. Were we not to have access to that it would be serious, incredibly serious. And also all the social legislation or most of the EU social legislation which features a lot in debates in this country at the moment is designed to provide a level playing field across Europe so that no individual member states has an advantage or a disadvantage in the single market because they have perhaps social legislation which isn't quite as good as other member states. So all the social legislation was set up for that very reason. So you have health and safety regulations across the EU which employers and governments have to undertake because if you had stringent measures in one country and measures which were pretty lax in another that would give the country that had the lax regulations far more of an advantage in the single market. And I think this is worth actually holding on to because that's the point of the social legislation. It's to actually provide not only good working conditions across the EU but also to provide a level playing field in the single market. It's an absolutely integral part of it. The other thing that happened with the establishment of the single market was that qualified majority voting was extended. And that's where member states have waited votes in the commission and the council so that no one country can veto proposals. And that was an important part too of the agreement over the single market. Also in the mid 1980s Spain and Portugal joined. Of course after that was the cataclysmic events of the fall of the Berlin Wall and the end of communism which changed the face of Europe and were soon to change the face of the EU. And in 1991 not necessarily linked to it but 1991 was another important year in the history of the EU because that was the Maastricht Treaty which began to pave the way for the setting up of the euro. It was the agreement to move forward to a single currency and as you all know the UK had a knocked out from the Maastricht Treaty and we have not as yet joined the euro. Also in the late mid 1990s Austria Finland, Sweden joined and would also set up the Schengen agreement which is quite significant. And because the UK isn't in it we don't hear very much about it but the Schengen agreement actually did away with border controls. So it's now possible to go across Europe without a passport and it just has helped internal movement quite significantly. So the late 1980s and early 1990s were a very significant time indeed and of course in 2002 was the introduction of the euro. And then in 2004 the accession of the former communist countries in the east of Europe as well as Malta and of course Cyprus. So that's when we ended up being an European Union of 27 member states, 25 then in Bulgaria and Romania joined slightly later. And it's this enlargement and this actually absorbing the countries of Eastern Europe was the main reason why the EU was awarded the Nobel Peace Prize which sadly was ridiculed to a large extent in this country. I think very unfairly because the EU has amongst other things kept the peace. I mean there is an argument that Europe may have stayed at peace anyway but certainly the setting up of the EU and the absorbing of the former communist countries I think has actually contributed to a peaceful Europe. So that anyone born after the Second World War in Europe certainly in Western Europe has not had to face the threat of destruction which our parents or grandparents and great-grandparents suffered. And I think that is a big achievement and should be recognised. After the euro we saw the Lisbon Treaty which was aimed at modernising the EU after the enlargement and set up various things which have shaped the way that the EU operates now. We've got a president of the European Council now, it's Herman Van Rumpuy. We have the high representative for foreign affairs and security policy who is the British Commissioner, Baroness Catherine Ashton. The European Parliament itself actually has gained more power over legislation. We do have legislated co-decision as we call it with the Council of Ministers in most areas now, not all of them but most of them. And what that means is that when legislation comes through the Parliament, if it is legislation which is part of the co-decision process, we will go through it in committee, we will amend it, it will go through the plenary session of the European Parliament and then it will go to the European Council. If there's disagreement between what the Parliament wants and what the European Council wants, there will then be a process called conciliation where we aim to sort out those differences. In fact we have to sort them out because if the points at issue haven't been resolved within six weeks the entire piece of legislation falls so there's quite a pressure to get it right and end it. One of the other things that the Elizabethan Treaty did was to involve national parliaments more in the way that the European Parliament works and I think that's actually, I will have better lines of communication which I actually think is quite important because when you have layers of government and layers of legislative making bodies, I think you do need to know what's going on in each particular place and I'm not sure that that's always worked particularly well between the European Parliament and national parliaments. We've also reduced the number of MEPs obviously and we've taken on some new policy areas under the Elizabethan Treaty such as sport and tourism. So that's a brief run through of the key events in the history of the EU which I think has been quite a busy 40 years actually and the EU has grown from quite a small fairly experimental body and it's now very much part of our governance and something which I think it would be very difficult not to have now that we've been in it for such a very long time. In terms of the institutions themselves, the European Parliament works very differently from the Parliament where we are now. The whole EU concept is about consensus, it's actually about people talking to each other trying to resolve any points of issue and that runs through the whole way that the organisation works. It's not only because consensus is a good thing which it is. It's also because the EU itself has evolved so that no one country and no one part of the three parts of the EU, the three institutions has too much power. It's very much a balance and we're always trying to balance the interests of larger member states against smaller member states making sure that things are agreed and that the consensus actually works in practice. Very different I think from the British tradition and I think a lot of us found it an interesting experience when we first started in the European Parliament. We've also tried to make the Parliament quite a family friendly place. We work sensible office hours and when we start at 9 in the morning and finish at 6 at night and we have rather nice to our continental lunch break which is quite good except the Brits always work through the lunch break. So it has very much more of a working feel about it I think than some national parliaments and I think that's very good and it's really quite a good working environment. The other thing I want to say to this audience is that particularly since the 2004 enlargement English has become very much the working language. All the Eastern European countries had English as their second language so English is the social language and quite often and it's happening more and more meetings are conducted solely in English because so many people speak English as their second language.  This is an international institution with 23 official languages and the language that they choose to use as their common one is the language that we speak. I always find there's some sort of irony here about the British attitude to the EU which is somebody detached often at best to the fact that the EU has absorbed our language so wholeheartedly and it's just one of those rather strange situations and I think we would do well to ponder on that. Basically the way that the European political system works was set up on the French model so it's also like the French and very much like the American model as well so the European Parliament is a legislature that's what we do. The commission and the council are much more like an administration and the way it works is that the European Commission which is comprised at the top level of a commissioner, one commissioner per member state, the commission will propose legislation. Nobody else at the moment has the power to do so though I think the commission do take on board what's going on and what would be appropriate legislation and what's important and what's topical. It will then come to the Parliament and to the council of ministers to legislate to amend that and then to vote on it. It will then if it needs to be it will be interpreted by the European Court of Justice and then enforced by the commission so that's essentially the process and it's quite useful to know that and to remember how it works because I think what people find often a little bit unusual and unlike other experiences is the role of the commission and there's quite often criticism of the commission is not democratic. It's indirectly democratic in that commissioners generally have been senior politicians in their own countries and they are appointed by the member states. And there's also the question when it comes to democracy of the idea particularly prevalent I think in this country that the EU is not democratic, that there is a democratic deficit. It's true that turnouts in elections to the European Parliament aren't great but then turnouts to elections in this country for local government aren't very good either and they're about the same. The European Parliament and local government turnouts are similar almost always. And of course the European Council is made up of member state governments so that is elected. Of course everything does a five year term so you may not actually find the three institutions in synchronisation with each other. But I think there's a lot said about the democratic deficit as people choose to refer to it which is actually not true in practice. Of course there are things that could be done to make it more democratic and it is still based on member states in that sense it's not a European wide electoral body. It's MEP's return from individual member states and member state governments and the commissioners appointed by member states. But I think when you talk about things like democratic deficit you have to be a bit clear about the definition of that and although I don't think the system of election in the EU is completely perfect as nothing is I think it's not as bad as many of its detractors would actually make out. And of course the principle of subsidiarity which I mentioned earlier is important in this context because nobody in the EU seeks to legislate at European level unless they have to. So legislation and rule making are done at the lowest possible level and we're always very clear that that is the case. And most important issues are still done by member states taxation, social security, health education to name but some of the big ones are still done by member states they're not done by the European Union. So we're not talking about a super state or any sort of super national infrastructure really. We're talking about a few things and we're talking about the internal market and legislation which makes the internal market function better. We're talking a little bit about common security and defence policy and we're talking obviously these days about the euro. What we aren't talking about in the EU and what we don't do is any of those really basic things which are still carried out at member state level. The EU has achieved considerable amount of the last 40 years and one of the things I think I'd like to talk a little bit about is the whole concept of soft power. It's often used when talking about the EU and it's quite important. The EU does have representation across the world. The EU has been part of peacekeeping forces and we do have and not very evolved but there is a common security and defence policy. And there's also power and economic terms where because the EU is such a large trading block it obviously has power and influence in that way. So it's not power in terms of going out and showing that you are bigger and better than everybody else. It's much more influence and I think that's a good role for the EU because the EU after all is its member states and the EU as a block doesn't want none of us and I don't think anyone in the EU wants to supersede member states. So soft power is a very helpful way of achieving certain useful aims and one of the things I would like to cite in this is Turkey which still hasn't yet joined the EU and may not wish to quite as much as it did but there was a time when Turkey very much wanted to become a member of the EU and was very keen to comply with the ACI community and the criteria for joining and one of the things that the ACI states is that no EU member state should have the death penalty. Turkey did of course and they stopped it, they abolished the death penalty because they wanted to join the EU so that's a measure I think of the sort of soft power that we're talking about. There is also a lot of trade agreements which are very I think very helpful in the modern context and as we speak there is a potential trade agreement being negotiated with the United States and it won't surprise any of you to know that the EU and the United States together account for 50% of the world's GDP and nearly a third of the world's trade flows so a trade agreement between the EU and the United States would be a good idea. And so Barack Obama, President Obama has the same idea and he said that he is encouraged by this and that he's encouraged by the benefits of an ambitious and comprehensive market opening arrangement for agriculture and manufactured goods services and investment, the identification of ways to promote compatible regulatory approaches and tackle behind the border barriers and possible approaches to intellectual property rights. So that explains I think that quote from President Obama very well the benefits of trade agreements with other countries and particularly with the United States and the advantages that would bring to EU, the members of the EU, the member states in the EU and the United States themselves. Other things that the EU has done and benefits it has brought is a high level of environmental protection. The environment of course doesn't stop at national borders and is an obvious area to be carried out by the EU. The most environmental protection is based on the polluter pays principle and we've seen air and water quality legislation, waste management, biodiversity protection and the idea of environmental liability. Blue flag beaches were an EU initiative and things like air quality for those of us who live in London or visit London is of course incredibly important and the other environmental matter which is becoming huge is climate change where the EU has had a very pivotal role in tackling climate change and a system for trading emissions which will aim to reduce harmful emissions, harmful carbon emissions particularly by 20% by 2020. We've also done a lot on consumer protection. One of the things that is always cited which I'm sure you know is bringing down roaming charges for mobile phones. I was actually on the committee when that went through and it was a really helpful thing to do and it's just one example of very many consumer protection measures that have gone through. We also, even in the midst of the scandal over horse meat, still have a role in food safety. We've introduced a lot of legislation on food labelling and what goes in food and health measures and those kinds of things. So that's just a very few examples of the kinds of day to day work that the EU does. Of course we are facing at the moment a raft of current issues, not only the international debt, the sovereign debt crisis in the euro which is not resolved yet by a long way even though there are moves to establish banking union. That's still clearly got some way to go. The common agricultural policy at the moment which has always been a thorn in the side of the United Kingdom is at the moment going through a process of reform, not before time I might say. The CAP still takes up 20% of EU spending which is an extraordinary amount when you think about it and it also isn't part of the legislative co-decision process. So there's still quite a lot of major issues around agricultural funding. We're trying to get away from the direct support to producers which is the basis of it at the moment and to look more towards development, rural redevelopment and more environmentally friendly measures which are called greening of agriculture. But it's still got a long way to go and there's still quite a lot of vested interests who don't really want the system to change very much. But if you're interested in it, this is all happening at the moment so it might be, if you want to, it might be something you might be interested in following. We've also seen, to look at the wider political sphere, the rise of far-right parties across Europe which I think is quite a worrying development. There have been far-right countries in several European countries and I think this is something which we all need to watch out for because once those sort of things take hold it can snowball and get worse. Also, the other thing that I particularly wanted to talk about was the tension that comes and goes between the eurozone members and those who aren't in the euro. All the new accession countries are due to join the euro at some point though not all of them had. And this concept of a two-speed euro which is talked about sometimes and obviously if that happened the UK would be on the outside. Now I don't think personally that would be a very good place for the UK to be because we would miss out on being there at the table. And this is important. If you aren't fully part of an institution you clearly can't have a say in what happens to it and what happens in the EU does affect this country severely. And countries like Norway and Switzerland are often cited as being oh yes we can go leave the EU or only be semi-part of the EU and do what they do. Well Norway in fact and Switzerland both of those countries Norway particularly does implement EU regulations. They are bound or they choose and they are bound by EU regulations that they have absolutely no say in what those are. And the same would be true for any country that were to leave because the world is becoming a smaller and smaller place and we are tied economically or tied in terms of trade. And there's all sorts of ways that Europe is part of a whole and it's impossible I would say now for anyone European country not to be affected by that.  I don't think that's a very good place for us to be and I therefore think that we need to stay where we are make sure that we have that voice and make sure that we are heard and that we can therefore have some impact on the policies of the EU. The EU is here to stay. There is no going back and I think we should celebrate the advances that the EU has made. There are advances that I've talked about and there are other things as well. I mean my contention having been in the EU for a long time now having been a member of the European Parliament for a long time now is that the EU is fundamentally a progressive institution. Interestingly at the moment though I say that it's completely controlled by the centre right political parties at the moment. The majority in the European Parliament is centre right and in the commission and of course in the council of ministers. But even so the European social model is still very much in existence and the values that the EU has founded on are still very much there. There are of course huge problems and no one is denying the depth of the crisis of the euro but we mustn't forget that it's also a global economic crisis. I firmly believe that the UK has to stay in Europe and I think I will leave it with a question of what would happen were we to leave. Although President Obama quite clearly welcomes a trade agreement with the EU it's becoming even more obvious that that's as far as it goes. The United States is now much more looking towards the Pacific and I think that old relationship where the United States saw Britain as a bridge to the EU and it was all part of the special relationship and the wartime alliances. That's to all its tents and purposes disappeared and in a world where we see the rise of new countries such as India and China it's important and it's more important than ever that the UK stays within the EU within its large trading block our place is very firmly there and if we weren't there where would we go?


Hello, thanks for having me. I thought my talk is in an hour. That's why I kept you waiting. I'm really sorry about it Yeah, my my my watch is still in the in Berlin apparently So, um, yeah, so let's get to it. I'm gonna talk about the test directory scaling So who am I? I'm a freebies decometer since 2018. You may know me as zero MP I am now serving on the freebies decor team and Most of most days I work on cool stuff with false of Clara But in general I poke around things like documentation ports RC scripts tracing and ZFS and the outline of this presentation is as follows first I'm gonna tell you about ZFS and dive a bit into the implementation details of how it works Then I'll pivot to directories then we talk briefly both scaling and I'll give you some numbers about well, how the tuning of directories in the test Can work So to give you some Background of this whole work. So this is the the Benchmarking with it for a client Clara and The orange line is what the defaults the first does. So when you have many When you write many files into directory in the single directory in ZFS The performance drops, right as you can see but if you apply some tuning But I'm gonna present to you you can be where the green line is of course it depends on your setup discs the rest of your tuning but Essentially, there is some performance gains that you may try to get out of your system. So let's talk about ZFS So what is it? It's a copy and write file system It was developed at Sun in 2001 it was imported into free BSD shortly after in 2008 and as of 2023 free BSD uses uses something called open ZFS Which is a variant of fork. There are a couple of different implementations in the wild of ZFS and Open ZFS Works on both linux and free BSD This allows for more more cross-pollination of ideas more testing. This is a pretty cool initiative Um, so what does ZFS do? So there are a couple of things the most important ones are maybe data integrity so every block you have on your system is checksum, which is nice and When data gets corrupted ZFS can detect it and if it's not too much of a corruption you can also try to Correct it or at least report to you that hey something is wrong. You should really take a look at those discs Um, it also offers data consistency So even if your system crashes you won't end up with a disk that is in an inconsistent state Whenever you write to a disk in ZFS Your data is kind of saved on the side and waits for the checkpoint to come when the chain checkpoint arrives All your rights they are made official and now they are the new You know version of of the state like of what's happening on the disk So you never have an inconsistency there and also there is pooled storage. So There's no need to partition discs discs in advance. You just give all your Storage you have to ZFS and then you can create partitions grow them shrink them we don't have to Think in advance about how you want to split the storage you have in between different partitions Um Apart from that it does many other things like snapshots efficient the remote application compression encryption did application all kinds of cool stuff which are well It can do all those cool things because of the way how it's structured. So we could get many features out of the Well the implementation Which I'm gonna tell you about so I Would like to give you like a short Introduction to what happens when you write data to ZFS Right. So what we have here? We have uber blocks at the very top this is like the the root note of this whole tree that ZFS consists of and At any given point you have one uber block that is like an active one And it points down to like this whole tree that is all the data you store in ZFS So in this case, we have the uber block it points like many many days to take it first like Let's say for simplicity like to a pool and then to a data set and then some directory maybe and then we end up at this profile of the profile file Which then has some indirect blocks and then we have the actual content The data blocks which is in this case like one of them Let's say contains editor equals vi then the other block contains contains page or equals less, right? This is like the actual data starter So what happens when you write when you want to replace your editor vi with editor at because vi is too complex And you want something simple What happens is that you issue a right and ZFS allocates it like it copies the block and modifies it the way you want and it saves it on the side and then The whole tree all the parents are updated because now we have a new block so the block above it needs to Well have a new checksum because the content below changed So so you go up and then oh we have to go up again And then you have to copy the profile because you know The checksums and things and as you go up And you prepare all those new blocks What happens later is that when there is a checkpoint You start pointing to a new block that you just created in this writing period and the new uber block gets Well activated so from now on you have a new tree with all the updated data That's how we avoid inconsistencies right because first we write everything we need in this new Checkpoint right and then once we're ready. We just switch the root node, which is the uber block and we're good So what is the file in ZFS? It is an object and an object is just a group of blocks There's organized at the top by a denote which is something like an I note in in UFS yes and Almost everything is an object in ZFS files directory's data sets in this case when you when you look at the file object You have well the file object the red frame the yellow frame is a denote So it's this is like the the root of this sub tree And one of the things it points at is the block tree where the actual data is stored sometimes you cannot Sometimes it's like not like apart from the direct blocks where the actual data is stored. You also need indirect blocks which help you address more than Just a bit of data, right? You just need to build this tree a bit Has to be a bigger tree to store more data So those are the parts of a file object that you would like to Talk about today. So let's take a closer look at the files So what you can do in ZFS you can inspect Those objects. So what we do here is we create a half a gigabyte file On ZFS and then we inspect it with ZDB, which is like a debugger for ZFS Yeah, so let's take a look at what's happening in this output. So there are a couple of interesting things for example the the blue Highlight this thing Fletcher for this is The checksum algorithm and you can change it to different ones different algorithms offer Different trade-offs, right? Some of them are faster. Some of them are like slower But something you have a choice depending on your workload This may improve performance or like change the characteristics of your system the yellow highlight for example is the compression so compression is Compress all the data by default for you Which is nice because then when you read the data from the disk you have to transfer less bytes So it makes things faster and decompressing those things is usually quite fast But then of course depending on your workload you can either pick compression algorithms algorithms that are Faster but they compress Were in a worse way or you can pick the one that is like very CPU intensive But then it also compresses very well, right if you want to save some space on the disk Yeah, and then we move a bit lower and in the green highlight shows us the the block tree and So it says indirect blocks and then L2. This is like level 2 In direct block and then we have some level 1 indirect blocks L1 and then we have L0 Those are the direct blocks. This is where the actual contents of our file is Yeah, and in each of those entries you can you can see the Address and maybe the size the checksum It's it's pretty cool to the zdb tool. It's nice tool to know about and To know how it operates if you want to explore the test a bit more for example We have in in BSD's we have this wonderful file called flowers if you don't know about it You really should it's very useful So what we're gonna do we? Just for the experimentation we set the record size to 512 bytes so not too much And then we copy the flowers file into data set so now when we expect inspected with the zdb we Take a look at the output and we can see that we allocated exactly three Data blocks, which is what we expected because if you look at the output of LS at the top this file is One and a half kilobyte, so three blocks of five hundred twelve bytes should be enough to store it So that's cool. So let's what we can do now. We can use zdb to extract The contents of this file to take a look so we use org to like parse out the exact address and Now use it again with minus r we tell it hey on this pool called tank there is this address so give the contents of this block to me and as you can see We get the same outputs from zdb as we get from head Which is nice, right what we did here the addresses and copying and so on it works you can Feel that you have a bit more control over what ZFS does Yeah, these and much more we can do with zdb and like explore a bit what's going on there directory So from our wonderful manual pages We can learn that directories provide a convenient hierarchical method of grouping files while obscuring the underlying details of the storage medium and Directory consists of records Directory entries each of which contains information about the file and a pointer to the file itself So a directory is basically a map of file names to some identifier So let's take a look how it's how it's done We can do a couple of different operations to the directory Today we're gonna focus on path name searching. So lookups and name creation. I won't be talking about the performance of All the other operations. It's out of the scope for today What is the directories at FS it is an object just like a file the difference is that it has a different tree It's not that it's not a tree of Data blocks. It's a tree of zap blocks But then what is a zap, right? So the zap is a ZFS attribute processor So it's a key value store. It's a map and the keys can be strings and the values can be strings numbers areas of numbers Different things. It's very flexible. It's used primarily primarily for directories But it's not the only place where it's used in the code base It's implemented as an extensible hash table and it's case nicely up to hundreds of millions of entries in the Single directories. It's a good data structure. It's slightly complicated, but it's a good one It's also called a fat zap Because there is also a micro zap which is a Much smaller data structure designed to remove some of the overhead of this complicated data structure for smaller directories so a Microsoft is just a single block and it stores the mapping of filings to well Objects, so you cannot really store like complicated things. There is just You know strings to some IDs. It can only hold a maximum of 2047 entries by default and Keys are limited to 50 bytes. So if your file names are longer You won't be able to use a micro zap And values are limited to integers, but that's obvious and A micro zap gets automatically promoted to fads up. So whenever you Create the 2048 file in your directory ZFS will Change the micro zap into fads up for you and start using this whole Complicated data structure, right? Because like it's it's getting ready to handle a lot of files The same thing is gonna happen if you have a long file name One of the problems is that if you make a mistake in your home directory, for example And by some accident you have a script that creates like temporary files and you end up creating a million like temporary files in your home directory The test will gladly do it and it will work then you can remove all those fires But but the fads up is gonna stay there. So from now on whenever you run LS So the best we will have to traverse all this like extensive fads up that you just created and in order to get rid of that You would you will have to Well recreate the directory basically Which is not exactly something that you want to do with your home directory It's it's a known Limitation let's say and it's been People work on it if you want to check out the process the progress on the on the patch that is in flight now You can look for open ZFS issue number 14 088 so scaling This is gonna be like a very High-level introduction to what you can do to ZFS. So there are a couple of different ways how you can Change how ZFS behaves the first one is you set properties on data sets The other one ZFS create you can also set those properties during the creation of a data set Some of the properties can also be changed with CCTL and then depending on what property it is it influences The whole ZFS module that runs on your machine or it or like specific pools or data sets For some of the Tunables you have to set them before loading the module and then you do it with a loader for example So there are many different places sometimes you can get a bit confused where you can do it But in general the documentation is quite good some of the popular tunable that people Set or like start with when they're trying to Tune ZFS is our a time which is access time This is owned by default. So whenever you access a file It also gets Well, the a time gets updated so as the best has to write in a new block where the a time is, you know a new one This can impose a lot of overhead and if your application doesn't Depend on a time. It's a good idea to turn it off the record size so this one is a very How should I say Popular problem in when people deploy ZFS because their application is Writing in like different chunks in different sizes than then ZFS is storing data and this mismatch creates performance problems Well, usually because the test has to do more work than it should be doing So it's good to align it with your workload Another one that is primary cache you can change what you what you really want to cash in ZFS you can Either store like data and metadata inside or just metadata This makes sense. For example when your application does its own caching So that you don't cache data twice and then things like compression, right if you want to Have this trade-off of Spending more CPU time, but then having data more compressed to gain more storage. You can also do it with the compression Tunable and I will switch to something more specific to directories. So this is the first of two tunables I'm gonna talk about it's a it's a way to control the maximum size of a micro zap and It's this is CTL is called VFS ZFS zap micro max size The idea is that at the moment the default is for a micro zap is to be one of the size of 128 kilobytes and That's where those 2047 entries fit if you want to store more so we want so we want to Delay the moment when micro zap turns into a fads up you can well increase the increase this Maximum size of a micro zap for example to one megabyte and then you can store up to 60 like slightly over 16,000 files in this single block it was made and tunable fairly recently earlier this year and To give you some ideas. What are the advantages and disadvantages of a larger micro zap? so When it's larger directory object has to has to It has less direct blocks, right? So when you're reading you don't have to read this whole tree of fads apps you just Fetch this one block and you're good because now, you know the whole structure of a directory On the other hand When you are writing to a directory, you no longer update You know small blocks you update this one huge block. So if you write to your directory often This can lead to an overhead. They're like always trade-offs the other ZFS directory tuning thing is a size of indirect blocks It can be controlled with VFS ZFS default IBS the default is 17 which is 2 to the power of 17 which is 128 kilobytes It's available on free BSD since a long time, but then It was also made a tunable solution three on Linux and Advantages of having smaller indirect blocks is that You have to Process less bytes when you're reading or writing, right? But on the other hand you may end up with a larger tree that you have to traverse So again trade-offs So let's see how those tunables behave when you actually use them and run some benchmarks talk about some numbers I Prepared two benchmarks for you today one is a look up benchmark with me when we just Recursively traverse through the directories and we just Process every entry that is there and another one is create so we create a lot of empty files Just to update the directory structures and see how it's How fast it gets I used hyperfine as my benchmarking harness. It's a pretty cool Piece of software if you don't know it, it's it's it's very nice for ad hoc experimentation And yeah, and the tunables you already know those are the ones I introduced in the previous slides So benchmark one So the measuring what we're measuring we're measuring the time to list files of all sub directories I'm just using like a very short C program that does FTW and We're gonna focus on three parameters here The files per sub directory the maximum micro zap size and then indirect block size This is what happens when you Have 16,000 files in directories So this number exceeds the default micro zap size, but it's Just before the it fits nicely into this one megabyte Micro zap before the fads up kicks in So as you can see here larger micro zaps increased performance Because we could stick to the micro zap for longer so Retrieving like reading out the record structure was only one one block and not many That's probably why we see the difference The indirect block size doesn't really matter here or at least not in this particular deployment and then We test 64,000 files, so we definitely exceed the limitations of both the default micro zap and And the tuned micro zap and as you can see there are no Differences in performance pretty much. It's it's it's just noise So once the fads apps kick in You don't really see the difference so One of the takeaways from this one is that if you want to tune it to your workload you should Really think how many files you're gonna store in your Directories there is no one silver bullet to just make everything faster Every time it has to be really adjusted to your own workload and your system Right and then Let's take a look at one more example of The lookup benchmark. So here we have 16,500 files So we exceed both micro zaps the bigger one was like just exceeded like 200 files ago But we turn primary cache off so we are not caching anything Which results in? the indirect block size being very relevant in this in this in this situation the micro sub size does not really matter anymore from what we see but the Performance of recovering smaller block blocks from the disk. It really makes a difference so as you see depending on the workload again or How your system is designed and how much memory it uses? Different tunables can do different things. They can have a different effect on your system benchmark, too So we're gonna look at the writing case for directories. So what I'm doing here I'm I time I measure the time to create files in directory. So what I do I just open a file So now I have an old an empty file and I sync it so that it gets sent to the disk And again the parameters are the same. So let's jump to the results Over here I'm running the benchmark with 247 files, so it fits both the default micro zap and the large one and there is no observable difference yet That's probably because the system is too fast or something So that we can't really notice Let's go further when we have 16 838 files, so we exceed the default one, but it fits into the The tuned one just barely We already see some differences. So larger micro zaps are more expensive in writing Because we overwrite them way more and they are heavier. They think we need to write more bytes, right? So it's slower So this can be pretty disastrous for your performance because now it's twice slower than it used to be so Yeah Measure your system before you tune However, when we increase the number of files to 64 thousand the the performance gap Starts to shrink. So there is less and less like the difference is shrinking basically right so That's a good thing so again Measure your systems So this summary There are two important tunables for directory scaling the maximum micro zap size and the indirect block size and The tuning takeaways are reads are faster with larger micro zaps and smaller indirect blocks and writes may slow down when using larger micro zaps Are they set many times tuning depends on the system? So you should really measure the system before tuning it, but Basically now, you know, which two levels are relevant to this part of that FS So when you decide that hey, this is like relevant to my systems to tune how fast my directory is working you can use those and See how it plays out. Do you have any questions? So should I repeat the question? Yeah, so the question is if I compared it to UFS No, I didn't But it's a very interesting But I thought about it the reason why I didn't do it was that it's It's very tricky to benchmark two different file system against each other especially since that FS has so many layers of indirections and caching and so on so I didn't do it because I wasn't sure if I can deliver, you know, reasonable results If you can talk about so the question is if I measured the globbing in a shell so that I'm not Looking looking up all the files, but like a subset, right? I Didn't do it But then when you're reading out of the rectory and you would when you're using globbing you I guess you You still have to read the whole Data structure just to see if you're matching anything So I think that would be the same The question is if the tunables are data set specific or they influence the whole system They influence the whole system from what I see Yeah, if it was otherwise, we would see at least, you know some a part of the city I would be a name of a data set or a pool. Ah, yes. Yes, of course. I wanted for the system to stay like I Wanted the directories to really be micro zaps and I check that with CDB after every So the question is if we can change this this limitation of micro zaps to not be 50 bytes, but maybe less or more there is no tunable that I know of and I Guess that what you can do you can just patch it out and see if you can fit more directories in there like entries there But that's like a manual operation. I would assume So you still have to read out the whole thing The interesting bit is that when you when you switch from a micro zap to a fat zap you just add one more block I mean the structure of everything changes But effectively when you look at the output of CDB you have one more block to deal with that answers your question. It's So is the question does it help to have sub directories or? Oh, yeah, nesting is very helpful if you can change your workload to Nest directories instead of fitting all the files into one directory This is one of the ways to make it. I mean then of course you have to do more lookups, right? Because you have to traverse the whole hierarchy But it's From what I saw this is usually the better idea Mm-hmm. Yes So the question is if ZFS has a special cache for names Ah for for the name like the name cash I Don't remember. I'm not sure if I ever knew So the question is if I have If I measured the the number of IO operations That were issued during the benchmarks. I Didn't measure it. I just wanted to have I mean I did it when I was doing the work for the client I didn't do it for for this like presentation of The general I in general intuitions of how to work with those tunables But if you look at the very first At this graph, this is right throughput So I'm not sure how to answer your question. I didn't do it. I don't have any numbers to like contact I can share with you on that It could be done


Kids who have books in their homes are more likely to do better in reading. We really wanted to make sure that we are doing everything to help these kids become stronger readers academically. So we decided that we would get into our kids' neighborhoods and give away free ice cream and also free books. Kids often say, I hate reading so when we get out there and put a book in there and they're jumping up and down, that's the goal. Sarah doesn't just stop at thinking something, she keeps going until she brings all of those visions into reality. If you look at her hashtags, she lists all of her passions, one of which is her books and braids program. I was trying to find a way that I could squeeze in more reading time with students. This is also a population of kids who often doesn't get the chance to go to a beauty parlor, so she just thought it would be a great combination. And it was really fun doing it with Ms. Middleman. I've always read Elfin and Piggy books, which are my favorite books. You should have taken him at... Books and Braids is really more than just reading. It's more about that one-on-one connection with a caring adult.  Do you think that girls should get to go to school everywhere? Yes! Do you think that... Girl Power is one of the greatest ideas I feel like Sarah's ever come up with. Originally, it was just supposed to be a small group of girls that talked about how girls could be anything. It was part of a literacy grant. Through the years, this Girl Power program has evolved into a program that really focuses on gender equality. Last year, I think we sent seven girls to school in Haiti. We bring in speakers. We have gone to the state capitol in Bismarck for Women's Day at the legislator. My hope is that they will consider leadership roles in their future. She just keeps pressing the envelope into providing just the best experiences for our students. She builds confidence. She builds an environment that those kids feel safe and loved in. These children have faced obstacles with education. My goal when they come here is to give them authentic, relevant learning experiences, but also to make learning joyful for them. Even if it's something seemingly small, it made a huge impact for our kids. She's changed the lives of thousands. He doesn't get any better than that.