Okay, so let's begin our interview. So first of all, could you please state your full name? Louis Jean Cabri. And what is your age? I'm 82. And where were you born? I was born in Cairo, Egypt. Okay. So as a child, what did your parents do? My father was an entrepreneur and businessman. My wife just, my mother rather, just a housewife, yeah. Okay. And as a child, what were your interests or your pastimes? Well, it was, what I remember mostly is because it was during the war. And I went to a British school, a British preparatory school called Gazira Prep, outside Cairo in the suburb called Zamalek. And there it goes to the British school, sports was very important. And as far as what I did, well, we had limited things. You couldn't buy, for example, I love toy soldiers, but you couldn't buy them because it was a war. So I did other things. We went to a botanical garden nearby and we befriended Commonwealth troops and tried to get badges and things from them. And I used to go out after air raids in the streets and collect shrapnel.  Did you ever feel the direct impacts of the war? Well, I have strong memories, strong memories of the air raids and going down to the shelter, the troops. I can even remember it's amazing, but it stays with me during the time of the Battle of El Alamein. It was about 70 miles away and we could hear the guns. And we saw the troops marching to the front. And I'm not sure because we were staying in an apartment, living in an apartment on the main street. And the troops would be walking columns and columns of them, eight or twelve abreast, I can't remember. And we sort of all went on the sidewalk, cheering them on and things like that. And we had many friends among the officers, all Commonwealth troops except Canadians because I don't think there were any Canadians actually in that front, in the Middle East front. So we had officers that used to come for dinner from New Zealand, Australia, South Africa, Brits of course, and so on. Now, in terms of your education at school, what were your strengths? Well, at the preparatory school it was British education with accent on the classics and Latin and so on and so forth. And I left there in 1946 and soon after the war because my father didn't see the future of staying on in Egypt as a businessman. So we emigrated to the United States and I went to high school actually in Skarsdale, New York. So after a half a year of the before high school, I went to Skarsdale High School in New York and after three years there, my father decided to move again and this time we moved to South Africa where I just had the one year of my trick and so I finished school there and went to university. My first university was in South Africa, in Johannesburg. And throughout your schooling, even though you moved a lot, was there a specific interest or strength that kept you throughout? Well, you know what? My hardest, the most difficult subject, the most untransportable subject, arithmetic and stuff like that was easy. The most difficult one was history and that should be of interest to you and that was because I had history with a cultural background in British in Cairo. Then in the US I talked about the explorers and whatnot which I'd never heard of and then one final year of history in South Africa where they talked about the Caffer Wars which I'd never heard of. So that was my hardest subject. But interesting, but hard. Well now you're well versed. Because with history you need a cultural background. So it's not just one thing or just memorizing dates and events and wars but if you're not living in the country it doesn't mean the same to you. Absolutely, for sure. So what did you decide to go to? In what subject you said to... Well I went to the University of Wartes-Ren in Johannesburg and I was interested in chemistry and I don't remember how into geology so I majored in geology and chemistry and over there the system is you take a bachelor's in three years and then you specialize in an honors year which I took geology because I sort of liked it and after graduating there I worked as a junior field geologist mainly in West Africa I worked in the Gold Coast before it became Ghana and I worked in Sierra Leone and in South Africa and so after four years of working as a junior geologist I realized that I needed some graduate degrees to get on. I didn't want to stay because the situation in South Africa especially was there was these big mining houses they were much bigger than they are now because now other companies got bigger worldwide but they were very big at the time Anglo-American and so on and you worked for them as a junior I didn't see the chances of moving up so I thought I needed some more education and I came to McGill to Montreal and McGill because in what I knew about in South Africa was that that was the place to go to the reason being that in the 30s many McGill graduates went to South Africa and they ended up by being the chief geologist of the big mining houses so since they had succeeded in that way it seemed that's where you go I didn't know much this is a day before the internet before you really knew anything about other places all I knew about Canada was that it was cold and McGill, that's really all we knew so we got married and came as immigrants now the reason why we came as immigrants not just as a student was because I thought well my wife would have to work to support me I would have to work in the summer I didn't know anything about the work situation and I just thought it would be easier having lived in many different countries to live in a country if you're an immigrant rather than a visitor and so when when I finished my studies at McGill we didn't even discuss whether we should stay because it was a no-brainer we liked it so initially the plan was to go there and come back we had no plan you're young, you get married but I just wanted to have the flexibility of working with everything open and really no plan I mean just to get better move up my career so how did you find Canada initially? well it was actually we adapted very easily being fluent in English and French helped because I was Belgian until I became Canadian I was always Belgian even though I was born in Cairo I was Belgian my father so we spoke French at home so it was not difficult at all my wife got a job immediately with rapid grip in Bratton in downtown Montreal as an industrial artist and I had no trouble getting summer jobs it was traditional there in McGill that the companies used to come and interview the students during near the end of the term and then you got summer jobs and I don't remember being asked whether I was an immigrant or not you just got a job and that was easy it made it possible for me to stay for us to stay so what was your first official job in Canada? in Canada the first job was actually what was called then the Quebec Department of Miners we did reconnaissance mapping in the part of Quebec that became a dam eventually I think his name was Solveille was the name of the chief geologist I was the assistant chief and we had a small party so what became it was a masters I did the masters first and in fact my only ambition when we came was to do a masters I hadn't thought further and actually I came to do what was called then the masters in applied which was the two years was in mining subjects mining and engineering subjects and the second half was advanced geological subjects so the idea that you would be more easily adaptable to work in industry that was the whole goal it was the MSC applied and then I got interested in research and stayed on and did a PhD research on my research was on the gold silver tellurium system it was a phase equilibrium study and I worked my professor was Lloyd Clark and he was very young he had just been appointed I think in McGill he was a recent graduate he had done his studies at the Carnegie institution which was very well known research in phase equilibria and that sort of thing and so I actually helped him build his original lab there at McGill and that's what I did my thesis on was phase equilibria in the gold silver tellurium system and tried to relate that to how we find gold silver mineralogy in deposits Canada was a good place for that then so what did you do after your doctorate well I got offers from industry but I'd been working in the field every summer basically other times it was fly camps every week and all that sort of thing and my family was growing so I was looking for something that was more in town and got an offer from what was then the mines branch and the attraction was they had started a research program there called the sulphide research program and the leading light there was Ernie Nicholl the late Ernie Nicholl who unfortunately left Canada and went to Australia in the early 70s but that's another story but yes so that was the attraction and there was a very strong research environment at the mines branch at that time there was for example there was people like the late John Keyes who was a physicist there was Archie Gillison who was actually building an electron micro probe which was a new instrument at the time that you couldn't even buy commercially he was building one for CANMET there were really top scientists our division chief was Dr. Alan Prince who eventually headed the inland waters branch when it was started when the environment became important we had Dr. Bob Cunningham in physical metallurgy we had other important people in metallurgy for example in extractive metallurgy and they were the ones that during the war and after had developed the Blind River and the El Dorado uranium deposits CANMET itself what was still mines branch then still had a pilot plant and I think was instrumental in many mines being started in Canada because they did the piloting then the old mines and my section head when I came he was Morris Haycock now he was a graduate of Princeton had been many years at mines branch and I didn't have much of an opinion of him when I arrived because he was only a nominal only nickel was really doing everything so he was there and he seemed to spend his time making extraordinary lunches for himself he would use a fume hood and fry up actually potatoes and things with it it was just amazing and you sort of watch us from the side but later on I happened to obtain a sample mineralized sample from a collection from Princeton University that he had worked on and I was amazed at how astute his observations were in the early 30s on that sample and now of course we could do other things with it that he couldn't do then and in fact and this is relevant to CIM Morris Haycock published a paper in 1937 in CIM bulletin that had to do with the grain size of gold in different deposits and how it affects processing minimal processing and he had done a lot of work with so many different ores across Canada that he was able to tabulate and arrive at some kind of logical classification of grain size to when you start to observe the grain and when you can't see it anymore it was then so called invisible gold and his paper of 1937 is still relevant today so my opinion of him changed and in fact I named the mineral after him Haycockite, yeah we can talk about that we can ask you partly why they're called what they're called you have discovered a few minerals so maybe you can describe them tell me what they are quite frankly I can't remember the number I can remember just some I was arrived at Canmet in 64 and as I said there were all kinds of analytical developments that were important to mineralogy and metallurgy at that time one of them is called the electron micro probe or the scanning electron microscope and as I said Archie Gilson was trying to build one but he gave up the geological survey next door they had an early prototype model called an alien micro probe now what a micro probe does is it enables you or the person whatever he is or she to analyze a particle a very tiny particle it could be as small as one tenth of a millimeter okay well that's small that is on a polished surface we call those things polished sections you cut a rock make a slice and polish it and then you can look at it under the microscope in reflected or transmitted light so with this instrument the micro probe you're able to direct a beam of electrons on that tiny particle and then you can get the characteristic x-rays that will determine the chemistry so you have the chemistry of a tiny particle let's say one tenth of a millimeter in diameter and you know exactly what it contains so it's the big jump from what I was brought up with which is you give your sample your freshest sample to the chemist they dissolve it, it's all gone and you get some result in the temperature often so here you get a not only do you get a result chemical result from something that is one phase, one mineral but you know exactly what it is so I was interested in this and I went over to Bob Trail was in charge of that Ilion probe at the time and so I talked him into setting it up I I read a paper about by the Russians about a certain kind of calcopyrite now calcopyrite is a copper iron sulfide it's very common it's the principal mineral from which you get copper, okay calcopyrite now they talked about this strange calcopyrite, it was very badly described and all but it intrigued me and so and remember those were the days of the Cold War and so I just fired off a letter on the letterhead paper and all but not to the author because the way the system worked there you write to the academician that presented the paper that's how their system was at the time so I wrote to him and asked for a sample now it was really people just laughed when I told them well two months later in a brown box I get a sample of ore from the famous Norilsk deposit in Siberia this is I mean nobody else in the west had a piece from Norilsk because Cold War secret copper nickel deposit massive, very rich platinum metals I mean this was top secret stuff you weren't supposed to give it away well I got it and I had a sawn cube about this size about 3, 4 centimeters across and so I sliced it and I took it over to Bob Trail and did a geological survey and I had looked at it microscopically first with the optical microscope and found these very bright white shiny little bits in there about a tenth of a millimeter slightly bigger and so on so asked them to analyze them and yeah so two of them were minerals of palladium now palladium is one of the six platinum group elements and it's very commonly used for example in many many alloys in industry in motor cars and for the exhaust system and so forth you know for pollution control so yeah so I had these two new minerals I knew they were new because I did some research on them and this was what led me into an interest in platinum group elements yeah it was just fortuitous you know sending a request for a sample and came in and so forth yeah so was this lab one of the first in the world to have the electron micro probe? no no no we were the electron micro probe is a French invention the first one was by Raymond Castang it was his PSD ceases in France and many people were trying to copy his design and so forth and the Russians had their own version of that in which they were the first to apply it actually for platinum group minerals and nobody else had but then I got doing it in Canada and so forth and became interested in that so very early for Canada then what happened was now that alien didn't work out Archie had to give up his project there were several companies competing and now there are two left in the world that produce these electron micro commercially and of course with the computers now because before we used to have all kinds of tubes and so forth running them and they would conk out and so forth but what happened though as far as Canada goes is we were among the first probably as a group of applied mineralogists to apply this technology to understanding quantitatively ores so we applied it and we even developed a system which was for many long time probably the fastest to do what is known as image analysis where you do a quantitative analysis of the grain size and the composition and the associations of minerals which is essential information for mineral processing and we had the system that which was really really very good in collaboration with Zeiss but for reasons that I don't know whether it was management at the time or whether it was people involved or what but we didn't commercialize it so we were pipped by CSIRO who are more driven in terms of commercializing things but they have a very good support in that domain and so their system became you know the dominant one until another company also Australian was formed who copied the CANMET system but in a more modern way and long history these two companies competed eventually one went bankrupt and now it's just one company yeah so you became principal scientist so what are there any other whether it's discoveries or projects you worked on that are worth mentioning well as I said the quantitative you know determination of ores and minerals was what we developed at CANMET and applied it to so that you know the ore process and metallurgists we were talking in the same language what they needed we could provide basically yeah there was it was difficult though because I don't remember when this started but initially when I arrived we were very very science driven and that is what attracted me but then we were introduced to the so called cost recovery which quite frankly never really worked well because it is in my opinion not really possible to compete we were not supposed to compete with industry in terms of the cost of things and yet we were supposed to get contracts and on the basis though we were better well that is subjective and so it was a difficult balancing act and then we had other complications because there were some managers that were hired with no experience in the industry at all I remember one of our managers he came from prisons now how could he be expected to make decisions on things that are very technical when you have that kind of background I mean it was really not fair it wasn't right but what could you do you couldn't go against what was done and then am I allowed to mention names sure it is well then we had a tornado arrived by the name of Erwin Itzkiewicz now Erwin came from industry I heard afterwards that he had been fired actually but he came from industry as a golden boy he was going to get us on the right path on cost recovery and so on so what he did was we were sent on many many courses feel good courses all kinds of things to to get to know each other and work together and as if we didn't work before together but the main problem he did was he destroyed our structure he destroyed we had the most marvelous technicians they were highly trained very very reliable they were like part of family and and he destroyed that by by sort of empowering the technicians so that they could create their own projects and so scientists were left without technicians maybe you've heard that from others but that was disastrous so very very few technicians were left to work with and here you were trying to bring in cost recovery do science and so it is a very difficult balancing act yeah now there was a mineral that was named after you yes by the Russians okay so how did that well you see you can't just find a mineral and decide it's new and then give it a name there is an international commission with I don't know how many countries are members but maybe say 20-25 countries have representatives so you submit your data which is the data necessary to prove that that mineral is unique and you know based on chemistry and crystallography basically and other physical properties so you send that to the chairman of the commission and he goes over it it's a very rigorous thing and he often will return it to you with more things, more data needed or whatever and then submits his approved text to the members of the commission and they have six weeks to vote on it and then when the votes come in and they have two votes they have one vote for the mineral the data itself whether the data are scientifically solid and secondly on the name so that's what's involved so in this case the Russians found this mineral and they sent it to the commission and they wanted to name it after me which is very kind of them in recognition of my work in that area because it's a palladium copper tin alloy okay, yeah so and because you were known to work a lot with platinums yeah, yeah my work in platinum actually maybe highlighted by my two books and both of them are CIM books one is CIM volume 23 and the other CIM volume I think 42 or 40 I can't remember sorry anyway they had to do with everything about the mineralogy of platinum and recovery and processing and things like that yeah, yeah the first one was published in 81 and it was reprinted in 89 because it had sold out and it was known as the Bible to people in fact the first time I went to Russia I saw in all their labs they had a Xerox copy of it because they couldn't get the book itself they managed to and they had it bound as a book a course back yeah so for those who don't necessarily know much about geology or mining what why is platinum so important well platinum is you might say an essential element but it's really more than just platinum there's this platinum the main ones are platinum palladium and rhodium there are six elements but these are the three principal ones and they're essential to industry I mean you have platinum on spark plugs you have platinum alloy as I said in the exhaust to take care of pollution in cars platinum is a catalyst in many industrial processes cracking of oil you name it there are these alloys are present yeah so the generally they are high melting alloys and very very resistant to corrosion, oxidation and things like that there's an overall very strong resistant element so it was the element that was named after you was called kebricyt the mineral kebriite traditionally you end up with many mineral names with it kebriite and this was did it have to do a little bit with the first relationship you developed with the Russians because by then it was I think 83 or something like that I published quite a lot I mean my book had come out and I've done a lot of work in that area I found several new minerals in Canada for example we found a mineral in Sudbury which we named Sudburyite in the deposits there and that's a palladium antimony mineral and subsequently we found many more now through your career what would you say was the most dysfunctional or even challenging part whether it was a project or a specific position you held well I've alluded to that it was really trying to balance you know being on top at the leading edge of science and trying to do cost recovery in a government environment it is not easy at all it doesn't work well it's got to give somewhere now who would you say would be your greatest mentor Ernie Nicholl he was a great person and a great mineralogist and but he sort of he was looking ahead it was around 69 and already there was hints about changing the policy in terms of science and cost recovery and all that sort of thing and he wanted to continue being able to work as a mineralogist and doing research in mineralogy so he was enticed to go to CSIRO and he left Canada and we lost him for good and in fact he became famous there because he was given free rein like he had in Canada he had his own here and he did the same over there until then eventually at the time of his retirement they were also going on to cost recovery but he was well away and was not affected now a few next few questions are more of social questions but the first I always ask and that's throughout your career how present or absent yeah in my field or at least at Cannes Met at Mainz Brunch Cannes Met we had very few women they were either in procurement secretarial work the only women even then was a few in chemistry they had chemists that was it and that changed of course now for example because I was involved with several universities in terms of being junked and things like that now in geology, mineralogy it seems there are more women than men for example with many of the metallurgists I deal with they are women I deal with people at Extrata and all the people I deal with are women and things like that so it's changed considerably yeah and the next question is do you believe there's a disconnect between the natural resource world and the general public and if so or if not why well I think so I think we don't do enough to explain to the public what we do there's no question to me that the public sees mining as being dirty and contaminating and so forth so I don't think we do a good enough job there but I've never been in a position to really help in that respect because it just hasn't been my role but I think that is a problem and certainly we could do a lot more I mean we could have done a lot more maybe I don't know because I speak of something I don't know much about but for example they had a very big program and probably still continuing called acid mine drainage now this was to mitigate the effects of contamination due to the breakdown of sulphide minerals and acidifying lakes and things like that now they spend a lot of money and I think they must have had a lot of successes but does the public know about that? I don't know and that is the issues because I think that the mining industry really for my views of it is at the forefront of trying to mitigate and improve things but does the public know that? I don't think they do it is true what you most likely hear about if you're outside of the industry is pretty much only the bad things exactly and things for example like you mentioned the word cyanidation and the public just goes flips and things are going to be poisoned and they don't realize for example cyanide oxidizes in the atmosphere it isn't a stable compound and so forth I mean there's little simple things that could be better explained but we just don't do it I think it's also there's disconnect perhaps because the public also doesn't necessarily understand not everybody studies chemistry or geology and so therefore they don't necessarily understand these processes or these minerals question for you why did you get into geology and mineralogy and why do you like it? Well I was interested in chemistry mainly and when I went to the in Johannesburg and I went to the person guiding the guidance counselor there about what courses I should go in because I was interested in science and I was sort of nudged to geology saying it was such a good fit with chemistry and that's how I got into the mining world really and then I was and when I started as I mentioned I was really working as a field exploration geologist and I continued to do so when I came to Canada but then I got into research and one thing led to the other I mean you know I started studying very what is you know a little bit removed phase equilibria though they are very relevant to industrial processes and then just by trying to learn about all this new instrumentation that was coming into being which was so powerful that I stumbled on some new things that got me excited and I maintained my excitement I mean I'm still working as a consulting mineralogist and enjoyed very much I have a lot of clients and but I still do some publishing when I'm allowed to publish and so yes in fact we have a paper in press in the Canadian mineralogist a new mineral that I found in the positive near marathon Ontario and we called it cold wellite after the cold well complex now so this is work I did for a still water mining company they have the property and got permission very kindly because you don't always get that permission to publish information about that new mineral so we were able to characterize it I didn't do all the characterization work because I just did enough because when you work on contracts there's only so much of budget I knew enough about it to know that it was a new mineral unknown to science but we needed to do some much more in depth academic work so I've got some friends in academia that did that and together we're publishing that paper on cold wellite it's a mineral that has palladium and silver and sulfur so you retired but not really well no I mean I'm busy I can tell now the next question there's no wrong answer and it's an opinion question really but it is a near full so I'll read it out in your opinion are there any events people inventions contributions disasters anything really that you believe must be mentioned when talking about the modern history of the natural resources in Canada so it could be someone who you think changed an aspect of mining forever could have been a disaster that changed rules and regulations for the better yeah well I wasn't involved in policy well the rules and regulations when they were changed for the worst as I mentioned cost recovery I mean that did not help that did not help and they couldn't get now maybe there is no answer I don't know okay because I was not in a position I kept away from the politics of it so maybe there is no answer because if you aren't allowed to compete with industry and yet still do work for industry there's something there that didn't somehow meld very well to me that was not helpful at all yeah because let's face it but again the world has changed so maybe I'm what I'm saying is not that relevant because there's no question that so many mines were started in Canada because of the work done at the then mine branch and you can't say that anymore so that speaks for itself and what about if you look outside of the mines branch or can met from you simply following what happens across Canada has happened across Canada for the past 50 years let's say is there something that stands out that impresses you or you believe marks part of the history of the research really think that our group there in mineral sciences were very much instrumental in introducing the concept of quantitative mineralogy which is now taken as just a no brainer is that that is what you do in terms of industry and I think that it was really our group that put all the seeds in place and did enough of the work applied work to push it through and make it recognized because I think we all appreciate that the mining industry really is very conservative and they don't like to get into new things necessarily when once something works they'll just stay with it so I think this was being our biggest contribution the mineral division there mineral sciences was the quantitative aspect of mineralogy so just a few questions to close what are you proudest of in your professional career well I don't know I'm just happy to know that people use for example these CIM volumes on the platinum elements it's a very satisfactory to know that people use them in their labs and that's basically it really it's nice to see your work being used and last question to someone much younger like a student for example and I'll divide this in two parts but first what would you tell them in regards to their future if you were to give them a piece of advice or a life lesson if you're going to go in the mineral industry well I was going to ask in general and then I was going to ask specifically in the mineral well I don't know it's a question of you know being true to the evidence to make sure that whatever you produce is correct and the integrity involved in making sure that it is I think that's very important having been long involved in publication and being currently still a reviewer of many papers I don't see as much of that ethic these days as it used to be and I think it remains to be reminded that people should be reminded that they've got to be highly ethical and to be as correct as they can and also to cite things correctly there's all kinds of games being played now in different publications with different groups you know citing each other ad nauseam and so on so to keep away from that and to just really look at the data and be sure it's correct and why should they get into the mining mineralogy industry because it is very rewarding in the sense that you're always working with the unknown no matter what anybody might say or think there are no two deposits the same no two or exactly the same and I'm still learning new things so if someone at my age can be learning new things there must be some attraction if you like that sort of thing well thank you is there anything else you'd like to add no I hope I haven't said too many wrong things about people but the policy change there was really controversial and I'm convinced it wasn't successful well thanks a lot for your time


Market opened up this morning. There is a 184,000 plus share print that came in on the tape. That's an aggregate or conglomerate of orders that are triggered just at the market open price. All right, so be it. Who knows what in all of that, right? That's the key question, right? Because for 184,000 shares, I'd like to think that is pretty big. I mean, well, how big? Well, let's look at these orange lines again. You know, if we're calling 22 big, and now at this point after the market had opened up, 2150 became big, you know, this orange line, a big level for about 14,000 shares, then for about 21,000 shares on the SQ22. All right, well, if those are big, which they are, how big is 184,000? You tell me, how big is that compared to all this? So it's not like I'm calling these small, 2150, 22, up ahead 24, 25, and change. Those are respectively large, but when it comes down to following this as a key level, you tell me. Now you can imagine,


Everything is broken and wrong. Every day I see something and I'm like I would have fixed that because I'm a rational human being. Who designed this new nerf mascot? Who designed the new Miles Morales across the spider-verse suit? Who thought orange looked good on Spider-Man? Today I will start a new series where I write the wrongs of humanity. Today I will show you what I would do if I was God. First order of business. Make Marvel Comics stop doing this.  I can promise you MCU fans would have liked Miss Marvel the way she already was. We've never seen Miss Marvel on such a huge stage before. She's gonna be shown to people who have never even heard of her before. So why are we making everyone think she has light powers? You're telling me that MCU fans wouldn't enjoy or relate to a story about a character overcoming her disgust towards her weird powers as a metaphor for body dysmorphia?  Here's the thing it's heavily speculated that the reason for her powers were changed due to this guy joining the MCU and if they had the same powers it would confuse fans.  So why not make his space looking instead? All you do when you change the good source material is piss on people who already liked it the way it was. They do this to try to make new fans happy when they would have just been happy anyway.  Changing the source material can actually be good. For example in Amazing Fantasy 15 the words with great power comes great responsibility actually come from the narration written by Stan Lee. It's not a phrase said by Peter Parker or Uncle Ben. It's almost a phrase that is completely alienated from Peter or Ben the entire story. It's just for the reader to see. But in Spider-Man 1 instead of these words being completely absent from Ben or Peter it's Ben who says with great power comes great responsibility instead of the narration by Stan Lee. This change made the words stick a lot harder to Peter since Ben died because of Peter's selfishness Peter is more obligated to actually believe in this phrase. Okay I have a confession to make. I like this design. Now I know a lot of people get very angry when they change the Miles Morales design and they did. They changed it again for the new Across the Spider-Verse movie coming a Spider-Verse. Across the Spider-Verse movie coming up. Now now now I like it. If there's one thing I would say it's that I wish we had a little bit more time with the previous one. We only got about like maybe 10 to 20 minutes of screen time with it. I just wish we get to see it a little more before they ushered in this new one. But the more I looked at this costume there's something that really bothers me about it. It's this. Why? Why doesn't the line go all the way down to the leg? To all the way down to the boot. It bothers me. It bothers me. Like look at the hand. The stripes go all the way through the hand. Why wouldn't you do the same for the foot? The hand is like the parallel to the foot on the human body.  It bothers me.  They just cut it off. Miles just ran out of paint halfway through the leg and they cut it off.    Why? See look I'm gonna fix it. I'm gonna fix it. Look I just fixed it. There you go. Alright I'm gonna I'm gonna wrap up this video with the new Craven news that's came out about like him being a vegetarian or whatever and them trying to make him into an anti-hero. Look okay let's think about it like this. Everybody in their mom thinks Venom is cool okay. When you made a movie about Venom without Spider-Man it's like okay I mean we like Venom so yeah alright we like Venom enough that we don't need there to not be Spider-Man in it. Who? Who? Absolutely who likes Craven or Morbius enough to have a movie about them.  I like and it's not even like oh I think Sony should give the the label over to Marvel.   How? How am I supposed to root for you after you made Morbius?  They clowned you. They clowned you Sony. That's done. Okay?  It's over.   The fact that they had to go on Twitter to hire more animators for the project is sus to me. Like if anything why don't you pour more money into like the greatest Spider-Man movie ever made? And you can do that.  Like it's pointless. All these other like solo villain movies are just so pointless because unless it's Venom nobody cares.  Like nobody cares about any other Spider-Man villain more than Venom. I hate to say it. Venom is the most popular one. Alright that's the one that has the most longevity outside of Spider-Man and that's the one people care the most about. You're not gonna have any success with anyone like Craven or Morbius or even like some of the big ones. You're not gonna have any success unless it's like a superior foes of Spider-Man thing. You're not gonna have anything more. That's it. That's it for this video. This might be like a one-off series because this whole video is just supposed to be short things that I cared enough to complain about. So if I can find anything else that I can complain about that's a recent that I compile all into one video then I'll do that. But this might not be a series you see very often.  Thank you so much for watching. Bye.


Everybody forced here ripping three boxes for Rory B. Rory's got Two boxes 20 22 elements football and a box of 2022 gold standard football. Let's do the elements boxes first. Good luck, Rory Forgot it's not like 20 like 20 21 Brandon cooks 121 of 199 Xenon Marcus Mariota 26 to 54 he's been playing pretty well lately Hatch car Justin Herbert trying to win the game in OT here 28 to 50 they came with the ball, but I'm off but you'll have to with the ball. I guess make the field go get a Tony Pollard auto 62 of 125 And rps rookie steel signatures one Dale Robinson One Dale 106 of 199 one sec. We'll keep going Rory Got Trevor Lawrence 98 of 199 Sam Hallorookie 91 of 99 nice triple patch Alvin Camara James Winston Chris Olava a team chemistry triple patch Focus on the car. There we go Nice triple 9 out of 10 rookie auto Jordan Davis rookie future auto 79 of 199 and steel signatures is Patrick sirtan Nice sirtan 69 of 199 and gold standard now Got a Jerome Ford rookie 77 of 99 Joe burrow 8 of 10 He's over Broncos country, let's ride go on the burrow nice low number there Jersey card gold rush Tyler Boyd 206 of 299 newly minted rookie Jersey Romeo Dobbs 269 of 399 Abrams Smith rookie auto 181 of 199 Skymore rookie auto 99 of 199 and Malik Willis rookie triple Jersey auto 16 of 149 there you go Rory Nice Malik on the back for you appreciate the order Thanks again


am right now And that is we're transferring most of our water to the ocean It's just not A system that we're paying for except by the absence of it being Productively turned around and used by either our neighbors or ourselves I'm less worried about the question as to whether When we transfer water if we can transfer water as to whether we're going to get it back Because we're that water is going right now to the ocean. We're not getting that back So I think that's a problem that can be solved We transfer whether we look at it regionally. We try to transfer it. We work cooperatively Our our our neighbors pay something into us that helps the system And we'll get to the question of Reducing our risk. I think over time. So anyway, I just applaud applaud the work that's been done here Right on time. Thanks john. Is there any other member of the public that would like to speak to this? Sir, you can step right up any other member of the public now is your time to stand up Hi, I'm still jerry paul I also want to applaud the effort of the structure of water supply advisory committee um, and it seems like you really are following the uh procedures and guidelines that water supply advisory committee sat um I When you lose a coastal well due to saline incursion, it's a triple whammy You not only lose the well, but you have to find a water replacement source And you need to find a way to transport the water to the vicinity of where the lost well was And you have a time delay. You might lose six years or something With that water like john was saying goes out to sea Triple whammy We share an aquifer with so cal creek water district. I think we owe it to ourselves to fill up their aquifer ASAP and I think we're going to do it with stream water transfers in lube Starting this december And I invite you to If there is going to be a loop test accelerate it And run in parallel with whatever else needs to be run so we can actually Turn a valve. This is the one thing all this water discussion hasn't done in the in since 2011 when When uh, uh, the county issued the report on conjunctive use we can actually Do something with I mean actually move some water this december If we set our minds to it and I invite you to Set that as a goal and and then after that set us the big goal that we have big pipes um in the places mentioned and uh, uh, A thing I call lack refer you can use any word you want We don't have a quote On how much lack refer would cost widen two pipelines put it in a well that felt and get water rights jointly Please Make a quote the other things have quotes. D cell has a quote. Thank you, sir. Next speaker, please I mean council members commissioners and water department staff I wanted to touch a little bit on the earlier presentation too where I think it's so important whenever we talk about Whatever our water supply options are is that our usage here has been flat for Years and that if I recall correctly our water consumption peaked in the late 20th century to spite thousands of more people living here so when we get frustrated community members about Housing or homelessness or traffic that these issues are not intractable and we have to address them all continue like in concert and that the flat wire usage Pattern or trend that we've been observing is a long-term pattern and that we really need to look at localized collaboration not only across the region, but also within the departments like with public works and um and parks departments so that We utilize stormwater management techniques um, whether it's green walls or curb cuts So that when a Community member goes to get a permit They can actually know that they can make a little mini water resources of management project instead of just a remodel of their home Um, and big picture. I think that there's moving towards recycled water is the best protection We have against climate change variability looking forward because that is really the most sustainable option is to reuse What we're already extracting from the environment and not pulling every cubic foot out of the sands or rivers that we can because The fish really need that habitat that we've been taxing for decades. So thank you all. Thank you Next speaker. Is there any other member of the public that would like to speak on this item? Okay, you'll be our last speaker for the evening Thank you. Good evening. Becky Steinbrenner from Aptos Thank you all for your hard work and your good reports and your dedication to solving this problem um, I I have a question in the Recycled water plan. I was surprised to see that did a la viega golf course was not Included in that it seems that it would be a good use of recycled water to water the golf course Which I understand requires uses almost as much water as ucs. He does so there we go um And I would like to thank commissioner shifrin for asking the question about the north coast of fish Fish habitat requirements. That's been a question on my mind as well So here we have the north coast stream water that is um able to be sent wherever because it is pre 1914 water rights And so what I am again asking is that that water go more to the mid county area It's not just so cal creek water districts aquifer. I I am in a small water district Also in the parisma aquifer So we have to think about those other water companies those private well users and it's not just so cal creek It is the health of the total basin area I would like to see um Rainy collectors put in along the river and the turbidity Turbidity requirements adjusted so that you could take more water from the the river storm flash flash flows um I am really happy that so cal creek water district has had the foresight To put resources toward the pipe loop study and that as I understand from mr. Taj du for is looking very good So the problem that flint michigan had hopefully we will not have here between using Ground water and surface water and the chemistry problems that could come up. It's looking very good Thank you. I do not want to see you wrap it up. Thank you. Thank you My my final thing is it's like your final sentence if you could wrap it up All right I don't want to see things injected in our aquifers because if there are problems you can't get them back And my final question is how much are you willing to sell to other areas? And how soon thank you. Thank you Okay, we'll bring it back to the council for just fine and commission for final discussion I'll start on the left. Um, there's no action that looking for this just receiving the report So if anyone would like to provide some final comments as we go around the room, I'll start with you commissioner shifrin Well, I've been trying to contain myself. I haven't done too well um, so make one final point is that We and you represent the city of santa cruz And our fundamental concern and what the water supply advisory committee worked on what the commission's been working on what the council's been concerned about and what's been sort of Defined quantitatively is how are we going to get 1.2 billion gallons of water a year? That's seems to me the primary concern If so called creek water district wants to pay the city the cost of moving water From excess water that the city has into their aquifer that's well and good But unless there's some part of that which is going to provide water to the city when it needs it Up to the amount that the city needs then I think it's not the priority the priority is coming up with Strategies that are going to provide 1.2 billion gallons of water for the city That's from my perspective what this is all about and The other Aspects of this like just what's being called the in lieu. It all sounds great. It's very conjunctive. It's very Is it it's very cooperative, but it doesn't solve the city's water supply problem? And that's our primary responsibility And you know, I don't think that's going to get lost in the process I know the staff is very well aware of it But from what we hear sometimes it's not universally shared The final thing I want to say is just to thank the staff for their presentation tonight They took what is an extremely complicated and technical amount of information and made it relatively Comprehensible, so thank you very much and within a not too terribly long period of time Thank you Councilmember brown here here. Thank you for a very thorough presentation and report and for making it Digestible to people who are not as familiar with our water system As you all are and we don't spend all our time working on it. So, um, thank you. Appreciate it Sure. Wadlow Councilmember norion. Thank you. Okay vice mayor Watkins. I just had a few comments. I agree. Thank you so much for your presentation and for Helping us, especially newer council members get caught up Um, I also want to thank our commissioners here as well for their hard work and passion to help solve this obviously very complex Situation that we're in in Santa Cruz My insight is um a little bit outside of the scope of what we discussed tonight And it's essentially around and I'm sure many of you have followed Around sort of just health equity and access and looking at affordability And if you're looking at international examples, you can see it in australian south africa and such We're seeing that folks are increasingly being coming unable to access quality and affordable water And we know that our situation in Santa Cruz is um not to that level Thankfully and that we are seeing an imbalance of equity and cost of living and household costs as as it relates to affordability here And so i'm interested in learning more about and perhaps it's for a future discussion when we discuss our values about how we can look at health equity Look at some of the aspects of affordability for water and access to all Especially recognizing that one of the thresholds is a 30 percent You know increase uh or 30 percent more for the cost component Which could essentially or will essentially most likely be passed to the consumer But um, i'm sure many other council members also hear that we have um, you know increasing fees and imbalance of household costs and and I think that we don't want to see ourselves in a situation where we have to Try to repair extreme imbalance and and how can we be mindful about equity and How that fits into affordability of of accessing a key resource or essential resource such as water in addition to the addition all the other concerns that were mentioned previously about just the complexity of how to diversify the sources and and to um commissioner shifrin's uh earlier comment about you know We struggle with enough information constantly and also increasing technology and ability to understand to gather that information for The best decision we possibly can and that's where I instinctually go And I think we have to be in action because inaction is also a policy decision So how can we balance the two moving forward to ensure that we were thinking about long-term policy solutions? Which is incredibly complex, but I appreciate the opportunity. I'm learning every time I come I learned so so much. I appreciate the presentation opportunity and for all the hard work. So thank you wish your mechas unless City manager Bernal wants to say anything I think there should be a high level of urgency around trying to make the decision in the 2020 timeline And the decision was made tonight for a stage one alert. Why because we have a very fragile system We do not have redundancy. We don't have robustness. I'd like to give One concrete example of that in one of the wettest winners in a decade last winter in february We had a break of the pipeline that comes from block lomond and No water coming through there Very little water from up the coast because we had pumps under water We were trying to pump as much water as we could out of the belt swells But draining the storage we have in the city We were in within days or a week of taps running dry That's our primary water supply. We don't have redundancy and just a couple of months later We had an algae bloom that had to take it offline again. The good news was the river had cleared We could pull water out of the river like we couldn't earlier But if that had happened like summer Again into serious problems We don't have redundancy And in our system and and that should be a goal that's a near-term goal Not wait until the last Finest data point is found three decades from now So I think the urgency is there and that should be a key element in our thinking Councilmember crone Thank you. Yeah, I I want to reiterate but folks have already said but I really appreciate the The staff presentation tonight. It was quite illuminating and just you know taking a lot of difficult information and boiling it down and I guess the last thing I would say Was when you talked about disadvantaged communities? Rosemary and I assume you're talking about sp 350 and that there's some sort of I understand that there's like We're supposed to minimize the impact on rate payer bills. And so I I I do want to keep that out there too In because there's a definition of what disadvantaged communities are but just in general In this community to be be aware of those rate payer bills because I hear a lot about that And uh, thanks again for the presentation Mr. Nabaskin For about six years now since I got on the water commission. I have been unfortunately, I suppose immersed in water To a degree that I did not think possible And I will tell you that the thing that I appreciate the most From our staff is Their ability to manage what is a very complex system and a very broad range of complex problems It is not by any means a simple system and the solutions that we're looking at are not by any means simple solutions As we go through the decision-making process to comment on something commissioner shifrin was saying We might get there in 2020 and need to buy a little more time to make a good decision But at least our process defines how we will consider that so that we won't be just sitting in a vacuum when we get there And I think for all of us as we deal with the issue of having a sufficient water supply we should Manage to keep in mind to not let the perfect serve as an obstacle to doing the good None of these none of the systems we're talking about are perfect systems They all have pluses and minuses Things that we might like better about them Thing but hopefully we will stick to doing a data driven process That really enables us to sort of put away our personal preferences and look at what's really going to work for the city So that we have a functional system Thank you cut some over matthews A combination of the staff report and the power points is a great pair. So that Put it in a binder and refer to it I want to say amen to andy's comments about our primary charge from day one has to been to meet the long-term needs Of the city of santa cruz water customers And to the extent anything we can do regionally and collaboratively we will make the effort to do But our responsibility is for our water customers And I second the other comments about Moving towards a A functional decision in a reasonable amount of time that we we do not have forever on this Just a brief comment, please know that the relatively Few questions that you generally get during presentations is really a product of having incredibly thorough and detailed Presentation so I just I want to say that because each time you present to us We don't tend to have a lot of questions and I really just it really is the amount of detail and thoroughness that you Prepare for us. And so I just want to appreciate that The Continuity expressions of gratitude to staff. I'd I'd like to note the the level of credibility that We've developed with the members of the public who've been attending this subject for many many years and have come to appreciate The focused work that you all have been doing. I think that's a a big step forward and provides a foundation for us to be able to Implement the decisions we make down downstream Oh I'd like to say this and that um, you know, I think the process that's been set up as a good one in terms of one the The water supply advisory's recommendation is also One that set a foundation for us to look at not only our infrastructure But you know having to expand the water supply What I think it also speaks to is the water departments Look for the future about how to build out our infrastructure You know, we have not as a city really invested in our infrastructure as much as we should have and how we kind of move forward to Look at our needs not only in the water department others I think this is a really good example on how we should be really considering the desperate need for us to invest in In infrastructure citywide I want to congratulate the commissioners for all their work on this. I mean, I feel like we're in good hands With you as well in terms of your review and just by the nature of your questions and thinking And time that you put into it. I feel very fortunate to have commissioners Like that providing us with their advisory opinion. So I feel good knowing that the recommendations I'm confident are one that are based on really good sound You know reason and also based on available data and information So I thank you for that and then last thank again the staff not only for the earlier presentation But just the time that went in I think this was really time well spent and I'll look forward to future updates and And I think make it an annual thing. I guess we'll we'll probably have another one in next year where the commission will join us again So thank you Commissioner our councilmember matthews and just to state what I think is the obvious These materials would be put up on your webpage and available to the public See no further con comments. We'll adjourn the meeting and again. Thank you for being here this evening


Okay, so thank you very much for coming. Last session before lunch, my name is Christian Schweder. I'm a principal software engineer working at Red Hat. And today I want to talk to you a little bit about monitoring your Swift cluster health. So I'm working like for two and a half years now on Swift. I'm also one of the core members since about a year now. And most of the time I'm helping our customers and partners with their Swift deployments and trying to commit the work upstream back to the Swift community release. So let's get this talk started. So I've got all good things comes in threes, right? So this talk will be like separated in three parts. First, we're going to have a look at the Swift architecture because this is important. You need to know what you want to monitor. So we are going to have a look what processes are running on your Swift cluster, and what your Swift cluster consists of. Next is some basic monitoring. So Swift is shipped with a few tools that are really helpful if you want to ensure that your Swift cluster runs in a good shape and your data is started a durable way. And then we will continue with metrics. And metrics are really interesting, especially if you face customer requests because with metrics you can ensure that the quality that you deliver to your customers and your users is always in a good shape and that you can detect bottlenecks early and so on. So a short overview about Swift itself. So Swift is an object storage. That basically means you don't have a block device or a file system that you are mounting. All your requests to the storage system is done using HTTP REST API codes. So that means basically you put or get requests with your data. And so for this you need a new URL, right? So your user has an URL. It wants to, the user wants to store a document, for example, and you send a put request. And this put request is then sent to a specific URL made out of a account name, a container name, an object name. This is sent to the proxy server. And the proxy server, in this case, it's only one shown, but for larger Swift clusters, of course you have a lot more proxy servers. And the proxy server itself is like the gateway into your Swift cluster. So it accepts all the cards from the users that want to use your Swift cluster. And the proxy server itself then forwards these requests to some backend storage servers. If you're running Swift with the default of three replicas, which was, which is most likely at least until kilo, then the requests will be split across multiple storage nodes. So the object is taken and it's not only stored once at that time, but stored three times on different storage nodes on different disks to ensure a high availability and durability of the data. So even if you're one of these servers fails then or the disk is broken or so on, you have very good, well, not a very good chance, but you can ensure that the data is still there. At the backend servers, there are a lot of processes running. So I've only shown a single server on each node here, but there are multiple servers running. You have servers for the account databases, for the container databases and object servers as well. And at the same time, you have also multiple replicators for all this different kind of data. You have auditors and you have updates running there. What are these processes doing? Well, the replicators are ensuring that you have multiple copies across your cluster of these objects. For example, let's take this disk here, has a copy of your object. And if, for example, the disk here fails, the replicator will take care of that later and create a new copy on a replace the disk, for example. If you're running Kilo or later, we have a new feature called Erasure Coding. In that case, you don't have replicators, but you have a Reconstructor that rebuilds the missing fragments from your objects. There are more processes running. The auditors are ensuring that the data that you have stored in the past is still readable and that the data still equates the data that you stored one or two years ago or whenever. And it does this by reading the objects or the container databases and building a hash and comparing that hash to a previously calculated hash. And in case there's a mismatch, this object, for example, will be moved away to a quarantine section and will be replaced later on by the replicators. So there are a lot of things going on in the Swift cluster. A lot of processes here. You have, of course, a lot of network traffic also, for example, from the replicators, from the proxy servers to the storage nodes and vice versa. So you need to monitor all this. Happily, Swift is quite resilient to errors. So even, for example, if a complete server fails here, your data is still accessible from the user, which is really nice and makes a very good solution. What you can also do is you can group multiple servers into different regions and zones. So for example, you can ensure that at least one copy is always stored, for example, in a different data center or in a different rack. And by doing this, you can ensure that you have a really, really high availability and durability of your data. Because even if one of the data centers burns down, you still have a copy somewhere else available. You can also tell the proxy server to store data in a specific region first. This is a feature called write affinity. So the proxy server might use a local server, local storage server, or prefer a local storage server first. And these servers are then replicating data later on to other regions and zones. We'll come to that in a few minutes again. So let's talk a little bit about some basic monitoring. So what I always recommend is that you first start with your existing monitoring tools. Most likely it's, you already have a data center, right? So you have already some kind of monitoring systems in place. Nagyos, Ikinga, maybe HP's OpenView. There are a lot of tools out there and most customers have different tools and prefer their own set of tools. So I always recommend to start with them and do some basic monitoring like is a server available or do we have filling disk and stuff like that. Swift comes with a few tools included that make this a little simpler. So for example, if you want to know if all my services or servers are available, there's a health check middleware. And you can enable this health check middleware inside your servers in the pipeline. And if you query the server then with a specific health check URL at the end, you will get a 200 okay later on in case the server is available, of course. Another tool that is really helpful is Swift Drive audit which passes your log files on your server for errors on your disks. And if there's an error on the disk, it will unmount the disk and write another log entry. So why is it unmounting a disk? If the disk is unmounted, the servers and the replicators can avoid trying to store data on these disks and just use other disks as well. So they already know that there's something broken and don't try to store data on these disks or retrieve data there. One of the really useful tools is called Swift Recon. Swift Recon, well, basically it consists of two parts. You have one part, it's a middleware inside your servers and the other one is a command line interface. And this tool gathers some data, for example, from the replicators, from the auditors, from the updaters and so on. And it's a, you can send a basic REST API request then to the servers and it returns to you a JSON dictionary. For example, it says, okay, my latest replication run took 30 hours or 20 minutes or whatever or it fails. You need to do this for all your servers. So that's, you need to configure a lot if you use your existing monitoring tools. So Swift Recon simplifies this a little bit because Swift Recon just passes your ring files and the ring file will tell them, okay, we have like 20 servers here at these API addresses, these parts and Swift Recon queries all of these servers and tells you, okay, we have five servers here and two of them are down or one of them has a really long replication time and so on. The next one is a tool called Swift Dispersion Report. Again, it consists of two parts. The first part is Swift Dispersion Populate and what Swift Dispersion Populate does is that it stores containers and objects across your cluster and later on Swift Dispersion Report queries these objects and containers, but not using the proxy server but directly talking to the storage servers and not only querying one copy of your data but multi, all copies of your data. So Swift Dispersion Report is able to detect if there are replicas missing, right? This happens from time to time. For example, if you change the ring, in that case, for example, you add a new disk or you remove disk, change servers and so on, then it's a little bit of time, there will be only like two out of these replicas available and Swift Dispersion Report reports it back to you. If you want to audit a specific account or container, there's also a tool available for this and it basically lists all the container data, containers inside your account as well as the objects and checks if these are available. Okay, so we have some basic checks available if Swift is running, but this data that Swift is running is not enough to ensure that your customers are happy.  And in that case, you need some kind of metrics. You need metrics from the past because you probably want to compare the metrics from the past to your current metrics and you want to have some metrics from now to see if there's a problem at the moment and where is the problem. So you start collecting metrics and metrics are fine, but it is that we humans are not really good in analyzing metrics if we just view the numbers, right? I mean, from these numbers, you won't see that much. So maybe you see that on the right side, there's a timestamp probably on the left side, it could be whatever it is and you don't see anything obvious here on using these metrics. So what we really want is not only the metrics itself, we want a visualization of these metrics and if you visualize your data, you see immediately, for example, here sells some spikes, here's a spike down there, actually this data comes from this region, but you immediately immediately see that there's some problem or maybe not a problem but at least an occurrence in your cluster and then you can start looking into this further. So how do I get this visualization? Well, Swift is able to send metrics to a StatsD server and StatsD is a project that was developed by Etsy a while ago. It's a very small server that collects metrics, aggregates these metrics and can do something else with these metrics. So what we are doing here is these metrics are then sent from StatsD to Grafite and Grafite actually builds this nice graphs for you. Grafite itself consists of multiple parts. There's a, well, the tool that accepts these metrics from StatsD, for example, it's called Corebone Cache. There are some database files that are stored on the server in a specific format, it's called WhisperDB. WhisperDB is like a fixed size database with a kind of round robin data. So you can store, for example, some very fine-grained data for a short period of time and then tell Carbone Cache, okay, for the data two months ago, I'm not really interested in like seconds of resolutions. I may, maybe I'm only interested in like a five-minute resolution or something similar to restrict the amount of data that I store in the long run. And then Grafite Web, Grafite Web actually retrieves the data from the WhisperDB files and allows you to visualize the data and build some graphs. There's another tool here, CollectD. So CollectD is basically doing or collecting metrics that are not collected directly by Swift itself. So for example, you can collect metrics like CPU usage or memory usage, disk, the free space on your disk and so on. There are a lot of plugins available in CollectD, so you can collect a huge number of different metrics, whatever you are interested in. You can even write your own parses and add them using these plugins and this data gets then also sent to the Carbone Cache later on. One nice thing here is that the communication, for example, from the Swift object server, the replicators or any other Swift process is done using UDP datagrams. That basically means if the StatsD instance is down for whatever reason, that you're not blocking Swift and you're not blocking the operations that your users are doing. Of course, you are losing metrics then because you can't collect them anymore, but at least the Swift cluster itself is still working without any interruption even if there's a backend to store these metrics not available. Also, if you have a lot of servers, you can scale that out so you don't only have a single StatsD instance but multiple and they aggregate the data then later on. So, installation and configuration of these tools. In the past it was a little bit difficult, actually, so especially StatsD. StatsD is in written in Node.js, so you have a lot of dependencies here. Dependency is here. GraphEat itself is written as a Django application so you have more dependencies there and it was a little bit cumbersome at least. So, I'm with Red Hat. So, we have a community distribution called Fedora and actually in Fedora 21, everything that you need is packaged already for you. So, you only need to install these packages on the left side and you have everything you need to run GraphEat and StatsD instance which makes it really simple to start building and playing around with it and to see if this is something that you might like and that you might add to your data center. So, the first four tools, StatsD, Python, Carbon, GraphEat, Web are only run on this instance, not on your Swift nodes. Only CollectD should be run on the Swift nodes because you want to store and retrieve data from the Swift nodes itself. There are a few configuration files. So, in Swift itself, it's proxy server, account server, object server and container server configuration file. There's an example in the configuration file. It's basically just telling Swift, okay, you have a StatsD server somewhere at this IP address, send the data there. There are two or three more settings. One is the prefix. The prefix is quite useful because I always tell our customers, okay, if you're using the prefix, for example, use a storage, use a node name because then you can later on see the metrics from different hosts and separate them. Otherwise, you have a huge set of metrics but you can't see from which hosts they are coming from. For CollectD, it's also quite simple. You enable the plug-in that basically allows you to send the data to GraphEat. For StatsD, it's the same. You tell StatsD, okay, here's my GraphEat instance, send the data there. And then there are two more files, storage schemas conf and storage aggregation conf to configure Carbon. You really should have a look here and because Carbon aggregates data later on and stores data for some specific time, for some specific period. So what does that mean? Let's have a look at a very simple example. Let's assume you have a 10 second period and you are collecting two metrics. And the one metric is always the same level, it's always a seven, that's a green one here and the other one is a red or orange one here. That is a different metric. By default, all your metrics are averaged and that does mean in this case that both metrics have the same average value, right? I have 21 for three samples for the red one and five times seven for the green one. The average is the same, it's seven. So if you just calculate the average of this data, that's not really useful to you. For example, if you're interested in the amount of replicated partitions in your Swift cluster, it doesn't matter if it's seven for one run or maybe there are 10,000 runs with a seven but you can't see it later on anymore. So you need to take care of the aggregation method. And in this case, for example, for the partitions or for any counted values, you want to summarize the data because the green value here, if you summarize that up, it makes a 35 and the red one only a 21. There's a huge difference here. Also, if you're running a little bit older versions of StatsD and Grafite, you are only collecting the latest sample. And if, for example, Grafite is configured to have a retention period of 20 seconds and you send samples every five seconds, only the latest sample is recognized or stored at the end. So you have to make sure that the retention period is shorter than the amount of periods that you're sending data into this. Okay, so hopefully you have then a Grafite web installation up and running. And if you have it up and running, you are opening your web browser and this is what it might look like. So you have on the left side a lot of different metrics and in this case, I used a prefix to my storage, my node names as prefix. So I have multiple storage servers here, as you can see. And below this, you have a subtree with all the Swift processes that are running here. You can simply select a metric here then and it will show up as a graph on the right side. And what you can do then is you can apply, for example, transformations to these metrics. So in this case, it's a moving average over the last 10 samples. There are a lot more, there's a very extensive documentation available and this tool is really handy if you want to play around with it and see what kind of data you can get out of it. What is even more useful then if you have an idea of what kind of metrics you want to store is a tool or is a part called the dashboard. And with the dashboard, you can arrange multiple graphs in a single page. And not even that, you can also merge multiple graphs into a single graph. So for example, you are interested in the number of current type containers, let's say, from different storage nodes. So you can just drag and drop these graphs on top of each other and then create graphs where you have multiple lines in a single graph or multiple metrics. There's also a small editor included in this graphite. And it's basically a JSON dictionary that you have here. That's the configuration of the dashboard. This makes it really handy if you want to exchange dashboards, maybe with your colleagues or you want to store the configuration in a repository or whatever else. If you look into the source code of this tool, you can even just send the data using put and get requests using curve, for example, and have this lying around as files. You also can store multiple dashboards. And this is, I think this is really useful because by default, I create dashboards with a general overview, for example, then with an overview about some storage servers, the proxy servers, and even more dashboards, for example, then down to each of the single servers in my cluster. And doing this, you have all of your data quickly available later on because if you face any problems later on, you don't want to start to build dashboards at that time point. You want to do that probably in advance. So, that said, let's have a look at a few selected metrics. So first, we have the proxy traffic here. And that's the, as you can see, in the lower left corner, you have get and put requests here. The same on the right side. These are aggregated values. And as you look at those on the scale on the left side, these are megabytes per second. And so the red line and the pink line here, these are the traffic. And here are some milliseconds, which is the response time to the user on the proxy server. It's quite stable. So the data is not fluctuating very much. What you can see here is a light increase in the response time at the beginning of the graph. But it's stable out at the end. So it's quite stable at this area. In this case, it was because there was some replication still running, heavy replication for a lot of smaller objects. And one of the multiple servers were a little bit busy. So, no, the servers were busy at this area, not that busy at that area. So the response time was higher here. What you're probably interested in is the amount of errors that your users are facing and seeing. Most likely, these are four or four errors because your users are like, or maybe an application is using or trying to access an object or a container or whatever that is not accessible or not available at the moment. What you might look out for are errors like 500 errors or beginning with a five because these are, these might at least be a problem on your cluster. If you see a lot of 500 errors and ramping up, or ramping up 500 errors, you want to have a look at these errors as well. Here we have an average CPU usage on one storage node. And as you can see here, there's a, well, kind of periodic interval. So we have at a time of 605 and then later on at 635 again, an increase in the CPU usage. In this case, the storage servers were configured to replicate or to start replication processes every 30 minutes. So when the replication process hits in and starts, there's a lot more traffic or a lot more CPU usage going on on your servers and that's why the load is increased here. What you want to look out for is, in this case, the red line. So the red line is the CPU waiting time. As a general sum of rule, I think if it's a single digit value like below 10, that's okay, that's normal. If it's increasing a lot and you have a lot of CPU usage time, it's mostly because your container database servers are somehow overloaded because you're probably running as a container databases on spinning disks, not on SSDs. That's a problem. So you want to avoid a high CPU wait time here. And if you have a high CPU wait time, you should definitely have a look at what is going on on the nodes then. Okay, so the cluster I used for this is a little bit smaller. So you can see here the free disk space on a few selected nodes. And as you can see here on the left area of the graph, there are three disks and the free disk space is quite low actually here. So I added more disks to this small cluster. So first, a new disk here. And what you can see here is that the replication hits in. So the existing disks are increasing or the free disk space is increasing on these disks. And as you can expect, the new disk is using more space here. So that's fine. That is what you would expect. But then there was another disk added, right? The brown line here. So you would expect that all the other lines are going up as well. This doesn't happen here. So if you look at this line, the new disk was added and then the usage on this disk even dropped even more. This was, the reason for this was that the ring was changed or a different storage server on the ring was changed there as well. So you had multiple partitions moving at the same time. You applied it at the same time. And only after a while, the free disk space on this disk also increased on. So probably you want to avoid operations like this where you apply multiple operations on a ring and run them at the same time. So talking about replications, these are the numbers of replicated partitions on a cluster. So as you can see here, it's quite periodic. Let's take the green one here. It's running for some time and then stopping and then 30 minutes later it starts again, running for some time and so on. Same for the brown one and for this one, for the red one and so on. What is interesting here is that the blue line, it's running much faster, only, well, maybe a minute or maybe 30 seconds or so. And so it looks like a very high peak. So the reason for this is that this cluster was configured to use write affinity. What does it mean? The proxy server sent date new incoming data to all other storage nodes. These other storage nodes store the data and later on at least one copy of the data from the other storage nodes is then replicated back to storage server number one. So storage server number one is not replicating any data back to all the other nodes but the other nodes are always storing or replicating data back to storage server number one. And this is what you can see here. So sometimes it's happening that your storage servers are busy, which is normal. And in that case, a proxy server, for example, selects another node. In that case, these are so-called handoff nodes and you see handoff requests. That you're seeing handoff requests is quite normal. What you should look out for is spikes like these. So in this case, it's quite easy to, at least on smaller clusters to do something like this. For example, if you're storing a lot of zero-bisest sized objects and your container databases are stored on disks. Because in that case, the object servers don't have to do or don't have much to do then, but your container servers need to update a lot of small objects and a lot of smaller entries. And then they might get stuck and you see an increase in the handoff requests. So at the same time, you have so-called async pending files appearing on your cluster. Async pending files, it's basically when the object server wants to store or wants to update a container database entry and the container database is currently locked because another process is also updating the same container database. So the object server stores an async pending file which is later on catch up by an updater. And then this, well, not yet done update will apply to the container database. So if you have a smaller cluster and you're sending a lot of zero-bisest objects at the same time, then you will see that the async pending files are increasing a lot, which is okay at that time point. What should happen then is that the amount of async pending files laying around on your disks is dropping again later on. So you shouldn't see that these lines are increasing all the time. If they're somewhat stable, it's okay, but you always want to see that they're either stable or going down again later on. So, and of course, you don't want to look at all these graphs at a single time. You want to build yourself some dashboards. So this is a dashboard that I used in the past as a general entry point. So you have some basic overview what is going on in your cluster. So I was interested in the average CPU usage. And as you can see here, the waiting time is a little bit increasing, which should be avoided there. I'm always interested in the proxy traffic because if the proxy traffic has some drops or some heavy spikes in it, then I know one of the users is doing something that is not done in the past probably or something else is going on there or if it's dropping a lot, maybe there's an issue with the network connectivity then. The proxy timing here in this case is also interesting. So I put that onto the overview as well as some other metrics like the free disk space or error rates and so on. Okay, so I think I'm done for now. What I'm doing is I will upload the slides to the website and at the end of the slides, there are some example configuration files. So I don't want it to go into detail right now during the talk, but you see a few configuration examples that you need to build your own graphite instance so that you can start doing this. So thank you very much. And if you have any questions, please feel free to ask. And please use the mic. No clay, you're not allowed. Then I'm gonna jump in. So I do have one question. When you were sketching out the block diagrams of where you had the stats D in the carton cache and all that kind of stuff, how do you deploy that? Are you doing a centralized stats D listener or do you do that on local machines and then kind of coordinate across that? Or what does that look like when you've got, what's your pattern for doing that? Well, it really depends on the size of the cluster. I mean, Why? I mean, I can know that. Because if you have a lot of requests coming in, stats D requests, you probably want to have multiple stats D instances running, right? What I did in the past is to separate these stats D and graphite servers onto separate machines completely. So they don't have to do anything with a storage or proxy server nodes. They were running on separate bare metal machines to have enough performance running there. The last time I did this, it was only one machine, but it was a very beefy machine. So it was fine. I mean, what kind of ratio do you look for? Sorry? Like one machine per 10 storage nodes? No, it was like 24. 24, yeah. So what you can do actually is you can tell Swift not to send a metric on every request, for example, you can tell Swift in the configuration okay, I only want to have a metric sent out for every 10s request, for example, and to lower the amount of incoming metrics to stats D, for example. Yes, we also use stats D for metrics collection. Well, I mean, obviously we use stats D for the emission and graphite for the collection, but we don't use the graphite dashboards directly. You alluded to that has the API and you can collect the data metrics and then put them into different kinds of different tools to organize your graphs. But I think for somebody getting started with Swift or we say you should do that stats D and graphite, but we don't necessarily have anything that we can turn over as a starting point. And I don't really know much about the graphite ecosystem and I was wondering is there any way that as a community we could collaborate on some of these dashboards and share graphs? Is there a chef cookbook for Swift that would include the base set of graphs to get you started? And then even if you decide to layer something on top of that, that's not just the graphite graph upboards, but something that's presenting them to your operations team or something. Do you think that might work? Could that be? Yes, definitely. So I think that's a really good idea and I was thinking about the same for one of the Friday sessions and to talk a little bit about this. So it's even simpler because the configuration for the dashboards, you can exchange them like in a JSON dictionary, which is really easy. It's easy to read, it's easy to ship and to store somewhere else. And yeah, we should definitely look into this, I think. So what Clay just said also is if you don't want to use the graphite dashboard, you can use, you can, each of these small graphs that you see is basically an URL. So if you have a web page for your internal monitoring, you can also embed these graphs there if you want to. Any other questions? Clay has one more, I think. Oh, well, do you know if any other projects are doing that? Can I go, if I'm running Nova, does anybody have like, hey man, you should check out the Nova Graphite thing over here to get started? I don't know about the other projects. Do I also know? Internally. I'll reply to that one. All right. So at Catalyst, we are using graphite and also Grafana as our dashboards and we've published on GitHub sample dashboards for SAF based on the pretty much same architecture, same structure, collective. So it's definitely possible achievable and we would be very happy to collaborate with other people on doing the same with Swift because we're implementing Swift for object storage. So, yeah. Awesome. Sounds great. I can tell you the, yeah. Yeah. Yeah, sure. Yeah, put you, okay. Okay. Yeah, definitely. So, yeah. So just to repeat this for the mic. So the idea was that we have more and more data. We have cabana data or you have cabana data, you have graphite data and so on and to really collaborate in the community to exchange common set of dashboards because everybody probably wants to run some kind of dashboards and to have a good starting point for your metrics that you are collecting because a lot of time we're doing the same work probably. Yeah, please. Could you please the mic? That would be great. So with your stats, do you ever look at trends and history and do anything with that? Yes. So personally I do. I didn't go into detail in this talk but yes, you can, there are some tools in graphite included to monitor trends. For example, if you look at a, let's look at, for example at this line, it looks quite stable, right? But in the long run it might be that one week ago it was not 80 megabytes a second but maybe only 60 and next week it's 100. So you don't see that if you just look at a very short time frame but this is the reason why normally I use trends and add them as well so that I can see okay in the long run it's probably going up or it's going down. Where it really comes in handy is the free disk space because then you can somehow predict when your disks are running out of space so they're now okay I need to order new disks within the next six weeks or so. So that's where it's really helpful and of course, well, a lot of metrics as well. So with this graphite it's only graphs but what you can do is because all of the data for these graphs are coming as a JSON dictionary you can just fetch the data and then parse the data in other external tools if you want to. Yeah, for example. Hi, my name is Sridhar, I work for a company called Vedams. We use the stats D metrics to collect it plus do some machine learning algorithms on those and try to figure out if maybe some services are down, your disks are bad, it's a self-hailing system so it's automatically going to go in, the system is going to work but maybe with a less performance kind of thing. So if an IT admin is interested in figuring out I want to know right away, services are bad, they've failed, disks are bad. So we try to use stats D to kind of figure out if some hardware is bad or services are bad and that we have open sourced. It's in GitHub, it's called Tulsi, T-U-L-S-I. So if people are interested to take a look at it, can just go with that. Thank you. Hello, it's a bit related to this project I think. Do you have some kind of tool to alert when something strange happens? I mean, if 500 hares increase suddenly? No, at least not a tool that is available on some of my wrappers but what I'm normally doing is if you have existing monitoring systems in your data center that we build like scripts around it or configurations that send alert to someone if for example there are more than 10 or 100, 500 errors in the last minute or so that you get a notice, okay something really bad is going on here right now but it's not included by default. Any more questions? How have you found the graphite language, the filters and expressions? So for example, you have one on disk free. That graph gets cumbersome in for example like Dona's cluster. And so one of the things that we try to do is apply some filters to sort of wrap some stuff up so you're like on this storage node this is the aggregate storage but then here's also a line that shows you the most full disk so you can watch. You don't have to have 6,000 lines. No, definitely not. Maybe down just to 100 nodes. So especially for disks, which is a good example I think there are some filters like show me only the top most metrics or the lowest metrics. So if you have for example disks and you have let's say 8,000 disks in your cluster you maybe are only interested in the lowest 20 disks that are running out of disk space right now. And something like that. Or you might even want to group these metrics into different regions or zones that you know okay and this rack I have three disks running out of space and that rack is fine and the next the third rack is like completely full or whatever. Sometimes. Yeah, so it can be overwhelming at the beginning fully agree but I think you get used to it especially if you use through the documentation and if you notice that you can apply multiple operations onto the same metrics you need to play a little bit around with it. Definitely. You did this all the time right? Yeah, I knew it but it really feels like something we can collaborate on. Yeah, definitely. Something that I've had to string together to get it to what I wanted. Oh yeah, that's true. That's all right, Christian's talk next in Tokyo will be on Grafana. Yeah. Yeah, thank you.


Hello, I am Dr. Murli Krishna, Consultant Cardiac Surgeon at Manipal Hospital, Bangalore. Well, the day-to-day activity is how fast the patient can resume his normal activities, will vary from patient to patient.  It depends on each pair of their pain perception, if their pain, since they are on analgesics, what we usually do is by fourth or fifth day they can resume their normal activities and postoperative recovery is quite fast.