This is Anna Adamek and I am in Kingston in Bath interviewing David Lloyd, August 10th. So could you tell me your name and where you were born? Oh, David Lloyd here. I was born just at the end of the war in 1943 in South Wales. My father was a miner, he was a Welsh mining valent. There was a miner when I was born, but then he moved out of the mines into a factory. I went to grammar school in South Wales and then went from there to University College Swansea, which was part of the University of Wales over three or four campuses, Cardiff one, Swansea another one. So I went to Swansea, it was on the ocean, which was the main attraction. And did metallurgy there, did an honours degree in metallurgy and then after that did a PhD also at Swansea. Worked on a PhD project was on fatigue behaviour of high purity iron. So just going back to your childhood a bit, did your parents encourage you to be involved in science? Not particularly, they were working class parents, they supported me in going to school, going to university. That was a driver because you didn't want to go down the mines and in those days that was the option. So down the coal mines, so that wasn't something that they wanted me to do, so they pushed me into education and so on. And at that time you didn't need a lot of money.  So it was a good choice and nobody else in the family had been to university before me. So they were supportive and so on, so it was good, I enjoyed growing up in Wales and so on. And at Swansea, Swansea is a relatively small university then, it's not now, I don't know how many students they had now. But in those days it was a couple of thousand students and the metallurgy department there was relatively small but it was considered a very good one. And I got in the metallurgy, I don't know, I did at grammar school, I did maths, physics and chemistry. And it was always the solid state physics that I found interesting, crystals and so on. And it was a toss up really between going into geology or going into metallurgy, both were attractive to me. And the good thing about Swansea is that the metallurgy course was fairly flexible. So you could take things in other departments. So they had a geology option that you could take and they had some physics options. So it wasn't just all metallurgy and that was the attractive thing about it really. And in the end I decided the metallurgy was what I was more interested in getting into. I thought that at that time of course there were plenty of jobs in the UK if you were in the physics or material science sort of area. Not the material science was a term used in those days but anyway. But there were good labs there and so on. So there were plenty of opportunities for jobs in metallurgy. So I thought that was something that I'd be interested in doing so I sort of went and did. It was good fun, I enjoyed it all. Who were your professors? Oh my main professor, well my PhD was done under Professor Greenoff. The department at Swansea was well known, had an excellent reputation for creep behavior. With people like Wilshire and so on were there and they had a very strong orientation towards creep. But creep never really caught my attention and I decided that I would want to do something different. And Professor Greenoff was a very interesting character and he was involved with one of the railway companies and so on. So I used to go with him and cast wheels up and various odds and bards when I was an undergrad. And when he came to doing a PhD I thought about going to another university but my parents sort of said, well I would like you to stay in Swansea really. So I decided that I wasn't particularly interested at that time in creep. And Professor Greenoff was interested in fatigue. And there was some interesting work being done in fatigue at that time. It wasn't as well established as creep was. And he had an interest in whether you could mitigate some of the fatigue damage after it had happened. Whether you could cure it essentially. So I did a thesis on fatigue of high purity iron initially and a little bit on subsequently on some steels. And it was good fun and it was okay. Did you have any other memorabilia advisors, academic advisors? Not really, no. I mean in those days I played soccer and cricket and whatever. The department was such that a lot of a profs played with us really. So George Wilshire played with us and whatever. So it was mainly Greenoff. And PhDs are a little bit different in the UK relative to Canada. The profs, it's very much hand off. They present the problem. They suggest what sort of things you might look at in the literature. And then that's it. And then maybe every month or whatever you'd sit down and chat with them and see what disasters you've propagated in that period of time. And then on you went really. So it was fine. And my thesis was examined by Dennis Hull, Derek Hull. And that was it. It was fine. It was good fun. Very enjoyable. So you completed your education in Wales and then why did you move to Canada? Well, I got interested as part of the fatigue work and whatever stuff that I was doing. I got interested in plasticity. And I remained interested in plasticity of metals ever since really. Deformation, fracture behavior and how microstructure influences things. And I had read some papers that were very helpful, me understanding what was going on, by David Embry. And he had come over to McMaster and he was at Mac. And I knew that Mac was a very good department in metallurgy at that time. It was a real powerhouse for its size. And it so happened that Dave had a postdoc available. And so I wrote to him, talked to him about it and he said, how can you come over? And we had been married for about just over a year. And my wife was keen, let's go and try it. And so the intention was to come over for a couple of years in a postdoc and then go back. But we never got around to going back basically. And so I spent a couple of years at Mac with Dave Embry and he had some super students. And the department was really top-notch, really bright people and interesting people. And it was really good. So I spent a couple of years there. And then when the postdoc came to an end, I thought, well, what shall I do? Shall I stay in academia or shall I do something else? And we hadn't seen the West. And Chris Tangry at the University of Manitoba had a research associate position available. Now the materials department in the University of Manitoba was part of mechanical engineering. And there were basically three profs in the materials department.  We've got the group. We have money for a research associate for a few years. Why don't you come? And that's what I did. And went to Manitoba, worked particularly with Chris Tangry and Mahesh Chattervedi, two really excellent scientists. Really interesting stuff. And they were doing very different stuff. Chris was interested in basically the fundamental behavior of grain boundaries and how grains influence things in terms of deformation. And that sort of initiated an interest in grain boundaries that I've had ever since, essentially. And Mahesh was more interested in super alloys and hexagonal close packed systems, magnesium. And he and I did work together as well. And it was a good fun. It's a nice small department. And a lot of interaction with students and graduate students. How did you find the difference between academia in Wales and academia in Canada? No, there wasn't. At that time, there was really no difference. The cultures were essentially the same. And certainly, at least in physical metallurgy anyway. So there wasn't really any major difference that struck me. There were talented people around. And it, between Mac and Manitoba, that gave me an opportunity to interact with people at other universities, meet people at other universities and companies and so on, really. What do you consider as a mentor? Oh, well, at that stage, I had three mentors. And the three people I mentioned, Dave Embry, Chris Tangry, and Mahesh Chattavadi. I learned an enormous amount from those, I guess. And that probably dictated my approach to science from that time on. They taught me how to approach problems, how to think about problems. And the essence of trying to understand a problem to a stage that you can actually predict and quantify what's going to happen. And that sort of approach I've used ever since really. That's how I approach science, essentially. And there were three of them that were really incredibly influential to me. I mean, they were really significant in terms of that. In my early period, really, those are the three people that had enormous influence on me. And do you have colleagues that you work well with, especially? I've never had any difficulty working with anybody, to be honest. But I have to admit, I only work, and I've been fortunate enough to be always in a position where I've been able to work on things that interest me and provided it interests me. I've never had any difficulty working with people at all. But as I say, I've been fortunate in that I've never really worked on anything that didn't interest me. I've always been able to avoid doing that, which is rather interesting. Luckily. So, you know, I... And interacting with people. You know, science is really... The majority of science advancement is incremental steps. And every now and again, there's a quantum leap, a significant change. And that may be due to a process, or it may be due to somebody getting an understanding that just wasn't there before. And for that, it's... Science of root is always a group exercise, really. And it may not be necessarily somebody in your department or that you're working with. But it's largely a group activity, really. And I've been fortunate to work with, particularly throughout my career at Alcat, with a lot of bright young people and so on, and still have contact with a lot of them and still do science with some of them. So it's... I've been very fortunate, I think. You know, we... Life tends to influence you when you... Have an influence on you in terms of when you're born, right? And I was born essentially at the end of the war, so that didn't have any influence on me. And then we went through that long period when universities were expanding, industry was expanding, the economy was... It was a gold mage, at least from my perspective, anyway. And if you're fortunate enough to be able to take advantage of that, and I was, that's really good. Today, well, that's a different story. We'll get to that. You chose to do your research in an industrial setting, so why did you go to academia? That's interesting. You know, I had done my PhD at university, been at Mac at university, and then at University of Manitoba. So I basically done nothing other than at university. And I had a gut feel that maybe doing something different, that this, you know, I was what, in my late... I don't know, in my late 20s, I suppose. I thought, well, maybe doing something different might be good. And particularly with... At Manitoba, with Mahesh Chattavadi, we did some work on Super Alloys, and these are real materials that he had actually been used. And I realized how complicated real materials were. So that attracted me towards real materials. But in addition to that, I decided that while I'm a reasonably good metallurgist, I wasn't really interested in what I call dot in the I's and crossing the T's. The detail really sometimes... Sometimes you need to know the detail. But in general, I was more interested in the broader issue. And if I understood a problem to a level that I could make use of the understanding and quantify what the issues were, that was as far as I was interested in going. I wasn't interested in or capable of really looking at things at an ultra-level of detail, really. So I suppose I was a bit of a generalist. And I thought that academia really should be reserved for those who can really make the step functions, not the incremental functions. And that was my perspective. And I thought that going into industry would be an interesting thing to do for a change as well. So I thought about going back to the UK. And again, at that time, jobs were not difficult to get. So I had job offers in the UK, from the US. And in Canada, I knew a couple of people at the Alcan Lab in Kingston, Larry Morris particularly, and a couple of others.  I think I was giving a talk at the CIM in Montreal or Quebec City or somewhere. And he says, well, it's on your way back. When you come in, have a chat. So I came in, and I only then realized that the Kingston Lab, the managers of the Kingston Lab had decided to broaden the perspective of the Kingston Lab. Alcan at that time had two physical metallurgy labs, one in Bambry and one in Kingston. And the Bambry Lab was more fundamentally oriented. They did very fundamental stuff, very good stuff. I mean, excellent, superb lab. Very fundamental stuff, whereas the Kingston Lab was more application engineering oriented. And the management at the Kingston Lab had decided that they wanted to look at some other aspects of aluminum metallurgy rather than just the straight application side. And they asked me what I'd be interested in coming. And at that time they brought in a few other people to basically form a separate group. And it was nominally a group that did core science, as it was called. The orientation of the lab was essentially technical assistance and applications. And they wanted to broaden it to have more fundamental work done. Not that there was a couple of people doing, I thought, fairly fundamental work, particularly in the extrusion area and so on. But anyway, they wanted to broaden it. And John Wilson and Jeff Tarble said, well, why don't you come and think about some new materials? What sort of new materials could Alcan get interested in? And I think sometime around then, David Calder, who was CEO of Alcan, had decided that Alcan needed to diversify its portfolio. Alcan traditionally is basically a smelter company. And anything that he was doing was a means of moving metal out of the smelter and sell it, really. And he decided that the company should have some research going on to enable it to diversify. And he brought in, as Chief Technical Officer, Hewin Edwards, who was Dean at UBC, I think, and also was a Queens. Anyway, he took over that portfolio and essentially had, I guess, the mandate to broaden Alcan from being on the fabrication end, being a foil and can company, which he could sell thousands of tons of metal, to doing something in, where would he go after the can? Well, because it's never gone anywhere after the can. The can is still there and foil is still there. But anyway, that was the mandate. So we looked at a whole range of different sorts of things and got involved in a whole bunch of different things at a fairly fundamental level, I suppose, to understand how various things worked. It stretched from, well, I don't know, one end optimizing the alloy for the powder that went into the solid propellants on the shuttle to automotive alloys and metal matrix composites and various stuff. Do you remember your first day at Alcan? Sorry? Do you remember your first day at Alcan? No, that's too long. I don't remember my first day at Alcan. Nothing sticks in my mind about that. That's, oh, Lord, I don't know. That's 60 years ago. I'm not quite 60, but 50 anyway. So how did you find the production side, cooperation between research at Alcan and production side at Alcan? Well, they're two different cultures, right? And the production side doesn't want to do anything new if you can avoid it, and that's understandable. They're doing well, they're making money, whatever, and now you're asking them to do something new, bring in a new material, change their process. But the knowledge there, Alcan was very lucky in my view anyway. It generally had very, very good people on the production floor. So really, before you thought about introducing anything, new technology or whatever, it was very wise to go and chat to the people on the floor and they would tell you all the reasons why they really didn't want to do it and why and so on. And you learned from that, and then you could make a decision in terms of what made sense from their perspective. And introducing a new technology to a production operation, it's a full-body contact sport. There's not two ways about it, because you were trying to get them to do something new, to take a risk and so on. And you can understand why they're reluctant to do that. But if you could make a convincing argument and you showed how it would help them and could be twisted and tweaked so that it really didn't cause too much of a problem, that was good. And the people on the plant floor at Root knew far more about what the hell the material was all about and what the issues were than anybody else did, because they were hands-on day-in, day-out. So you had to work at that, and you inevitably identified particular individuals in the various operations worldwide that were worth talking to and learning from. And if you made that sort of effort, then it was okay. But it was an element that you had to work at, really. And I think in many ways from my perspective, since I was interested in the science and the technology, not the business end of it, it was an interesting interchange, really. Did you work in English or in English and French? No, English. There was no communication. You see, the smelter operation was pretty well all French. But in fact, my torture was French and they had tortured us English and enabled us to get on. You know, I remember being in Alvader on the plant floor on the caster one day, and I needed to twist a nut. And what the heck is the French for a spanner? And I was indicating what I was needing and a colleague from Avada said, oh, a wrench. But, you know, my French is very rudimentary, really. It didn't cause anything. It didn't cause. I never had an issue with that, really. But I didn't spend a lot of time in Quebec anyway, really. You know, that was predominantly smelter. It was only, they had the bell caster, a bell caster up there and later, much later on, they put a metal matrix composite plant into there. And those are really the only two areas that resulted in me spending any time on the plants. You worked in different operations around the world, I believe, for Alcans. So could you tell me where you work and talk a bit about different research cultures? One of the things that the Kingston Management Group decided to do was try to interface the two, the Kingston and the Bambry Labs initially. Because we realized that we were pretty thin on the ground technically with research scientists and so on relative to the alcoves of this world and the passion and whatever. So one of the things that the lab decided to do was try to integrate the programs between the Kingston Lab and the Bambry Lab. And that was interesting. And to do that, they asked me to go to the Bambry Lab for a year and eventually I spent a bit longer than that there but roughly a year. And that was an interesting experience because as I said, the culture of the two labs was a little bit different in that we were, even when we were doing new alloy development or whatever, it was strongly oriented towards a particular product or application that we had in mind. Whereas some elements, not all, but some elements of the Bambry Lab were essentially almost a university sort of approach. So it was a rather different sort of culture. And when I went over there, Alcan had just taken over British aluminum and one of the big projects was aluminum lithium alloys for aerospace. Bambry did a lot of aerospace work, aerospace research. We didn't in Kingston, we weren't in Alcan in Canada, it was not in the aerospace business. So the aluminum lithium activity was a strong project, a fairly large project at that time. So I got involved in that and a couple of other projects that I was interested in as well. And we were over there for a year or so. And that subsequently set a model then for people doing interchanges and we worked out, I was the guinea pig I guess, I went first, but we worked out then what sort of interchanges were relevant, what sort of people we would bring across to Kingston and other people we would move to Bambry. And that resulted eventually towards the end of the period of Alcan, the programs, there were joint programs between the two laboratories. People in Bambry had responsibility for research and development programs over here. I had responsibility for stuff going on in Bambry at the technical level, not at the management level, but at the technical level. And we succeeded in bringing the two laboratories a lot closer together. And I did a short stint as a research director and with a research director in the Bambry lab at that time, we essentially totally combined the core programs between the two laboratories. And that was very good.  We won't go into that one. Well, I'd like to hear about that. Well, it's a management decision, I guess, money or whatever. We had essentially, after British Alcan, we then, in best, bought Alice Swiss. It was supposed to be a merger, but we bought Alice Swiss. And Alice Swiss had this lab in Switzerland. And it was a really nice lab, a nice place, close to the German border. Do you remember which town? Noyhausen, I think. And it was a different culture, again, really. And so we then had to try and integrate programs between Bambry, Kingston, and... No, Schafthausen. I was in Schafthausen three years ago. Jeez, memory. Anyway, we had to try and integrate those programs. And I guess they decided that that wasn't going too well, or they, the management, and in their wisdom, they decided to close Bambry lab rather than Schafthausen. I guess they wanted to be in mainland Europe, or whatever. And anyway, so they did that and they closed the Bambry lab. And they transferred quite a few of the people. They identified people who they felt would benefit from going to Schafthausen, or Alcan Swiss would benefit from them. So those people were identified, and they were transferred to Schafthausen. And the rest of the people were hauled well. They were paid off and went into a private consulting company called Innovale, which actually did survive and did very well. I mean, it's not called Innovale today, maybe not, I don't know, but it's done very well. But as a result of that, of course, whenever you close a lab, you suffer from the fact that you lose people. And when you lose people, in addition to losing the people, you actually lose the knowledge base that those people have. And you never recover from it. It just does not happen. The historical knowledge base is gone and the people say, well, we got the reports and thought, yeah, well, nobody reads it. So you lose that knowledge base. And I think certainly R&D in Europe, from Alcan's perspective, really declined drastically in my view from that time on. And of course, subsequently, we had a similar thing happening in Canada. Anyway, so I did that. I spent, on a permanent basis, or semi-permanent basis, there was a boundary stint. Then I did smaller stints really in response to specific program needs. So we had programs in Nippon Lake Metals. And what was the other one? Oh, dear. A file company in Japan that Alcan owned at that time. And Toyel, aluminum, it was called. Toyel was involved in powders and they wanted to increase aluminum powders. It's a really tricky thing. You've got to be very careful with aluminum powders. Anyway, we had a very good atomization technology and so on. And we decided to put, the company decided to put an atomizer into the Toyel operation, which was just outside of Osaka. So at that time, I had been doing a lot of work on aluminum powders, primarily associated with the space program and so on for the solid propellant. And so I went there back and forth to Japan for a year or so, pretty regularly, when we were putting and commissioning that operation. And then I spent some time while in the Swiss operation and some time with the German operation, automatic products and so on through the German operation. So, yeah. Very interesting. Well, you know, obviously, time in the States, to the plants in the States. I mean, Alkan was big at that time and had a lot of its fingers in a lot of operations and so on. So it was an interesting company to be in, provided you didn't mind a lot of traveling. Mind you, traveling was a lot easier that day. You didn't have to be at the airport three hours before for an hour flight. So, you know, it wasn't that onerous. And Alkan looked at you very well. So you weren't in super eight motels or whatever. They appreciated that if you were traveling and you were way over a weekend, they would look after you and they were very good. It was a super company to work for really at that time. So, you know, it's always interesting when you spend time in overseas operations because the cultures are often very different. And, you know, in Japan, in the labs and whatever, the boss was there and nobody left until the boss leave. Even if you weren't doing anything useful, you still stayed there. So that was a different sort of approach.  So it's always interesting. But people were, I always found, I think most of us found, people, the company, the culture of the company was one of friendship and cooperation.  But at base level, people were always very friendly, very helpful and go out of their way to help you and so on. And particularly in Japan was always a tad of a problem because they always wanted to entertain you because they, you know, they felt that you couldn't survive on your own in Japan. Mind you, in those days, it's a lot different than now in Japan. In those days, you had to be very careful where there wasn't much English. So we had some key signs that indicated to us where the washroom was, where the exit was and stuff like that. But they always wanted to entertain you. And after you've spent a day doing technical stuff and all you want to do is put your head down, you really couldn't do that. You know, the people in the plant or in the lab, it was a sort of thing for them, to a perk for them to be entertained or to take you out to entertain you. So it was pretty arduous in those days. But now it's a lot better. I wanted to talk to you about your patents. You have a number of patents. So can you talk a bit about your innovations? Well, you know, I worked on such a very broad range of things. So I have patents, I guess, from powders to aerospace alloys to automotive alloys to metal matrix composites to twin roll casting. They're just things that come out of the work. And you know, we always, when we reviewed a project, what we had done, we always decided then, were we going to patent it, were we going to publish it, or were we going to keep quiet about it. And we had very good patent department that gave input on that. And it wasn't the decision that I made. You know, you'd say to the patent people, wow, do you think this is patentable? Is this something we want to patent? Or what do we want to do? Do we want to publish it? Or do we just want to report it and keep it under wraps in the company? And that's what we did really.  A lot of these patents are done for business reasons as well. And that's why it was important to involve the patent and the business people on these things, because you know, you might decide to put a hedge around the technology, really. And one way of doing that is to patent stuff that really, you're doing it from, you're not trying to make money on the patent. You're doing it to protect a root patent or something in the technology. So there are a lot of different reasons for patenting something. And you know, you had priorities. A U.S. patent was the key patent that you wanted. And then a European patent and then a world patent, really. And those are the major roots of the patents that they would go through. So it wasn't something that you particularly got any brownie points from. You didn't own the patent. The company owned the patent. Why do you've got a silver that you used to without kind of used to get a silver dollar for the patents? And that was good. Because usually there were technologists that had contributed just as much as anybody else on these things, and they never got on the patent. So I used to give the silver dollars in a nice little container sort of box to the technologists and so on. Because they deserve as much as anybody else. And that was good. But I never really thought about patents or really innovation, actually. Usually with a project it may be a new material. So that you thought maybe Alkan, some part of business in Alkan could take advantage of. And that's how we got into really the powder met area. Really my interest in the powders was nothing to do with explosives and nasty aluminum powder that blows up at a drop of a hat. But it was really the fact that when you produce an aluminum powder it was rapidly solidified. And the metallurgy of that was interesting. And how you stabilize the powder so it only blew up when you wanted it to blow up was interesting sort of problem. And that's how I got into aluminum powders, not powder metallurgy. The atomic number for aluminum is 13. 13 is an unlikely number. And while aluminum in many respects is an amazing material with its alloys, it's just incredible. When you can have strengths from 50 MPa to over 700 MPa, depending on what you've done with alloys, it's incredible range of strengths. But it has major limitations as well. Room temperature is a pretty high fraction of the homologous temperature for aluminum. So thermal stability is not a big thing with aluminum. It's a big issue. I've spent a lot of time looking at ways of improving the thermal stability of aluminum alloys. So it has a lot of drawbacks in many ways. But you can do a lot with it. It's pretty amazing really what you can do. And it's carrying on all the time. So it's still interesting material. I've said a lot of things about aluminum alloys that I'm particularly ignorant about. But anyway. What was the most difficult project that you worked on? Or something you would identify as a failure? A failure? Well, I don't think there were many activities that we carried on, that research and so on, that we carried on that were technical failures. There were a heck of a lot of failures for business reasons and whatever. But there were cases where research was, after we'd spent a lot of time and effort on research, the results were disappointing. But they were mainly, in my view, from business decisions. And they might be perfectly valid business decisions. I mean, I don't know anything about the business side of things. But we spent a large amount of time and effort on continuous casting. Because some of us, Larry Morris particularly, who is a very close mentor of mine and friend of mine, no longer with us unfortunately, in the Kingston lab, he and I and a few others believed that continuous casting at the end of the day was where a lot of the aluminum was going to be produced. And there were two casting technologies within the corporation. There was the beltcaster, which was a thick, it cast a slab continuously. And it was thick and it went fast, so the tonnage was very high. And then there was another casting technology, which was a twin roll casting technology, which was thin and very slow. And the difference between the two was that the heat flux in the beltcaster was low, whereas the heat flux in the twin roll cast was very high. Apart from a metallurgical point of view, it makes a huge difference to the microstructure and so on. And Larry and I and a group of other people in the lab felt that the twin roll cast was really, at the end of the day, was where the major metallurgical advantages were. Whereas the beltcaster was a large tonnage cast, but it produced, let's say, metallurgically uninteresting material. It was fine and is fine for foil and so on, but the technology at that time, Larry used to say it was an engineering technology. Engineers run the program, you know. So we spent some time understanding primarily the twin roll caster and developed solidification models and so on. And Alcann had, I think, a knowledge of twin roll casting. It was second and none. We understood what allies you could and could not cast, what made sense metallurgically. And we did a lot of work with the University of Oxford, the solidification group at Oxford and so on on twin roll casters. And at the end, we could do things that nobody else could do. We could cast magnesium. A lot of stuff now in the literary casting magnesium allies. We had a small magnesium twin roll caster operating at that time, the home magnesium electron. And we had a small twin roll caster producing magnesium alloys 30 years ago, probably earlier than that. No, it was never published, it was never spoken about or whatever. But we could cast magnesium, the traditional magnesium alloys on a twin roll caster. No problem. Wow, there's always problems, but I mean, you know, you could cast it. The twin roll casting technology was, our knowledge base was extremely good. And then Alcann and his wisdom decided to get rid of his twin roll casters. A grave error. But we also made an error, that we being the scientists, people like Larry and myself, we should have concentrated more on trying to improve the heat flux on a belt caster. Alcann liked the belt caster because he was hundreds of pounds coming out of the caster of Robber, so they put it next to the smelter and he was an okay caster. He just metallurgically, it didn't produce anything particularly exciting. Well, not useful, it did. I mean, we produced a foil material of it and routine alloys and so on. And that was good, that was fine. But from a metallurgical point of view, from our interests of doing something new and different, he couldn't do it. And it was only actually when I had centrally retired that the group in Kingston, Mark Galno and the group in Kingston, Novelis at that time, developed what was called the flex caster, which is a belt caster, but it casts thin, five to ten millimeters thick. And its heat flux is very high. And we did not do that. We did not look at the potential of doing that at the time. Probably because, well, we probably thought the engineers wouldn't want us to go thin. But the flex caster really is a significant advance. But that was only done much later on. And now Alcatel, Novelis, then decided that he wanted to get out of continuous casting. So there's no continuous casting in the Novelis operation at the moment. Now, the flex caster, though, is in Japan. The technology was licensed or whatever to nip on light metals in Japan. And they have a flex caster over there casting a variety of alloys, including some of the 5,000-mal mag alloys. So that's an interesting development and is metallurgically interesting because the microstructural issues now are very different than they were on the thick slam casters. So that was... When they got rid of the twin-roll casters and so on, that was a disappointment, I think, because we'd spent a lot of time and effort. And we understood it so well and could take advantage of it, really. And it was a shame, but I think it was. And what is your fondest memory? Oh, I enjoyed the whole period there. But I think the... In fact, a friend a couple of weeks ago who's also retired was saying, those years in Kingston, because he was in Kingston as well at that time, it was a pretty fun time. And if you were a scientist, when you were interested in science, it really was. We had flexibility. We could do... Provided you could make a case that you were doing something that could have relevance to the company, to the corporation. You could get enough money to demonstrate a principle or whatever. And so it was a great period. But I think, and he was saying to me, well, you know, what way? I don't think he put it as what was your fondest memory. But he said, you know, what did you enjoy most about being there, in addition to the science? And we were fortunate enough, well, when I was a principal scientist, I had a budget I could pretty well do whatever I wanted to do with it, provide I could justify it at the end of the year. But we initiated bringing in people from graduate students and so on. I asked the students from overseas. And we brought in and supported a lot of really bright young people. And often we then subsequently hired them. It was really one of our main hiring routes, you know. But it gave us a route into the unit with the people in the various universities and so on. And often we'd support their PhD. We didn't really take anybody below PhD level at that time, the odd masters, but most of the time it was a PhD student. And we often supported the program because it was incredible value for money, you know. A PhD student, I don't know what it is today, but for me it was incredibly cheap to support a PhD student, double the value of the PhD student as research money for their project at university. And it gave you input into the program, into the universities, and the return was incredible.  I mean, it was far cheaper than bringing in somebody new, untrained or whatever. And we developed close contacts with them. And I think the good thing in looking back is how many of these people, I still have contact with a lot of them, who have gone on and they're now people like Dr. Mary Wells at Waterloo and various other people all over. We really know profs at universities and so on. We still have contact with them and so on. So I think that, to me anyway, both doing the science was great, but interacting with the university. And we interacted, well, the materials group interacted with a lot of universities. I mean, and it was all... I can't remember. I mean, some project didn't work out, but you always learned something, even if the project didn't work out. But we had programmes, stuff going on at Dalhousie, UBC, Mac, Toronto, overseas, you know, Australia. I had programmes going on there, programmes in the UK. And tying in with a boundary lab was good, because then we could take advantage. They had these European, large European scientific programmes. And again, for Dollarwise, next to nothing. We could get involved. We could set up a project between Bambry Kingston and then put a programme activity together as part of the European programme. And they were incredible, you know. We had activities at Oxford, Cambridge, Manchester and so on. And I guess probably, well, probably over a third of the group's budget was actually with university staff. Universities worldwide really did some work with the university in Senda, in Japan. MIT did some powder work with MIT people and also some casting work with the MIT people. So it was great, you know. You mentioned Dr. Wells. You mentioned Dr. Mary Wells. Did you work with other women? Oh yes. It was, if the individuals were talented, that was what we were looking for. And I worked, well, with Mary and also with Shazad Ismaili who's now also at Waterloo. And quite a few female undergrads and graduate students and so on. So we always had a... Well, really, we never really thought about... I don't think any of us really thought about women or non-women really.  If you're a good scientist, great. If you're a poor scientist, didn't make any difference whether you're a man or a woman, we weren't interested as the way it goes. But yeah, no, we... that was never an issue at all in the lab. Novelis took over the lab in 2005 and then closed it down in 2013. So what are your thoughts on this? Well, I was fortunate in that really I got, you know, I got sick in 2000 and they gave me five years to live. So I thought, well, we have to make some adjustments here. And when after the initial treatment I only went back to work for a couple of years and Novelis had then... it had been taken over the lab. And I didn't really see much from you to do... I didn't really... health-wise I really couldn't function as I had anyway, so it wasn't that big. But I didn't really see much synergy with the things, the culture of it, the things that I was interested in. Although one area that I did quite a bit of work on was one of the people... there was a tie-up with Wagstaff casting operation and Bob Wagstaff, who's an incredibly innovative guy, had come up with this technology of being able to directly cast a clad product. Now, clad products and aluminum are really very important with a range of applications. But cladding was always difficult. Aluminum has this oxide skin. It really doesn't like anything else. And Bob had come up with this technology of making a clad ingot on a DC caster. Incredibly clever technology, really. And I was interested in it... Now, Bob's interest in it was for brazing sheet. And that's great. My interest was not in brazing sheet at all. My interest was whether you could influence the plasticity of a material by cladding it. And how could you do that? How thick would the cladding be? What would the alloys be and so on? So we had a small project with Bob. Because Bob would cast anything. You'd tell him, oh, yeah, we'd cast it. So we had a small project to understand clad materials. What level of cladding do you have to have for a particular application? What would it do? And I was particularly interested... Again, my orientation is plasticity issues. And I was particularly interested in improving the apparent bendability of aluminum sheet. Aluminum sheet has issues with its bendability.  And I think now we understand it pretty well. We understand what's going on, what the issues are, and whatever. But I was interested in if we clad the material, would that influence the bendability? Because I said to Bob, it would have a huge influence on the bendability. And other people say, ah, forget it, it won't influence the bendability. Well, it does have a huge influence on the apparent bendability. And essentially, I did work on that project pretty extensively, because that was something that I could do without travelling or whatever. And then in... Oh, I don't know what... Well, I retired in 2000, officially retired in 2005. But I hadn't been doing... I hadn't really been doing much for a couple of years before that. But anyway, I officially retired in 2005. Then, so, Navellis came in, and initially, there was not significant change, I don't think, in the company, really. Because it was... It was a... What Navellis was is essentially the fabrication arm of Alcan. So Alcan started as a smelter company. Then it moved through the war periods and so on to be more and more fabricationally oriented, so it had rolling mills, extrusion plants and so on. And David Culver then came in to try and diversify from the can and the file to other products. And then Navellis essentially was essentially... was the fabrication arm of Alcan and was sheared off. So Alcan went back to being a smelter company in essence, I suppose. And then... So initially, Navellis was the fabrication... what had been the fabrication arm of Alcan. And then Navellis was sold off to an Indian company. And it then was essentially... essentially got out of continuous casting, got out of, well, it had already got out of extrusion and so on and whatever. And was concentrating on sheet products, really. Which is fine, I guess, from a business perspective. But it really didn't have the same... at least it appeared to me that it didn't have the same approach to R&D and so on that Alcan had and whatever. But by that time I wasn't well and maybe it was just me. It wasn't your priority. But then it was then the management, after the Indian group took over, the management went to the Americans in the States and they decided that they wanted to move the R&D center to the US. No, it's interesting. Many years before when Alcan was looking at R&D and what it cost and whatever, we looked at where was the most economical place to do R&D. And should it be in Kingston? Should it be in Bambri? Or should it be somewhere in the States? And we had a small lab outside of Chicago where there was a lab in Canning and so on.  And it was a no-brainer. Alcan really was getting a big chunk of his R&D free. It was a tax-free break. And cost-wise it made no sense to move it from... because we were looking at would we close Kingston lab or would we close the Bambri lab? Or would we keep them both going? Whatever, whatever. And we looked at the costs. Bambri was more expensive than Kingston. But if you looked elsewhere, the States made no sense to us. And I knew that because whenever I had projects at American universities, it always cost me more than double what I was paying either in Europe or Canada. But anyway, so we decided that we weren't going to close at that time either the Bambri or the Kingston lab. We were going to keep both of them open because both labs got big tax breaks and so on. And you know, whatever. But Nobel is in its wisdom decided to close the Kingston lab. And because they felt that the people would rush to go to Atlanta. Well, Canadians are a bit brighter than that. And essentially, I think maybe 10 people went to Atlanta and nearly all those were the senior management so they were getting good financial deal salary-wise to go. But a lot of the young people just didn't go. And there was no problem for them. They got jobs at university, at CanMet, at Atomic Energy and so forth. They didn't have any difficulty getting jobs. So in fact, again, the intellectual knowledge base was essentially wiped, greatly reduced anyway. And it made decisions like stopping continuous casting at a time when the technology of continuous casting was moving ahead with flex casters and high-speed twin-roll casters and so on. Massive changes in the technology. And that's the time that they decided to get out of the business. But I say business, that's what they paid the big bucks to do. They fortunately did not get out of Automotive Sheet. And that's another project that worked on for 30 years or 20 years anyway, optimizing Automotive Sheet. It was an enormous technical struggle to keep Automotive Sheet going because a few of us felt that aluminum in aerospace, sure, aircraft, great, but there were huge opportunities in transportation, we felt. And getting the weight down in a vehicle made them in a sense from the consumer's point of view and the automotive point of view and company point of view as well. But it was a hard sell. And while Aluminum Sheet has been in Automotive for many, many years, and a large tonnage project, it took over 20 years to get to that stage. But now, the dominant alloys are essentially Alcan alloys, both in the Al Maag and the 2000 series, the 6000 series.  But essentially, it's Alcan alloys are the predominant alloys and the predominant understanding of how to form these things and so on. But if you want to talk about that, I don't know if you're going to talk to him, but Mike Wheeler is the person to talk to. And he did a young man's job keeping Automotive Sheet alive within Alcan when a lot of people are like, what are we going to do? We got all this metal going into the camo. Are we going to recycle these alloys? There's always enormous reasons for why you don't do something, right? Anyway, but it led to very good understanding of what makes 6000 series, particularly medium strength, 6000 series alloys, thick and so on. And today we can quantify pretty well most of the behavior of the 6000 series alloys. Not that a lot of that's in the literature, but I'll know it may be forgotten. But anyway, there are still a few of us around who can do that. But so that was a big effort. And it took up many, many years to bring that to fruition and led to improved adhesives, improved loop. The fallout from that program in different ways that were then applicable to other activities of the company is almost immeasurable really. And often managers, managing science is very difficult because it can be expensive. It's difficult to do. You need bright people to do it. And it's risky. And at the end of the day, you cannot, it's not like building a house. You can't say, okay, we're going to start today and six months down the road, we're going to have a house. It's not how science works. So it's very difficult for people to manage science because it really depends on groups of individuals doing a whole variety of things. And to a manager, it may look totally disconnected, disconnected from what they see as what the objectives are. But almost every project that I've worked on, we've got a good understanding of what's going on and whatever. It's had fallout in other areas. And also, often when you have something like automotive sheet, a lot of what we did in terms of the processing of automotive sheet, optimum processing of automotive sheet, we learned from stuff that people had done on extrusions 20 years before. So it's a very diverse activity. And there are a lot of things that come off of it that if you're planning or whatever, you would never anticipate, really. But you need bright people to identify and say, geez, you know, why don't we do that to this? And you need talented people. You need people who are prepared to go through the grain work and understand what's going on. That's not the mentality today, I'm afraid, but it used to be. Yes, so what are your thoughts on the metallurgical industry today? Well, you have to appreciate that I'm getting on now. So my perspective is probably very wrong and very different. But it seems to me that physical metallurgy has never been in worse shape in Canada. When I came to Canada, Canada was a powerhouse of physical metallurgy, really. Not only did we have several really good labs. You know, you had McMassie at Toronto, you had McGill, and west you had Albert and UBC, all doing very, very good physical metallurgy stuff. But in addition to that, and the key thing was, in addition to that, you had corporate and government labs that were doing really good stuff. I mean, at NRC, you had people like Brzezinski, a world authority on deformation and work hardening, and his group, second to none. You had the Alkan operations.  These were all research labs, some better than others, but they were all active in research and supporting, researchers that mean physical metallurgy, and all supporting activities at universities and in their own facilities, and had people who, you need people in industry who can talk to people sensibly and at pretty well an equivalent level to the university people or the government people. And now, name one physical metallurgy lab in industry in Canada? I can't name one. There may be one somewhere, but I can't name one. They've all gone. And it's a result of corporations taking over who really don't have that mindset. And when Rio Tinto or whoever takes over another, they're not interested in Canada. They're interested in the bottom line for their company. And you look at Canada at Alkan now, I don't know. Does it have 1500 workers as part of Rio Tinto? I doubt it. And does Rio Tinto do any research? Maybe in the smelter side of things, because it's basically a large tonnage commodity company. But does it do... It may do innovative research, I suppose, in smelters or whatever. I guess it does. But in terms of physical metallurgy, there's nothing really. Even the NRC lab is still very good people there, but jeez, it's tiny in comparison with what it was. CANMET struggles on, bravely, I guess. But you need several elements for a research activity to be viable. And we go... Canada has gone for overseas investment and so on, and it's essentially wiped out the metal industry, essentially. There are a couple of small companies, you know, but it's rather depressing, I think, really. And we still have some very good people in university, but even the universities now. We have Waterloo, who's really an outstanding group of people there. We mentioned Mary Wells, but Mike Walswick was there, and a whole bunch of people there were. Shazardis Mali was there. A whole bunch of people there were very good. So we have that, essentially. Out in UBC, we have Warren Poole, who came from McMaster, and Chad Sling, sing player and so on, too. World-class physical metallurgists. I don't know how they survive, really, to be honest with you. We always, through a couple of things, we manage to have a program on aluminum scandium with Warren at UBC. And Warren Knight and Dave Embry still do a little bit of science together when we can, or at least think about science together when we can. But you've got UBC, you've got Waterloo. I really don't know what Toronto does much these days. The big and McMaster seems to me to be, well, the light, the real lights in McMaster, now retired, Dave Embry, Gary Purdy. Well, Dave Wilkinson is still there, but now he's VP Research or something. So I don't think there's much what I would call physical metallurgy. At least physical metallurgy interests me at Mac going on these days. There's the odd individual doing something interesting. I only look at, I don't know what McGill is doing these days in physical metallurgy, but I mean, I review, I'll act as a reviewer for papers, for actor, met and met trans and whatever, and I don't see much of interesting fundamental stuff going on these days other than, as I say, at Waterloo and UBC. But, you know, it's sad. What are you the proudest of? What are you the proudest of? Oh, I never think of being proud of anything. I survived, I suppose. No, I think I've brought along the careers of a few people who have done very well. And I'm proud of that, really. And they're all over, and I think that probably is what's pleased me most, really. The science has always been important, but the people are the important thing, I guess, everybody says that.  What do you miss? Why miss the people? Well, I guess it's generally sad, but I think it's also true, really. But it's very sad to see the decline in metallurgy, physical metallurgy in Canada, and to some extent worldwide. There are still a few places in Europe that are doing physical metallurgy. And there's a little bit, I shouldn't say that Queen's has a few people who are doing some nice stuff as well. Shig has now retired and Red Smith has passed on, unfortunately, and a couple of the other people they have passed on. But it's a small activity, really, and not what I would call physical metallurgy. And I don't do it now, but I was asked a couple of occasions to interview prospective students, and it's pretty disappointing, really, in terms of their... There might be whizzes and micro-electronic materials or whatever, but they wouldn't understand a microstructure if it hit them in the teeth. So, you know, but it's probably all unfair. It's just, that's my impression. And they appreciate that. But, you know, I'm sure... The disappointing thing is when you see the corporations who had, if you're a physical metallurgy and you're interested in research, it's just really disappointing to see the huge decline that's happening. And it'll never recover, no, I don't think. But there is, you know, there's still some interesting stuff going on in places, an odd place in Europe and Japan, certainly, and I presume in China, I don't know, but anyway. Is there anything else that you would like to tell me? We've been depressed enough. I think it's fine. Thank you very much. I really appreciate it. That's okay.


Welcome to Cooking Up a Storm with the National Center for Atmospheric Research. I'm Tim, a science education specialist at the NCAR Mesa Laboratory in Boulder, Colorado. Hi everyone, and I'm Chef Nancy from the Mesa Laboratory Kitchens. And today we are going to have some fun with science and food. Honestly, I'm feeling pretty hungry today, Nancy. In place of a science experiment, could I cook something with you today? Oh, of course. I'd love that, Tim. Listen, I'm about ready to make some French bread pizzas, and I have extra bread made. So why don't I send some over to you? Oh, it's fast. Here you go. Sure, you're welcome. Hey, for those of you who want to make pizza with us today, you can find a recipe for my favorite homemade dough and sauce linked in the description below. But remember, you can always use your favorite crust recipe, or any store-bought pizza dough or crust that you might like. Today we're going to start with this French bread that I've already made. And I'm going to cut it in half this way, like that. Okay. Like this? Exactly. You got it. You know, Nancy, on my ride home from the NCAR Laboratory today, I was thinking of something really cool that NCAR scientists do. They make climate models. Okay. You were thinking about climate models on your drive home, really? Yeah. So climate models are computer simulations using math equations, real data, and lots of computer coding to show how all of the earth systems might interact and behave over time. Okay. So when you say earth systems, earth systems include land, water, air, ice, and living organisms like vegetation, which all interact with each other, right? Yes. Yes. And climate models are based on global patterns in and between these systems. Okay. Maybe it's just because I'm hungry, but I think I actually see some similarities between climate models and making pizza. Okay. Between pizza and climate models, seriously? Sure. Tell me, Nancy, what are the basic ingredients that make up any pizza? Well, okay. All pizzas start with crust, like this French bread that we just cut. Okay. Let me see. Did you get yours cut, Tim? How's that? Perfect. After the crust, everything needs some sort of sauce. So I made this basic pizza sauce using crushed tomatoes, tomato paste, oregano, fennel seed, basil, and salt and pepper. Okay, Tim. So you need to spread a layer of sauce on your crust, kind of like this. Okay, Nancy. I get it. So all pizzas start with crust and sauce. And simulations have, climate models have land simulations that are like our crust with different valleys and mountains and landforms. And the sauce is another part of the simulation. And it's like the oceans, a climate model pizza. Pizza sauce ocean. Okay, I like it. All right. So next, I like to put my favorite toppings on the pizza. And today I've got some chopped bell peppers. I also have some sliced onion. And on my pizza today, I also have a little bit of sweet Italian sausage chunk. That's my pizza. Wonderful. My favorite toppings are green chilies, mushrooms, a little bit of vegan cheese, and some roasted sundried tomatoes.  This is going to be my dynamic sea ice component. And these are going to be icebergs. So all my icebergs are sundried tomatoes. And then my mushrooms are going to be my Arctic. Okay, Tim, I gotta tell you, I am never going to look at pizza the same way again. So hang on, let me get this straight. Land is the crust. Oceans are my sauce. My toppings are my sea ice. Okay, I get that, but I have to ask you, so the last thing that I add to my pizza is some shredded cheese.  Well, hello. Yes. Cheese is exactly like the atmosphere in our climate model. The atmosphere is a blanket of toppings that are made up of particulates and gases and chemicals, and they circulate all around our atmosphere. And then they actually move and interact with the oceans and the landforms underneath. Okay, cool. But I got to say it again so I make sure I've got it right. So my crust is my land. The sauce is the ocean. The toppings are the sea ice moving around.  Okay, I think I've got it. I think you do, yeah. And climate scientists spend years learning how these complicated earth systems work, and each piece of data we add to the climate model changes the simulations that we create. I get it. So just like each of my different ingredients adds and changes the flavors of my pizza. Okay, so here's the thing though, when pizza bakes, the cheese melts and mingles with the sauce and the toppings, and of course it changes the flavor of the pizza as it cooks. Yeah. Well, climate modeling is exactly the same, Nancy. The ingredients mix and mingle in a climate model too. For example, the water that's in the sea ice might melt into the ocean and then move into the atmosphere. Climate scientists call these fluxes. Fluxes, I love it. Well, I am definitely ready for some flavor fluxes in our pizza. Shall we bake them off? Oh, yes. Okay, so today we are going to cook our French bread pizzas in a 425 degree oven for 12 to 20 minutes, depending on the thickness of your crust and the amount of toppings and of course the thickness of the cheese. Baking our pizza is kind of like running a climate model. This means starting each separate simulation with the changing of the oceans and the sea ice and the land formations in the atmosphere. And each of these, those systems influence one another in the computer model just like the oven baking our pizza. Every model blends differently. Wow. Okay, this is incredible. And delicious. All right, so let's take a look at these ingredients and see how they come together. Ooh, I got it pretty done too. Okay, so look at how all the ingredients have come together on a climate model pizza. So each pizza has unique characteristics, complicated qualities, flavors and textures based on the ingredients and toppings. The ingredients change and interact in different ways affecting the outcome of the pizza. And in the very same way, that climate model data creates a variety of complicated behaviors and outcomes. You know, Nancy, I know you can make a lot of pizzas and can make the kitchen, but our earth systems are actually so large and complicated that we can't run our climate models on just a regular N car or Mesa computer. We need the N car supercomputer. Using a supercomputer is like cooking 300 pizzas on 300 different pizza stones, each with their own heat source. Okay, we would need a really big kitchen to house all of those ovens. Yes, we definitely would. But instead, N car has an entire building in China, Wyoming, just to hold the supercomputers that run our climate models. Nancy, do you always use the same ingredients when you make your pizza? Nope. Not in my world. There are no rules for pizza. And depending on what flavors I want to experience, I change it up with varying meats, raw or roasted vegetables, and different spices and seasonings based on the flavor profiles from around the world. So today's pizza, for me, is kind of a Midwestern standard issue French bread pizza topped with sweet Italian sausage, bell pepper, onion, and sometimes I add mushroom. But I encourage you to find out what works for your family, whether it's a Chicago deep dish with pepperoni or maybe a white sauce pizza with whole basil leaves, or maybe you might like a Hawaiian pizza that's chopped topped with some pineapple and spam chunks. Remember, everyone has different tastes and preferences. You know what Nancy? Climate models also have flavor variations reflecting different groups of developers located in different places around the globe. For example, some climate models might focus on simulating the interactions of the ocean and the atmosphere, like how the sauce and the cheese cook together, while others track the movement of a particular chemical throughout the atmosphere, like how the addition of my chilies will affect the flavor of the other toppings, the sauce and the crust. But all climate models include the same basic ingredients, a land of the crust, the ocean of the sauce, the sea ice or the toppings, and the atmosphere of the cheese. And exactly like our homemade pizza ends up with a unique flavor based on toppings, each climate model tells us something different based on the basic data or flavors the scientists add to their model. Wow. That's incredible. Tim, you just helped me completely understand how scientists model our atmosphere. Thank you. And I'll not forget it because I really like homemade pizza. Okay, everyone, you can find a link to my favorite pizza dough recipe and my favorite pizza sauce recipe in the description below. And we'd love to know how your climate model pizza turns out. So please tell us in the comments. And don't forget to share a photo of your climate model, I mean pizza, on social media using hashtag cookingwithencar. And don't forget to tell us your favorite sea ice topics because it'll change everything. Cool. Okay, thanks for exploring with us, everyone. We will see you next time on Cooking Up a Storm. Bye.


Welcome everyone. I'm I'm giving the first talk I guess today. Thanks for showing up early and I'm David I'll be talking about effective infrastructure monitoring for Linux with Grafana, I guess because I'm from Grafana and we also use a lot of Prometheus and Loki and node exporter so those things will feature heavily in this in this talk and So before we get started Let's do a quick Sort of catch up on what monitoring looks like at Grafana itself because we also run a couple of servers in a couple of clusters I think right now we have about eight clusters running 270 nodes I think was yesterday and for all our hosted metrics and hosted logs SAS offerings and We mostly run those on 16 core 64 gig machine so quite big things which is quite exciting and But I'm also curious on If there's sort of overlap with the technology that you folks are using so maybe just do a quick show of hands Who here is using Grafana? So yeah, that's that's quite a bit and Prometheus for metrics also, yeah with fewer and Anyone starting use Loki already? Yeah, okay. Yeah early adopters excellent and Jaeger Cool. Yeah, similarly, which will be quite a big thing for us next year getting a bit better integration for tracing and Monitoring mix-ins has anyone heard of this and is using this already. No. Oh, okay. Yeah, Chris. Oh You heard of them. Okay. Okay, that's good because a lot of Content from this talk comes from these mix from these mix-ins, but I'll talk about those later so just another kind of Philosophy that we also use a Kofana that we we try not to like we make dashboarding software But we don't want to look at dashboards all day So we try to just have monitoring by alerting so using the time series that we produce also to write alerts for and then only alert us when things are happening so that most of our info team can after work go off and make Shakshuka or whatever and Only have to worry that if something happens they get an alert on their phone or through other means and Only then they have to act so that's sort of the philosophy around this and Some of the alerting rules that I'll show later they also come from this monitoring mix-in and So Talking about monitoring for Linux for us. There isn't really a way around Prometheus and with the help of the node exporter So I'm gonna I'm gonna do a quick tour around this and initially I was looking at the node exporter website or at the GitHub repo and they read me and it just basically I'm getting select alerts and There's there's there's a bunch of collectors in there that offer you hardware and OS specific metrics and because this whole thing is written like a pluggable system and For me, this was all a bit kind of a confusing cloud so to say of of Modules and collectors. So I thought since I'm here in Berlin and Ben from GitLab It's also based here in Berlin and he happens to be the maintainer from Of the node exporter, so I just met him I just met up for him at the Korean restaurant and just talk about through How we can kind of make a better mental model of what all these different metrics in a node exporter do and But he I mean he'd like to talk Accelerate at a Korean restaurant. He just like to talk about kimchi the whole time and how he made 500 people's worth of kimchi one time, but he had a really good idea and he Remember this graphic who has seen this graphic before? Yeah, I guess we're the Linux user space conference That works and So this is a really good map from our brand Greg about Linux performance observability tools and Ben has also been using a bunch of these quite in his in his day-to-day job But the bigger task for him is always how can we also have similar? Information out of or that these tools also provide how can we have those in the sort of time series and luckily there's a bit of overlap with what a node exporter gives you and For us then it became a challenge. Okay, how can we map the node exporter metrics to what's? sort of a weight or on what we need to cover in the parts of a Linux system so we clean up this graphic and then Try to map the first couple of the first couple of metrics So if you look at the metrics that a node exporter exports They always start with node and then underscore and then and then these modules are usually the next term in the metric name And so we try to do this sort of mapping and Part of this talk now is to go a bit around these different metrics So the first and kind of classic example is CPU utilization what does CPU utilization even mean People are used to just looking at top for example, but what does that give you it's a bit unclear with time series and the node CPU seconds total. It's a bit more clear what it does because it gives you the the second spend in wall clock CPU in a wall clock time second On the CPU in the various modes that the CPU supports So we have system, user, idle, guest, and steal and all these all these different modes and so So here we're making use of this metric to draw the utilization graph where 100% would be Nothing running in idle mode and 100% of the CPU being used by Processes in non idle mode Yeah, so similar metric that often comes up for a CPU saturation is the load average which We have just been using traditionally because it's it's just been around for a long time and people have a vague idea On what it means. It's not really ideal because it's not just a CPU bound Kind of metric because It also tracks now a lot of uninterrupted task code paths, which aren't necessarily a Reflection of how resource-bound your CPU is so and for this metric we've Which is also not ideal either. We had to always normalize it by the number of CPU cores to have kind of a Zero to 100% saturation Scale basically So instead of load average what I hear what I've heard a couple of times now is to use the pressure-strolling indicator, which is I guess a new way of Auditoring how much pressure is or how much demand there is on the system from a CPU point of view and The difference there is that we no longer tracking the just a number of What is it threats That are waiting for the processor, but we're actually tracking the waiting time. So we're we'll get a better sense of How contested their resources and As you notice also, we don't really have to divide this anymore, but a number of CPUs because Now we're just tracking the overall time that they're waiting So another let's quickly go through memory metrics we use for the for the utilization here, we use the the fraction of the available to the to all the available memory and Subjectors from zero and from one and that that gives us the the use space and Similarly for saturation we've decided to use pages swap per second If anyone has a better metric to Kind of talk about memory saturation I'm also curious one feedback there and For Discuterization we're tracking iotime because so this is how often or how How long the disk operations are taking or how much time was spent in IO and Another interesting thing there is That we have a lot of metrics that actually map to Things that are given to you by IO stat So there's another sort of the space of the overlap there is quite big actually and there's an excellent article by Brian Brezel Getting into which of the different fields that I always that gives you can be represented by metrics and So one more thing about discs I think this is also something that features in every permit his training which is the fear I guess of discs getting full and We have some good alerts to or you can use the permissives queries to write some good alerts on how Fast or they will alert you when you're sort of on a trajectory that your disc will become full and in this example we're using That the disc or that the available disk space Should be or this should fire when it's less than less than 40% available and if the Available space is going to be zero if the trend on how it's filling up over looked at it over the last six hours If that's going to be zero in the next 24 hours, and I think this is a quite powerful Way to express these sorts of constraints But the and we This condition has to be true for one hour. So there's all as you notice or already there's a bit of a drawback there if the disk now fills up for some higher trajectory In less than an hour, you will have a problem, right? So you need to combine this with other alerts or other alert rules that track more aggressive Disc filling speeds, I guess and then you would you would tag them with a different severity and then route them to Maybe your pager and not just creates a warning alert and Yeah, so actually a bunch of these are defined in the note mixing which is Jsonnet based library of alerts and recording rules and dashboards and the node mixing for the has a lot of rules for Disconnectful, so not just Not just Let me just do this Not just the disk space running out but also running out of iNodes that can also happen So I really recommend taking a look at that at that mix in and even if you don't use Jsonnet in your In your organization, I think it's still good to look at the At the alerting rules to be you can just copy them out and if you if you think they can be improved you can You can also add You can you can open a good type issue on that same on that same repo and then other people Can benefit from from your suggestions Okay, just to run us up. This is some this is some network metric metric queries that we're also running and always like these these graphs where you can plot the transmitting rates on The negative y-axis and then you get these sort of nice comparison on how your incoming traffic is shaped with your outgoing traffic Okay, so here's something that happened to us recently. This is another metric that's being exposed about contract entries that tracks How full the contract table is on your system which is needed to I guess Take or take care of your of your connections and we recently had this we're running our nodes on GKE and some of the nodes just Hit that limit and then couldn't establish any new connections and so But this is also something we didn't have an alert for and this is where I just want to quickly show how you can use Kofana to To Write queries that will help you Write an alert. So let me check on this goes So I'm using the I'm using the Explorer view here and I'm already Zoomed zoomed into the time where this actually took where it is actually took place. So this was on The 12 and between the 12 and the 13th down here Notice the Shacking and then as you can see here, there was this big spike and we're hitting we're hitting one, right? so and if you compare the fraction of entries to the limit and it's one then The table is full obviously so this happened here and then What I then try to do is I use a split view to come come over here and Basically start off with the same query and then I start adding these These comparisons so I only want I only want time series basically where the Connection over where the fraction is bigger than 50% for example Then I also use I also want to only alert if they're bigger than the average of Of all the fractions and then another good Practice is to use the standard deviation like or double the standard deviation From that average so that so that you get So that you can rule out a couple of Faults false positives when for example Everything is sort of at 70% for example, then maybe that's fine. Maybe that's GKE is supposed to run but if some of those are Dba too much from those these will then show up here, right? So if we change this year to Let's say three And then run this again Okay, then Yeah, it's good Good good point No, I didn't work Yeah, so now we have fewer And then so on this side now I would be modifying this query and then when I'm happy with it basically I'm happy when I've identified The offending time series that are on the left and if they survive on this side I know I have an alert alert rule that would have caught this thing, right? And then this is what I would copy out into my alerting rule Okay, let's go back to the talk Is that working? Yeah, I've lost this window Quick here, I just have to Find my notes So there we are there the notes Oh God, this should be easier and then can I just go left? Yes Perfect All right, so a couple of collector gotchas It's good to be aware how many time series they produce and when you run a note exporter They it also runs or you can also exit on port 9100 on your system and then you see the huge list of things on the usual of time series and sometimes So there's a bit up to you then to figure out which which collectors you you want to enable to not have too many time series and then also Some legacy collectors they actually run scripts or execute programs and then so this puts another puts additional burden on your system For in terms of volume of time series, you could obviously write relabeling rules for to just drop a couple of these But maybe that's a bit too tedious. So I was pro tip from Richie Who builds data centers in a spare time and he tends to run two note exporters on each machine and one with a minimal set and one with a full set so that the stuff that's regularly scraped is is just contacting the one with the minimal set and if they need to investigate a bit further or look at the just some some things that You would otherwise have to log in the machine and Look at the proc file system You can then just go to the second note exporter that has the full set and look at the metrics that are exposed there and I'm told that there's a lot of savings to be had there. Also. He's a big fan of entropy alerting because when you run data centers some hosts sometimes run out of entropy Which to me always sounds a bit like something from the future And A little bonus collector everyone who's also here told me about this the the text file collector and This basically adds little text snippets from From files that are in a given directory to the exported metrics Text that's being parsed by your previous so and There's some some nice examples for this for example If you have If you have maintenance jobs like running backups or Pushing out new updates of your system you can track those with just writing it writing this This little line like when the last run was and then you could you could also envision and alert that basically checks this timestamp and compares it to the time that's now and let's say if it's If your backup hasn't run in like three days, you should probably alert and then another nice one is if you have If you have a lot of If you run if you run a lot of software directly on the metal Maybe in your whole fleet you have some features enabled on some and On on other machines you haven't we have a lot. It's good to have a general overview over Over how many parts of your fleet have a certain feature enabled certain metric like this just helps you track this then another fun one is Tracking how long your SSE discs are gonna last so speaking of rolling out and Managing whole fleet. I'm always a bit jealous when I look at the the git lab The gilab dashboards I highly encourage you to look there, too. They expose most of their infrastructure monitoring Dashboarding on the internet so you can go to dashboards get lab.com and just see how they are doing this Here in this particular example, they are they are tracking the kernel version distribution on all their nodes so here in their production system they seem to have around 200 nodes and The majority of nodes seems to be running a 410 kernel Yeah Which if we paid attention We'll probably won't have the pressure indicator yet, right? So this is how you can kind of see what sort of capabilities your fleet has and then they have a really cool one down there, which is the tracking which Which hosts are deviating from these majority versions and and then you can which also gave mean idea And I also heard this from Björn that at SoundCloud they had these leaderboards about outdated systems So you can write queries that basically a group by team or by cluster on Which team has the most out which team is running the most outdated hardware So these are the You just click here And See this works. Yeah, so this is our cluster now So we do have around 246 nodes for example, and I'm doing the Top-k query here Let's see if we have a couple more So there's some older ones Actually they're running I guess similar counter versions. They're just not very normalized. I guess that's that's one of the problems with this But Here on the other side, I'm using a similar query. We're amusing the top one and then I'm Counting again grouping by cluster so I can see which of these are Not running This majority Kernel version basically and so this is and then so a similar query You could use in a dashboard to build this sort of leaderboard that that I showed you so Back to this one So I guess the bigger the bigger question is also how should I organize all of this all of the views in Gofana and Here's an example from GitLab again, they do a really nice thing where In their general fleet overview, which includes all their all the hosts for and in this case the production system They do this thing where on Each row They have for example the CPU Utilization and on the second row they have the load average But they have grouped this in columns by the application tier right so on the from the left You have the web workers, then you have the API and then you have the git workers and this is sort of in your in your head how GitLab I guess is supposed to work and Here you can already see quite easily I think on How busy each of these systems is right now and I think this I think this is really helpful for Capacity planning for example, and then this is how we do this internally at Gofana we use the the mixing again because that gives us the the dashboards right away as code and that has a cluster dashboard as an aggregation of every node in the cluster and then also a Template query driven No dashboard which basic basically just runs this node exporter query and then returns us all the instances Which is our way of Service discovery or like no discovery because every system that runs a node exporter should expose this thing And That was it. Oh Yeah, we're also hiring But I'm also really curious if if there's any questions you have about the note mix in the dashboards or I'm also curious if there's any sort of funny alerting Rules that you had to write in the past. So that's it So I think your arm went up first so the question was what's the minimum version to be running to Have the mix ins and the answer for this is probably It should it should already work with five I think because it because the mix ins Is something that you run in your build pipeline to output json that you then feed into Grafana instance and I mean I think if talk was here He would he would say like you can go back to version two, right? But I don't know if I if I can make that promise. Yeah, but version five should be fine. Yeah Yeah, stop stop stop Thanks, Ed from packet Did you you saw I saw one metric that you pulled which was SSD related Drive drive health. Have you done on any any other bare metal monitoring of note of like looking at the underlying Hardware for the system that you're monitoring So so we we at Grafana. We don't but I would imagine that that this is commonly done and If there's any sort of if there's any sort of Little tool that can output something to to a text file Then this will be one of the text file collector use cases where you have a the cron job running and producing this Occasionally and the cron job would run its route and has access to all these things and then The the note exporter then exposes it and Prometheus comes occasionally and reads that value Yeah, yeah, so for yeah anything yeah anything that's not directly included as a as a module for For note exporter you can probably easily Replicate with the text file collector. Yeah, good. Thanks. Oh It's Casey. Hello. It's at the Korean dinner Korean lunch as well. Yes Do you know of any best practices for ignoring network interfaces that are useless?  Yeah, so we We we actually have recording rules that ignore Yeah, we have recording rules that ignore the looping interface for example So you would write you would write a similar recording rule So it would be ignoring useless something like that's literally a name that we chose for the recording rule Yeah, so Maybe not a really satisfying answer, but yeah Hi You create those rules the alerting rules manually, right? So are there any tools that you could use like say that? Okay, we've had a problem at this time and this time series show That something is going wrong and then you would get an automatic alert that will alert later on when the same thing happens again. Yeah, so There's two parts to this if it's if it if you can answer this statically like with like let's say a certain value should never be Hired and 70 or something you can use just Grafana based alerting where in the panel You can you can draw a threshold and then and then that is run in the Grafana back end as a continuous alerting service if it's more dynamic and sort of the threshold is Yeah, kind of dependent also on the rest of the traffic then Prometheus Prometheus Prometheus based alert will would be better But there's no currently there's no good interface on how to write these and I think I've shown you the way the way I would approach this just with this sort of split view where I can see this is all the results And I only want to filter out the ones that would have alerted, right? And then since the since the Prometheus alerting rules are just like queries Where then you just have to also say for how long should this condition be true? Then I think it's pretty easily copy and paste the ball. Yeah Thanks, so That's one more cool Just one more You mentioned one Component the PSI needed a 4.20 kernel. Are there any other kernel versions of note? I know this is a user space Linux conference Yeah, so this yeah, that's actually what I'm also curious about and maybe people here Actually know the answer and it's because that's also a feature that if you looked at earlier at At our majority version. It's a feature that we can use either since we're running on GKE and it'll it might take A good two years. Yeah before this will become available for Anything that runs in the cloud which is a bit sad but Yeah Cool, I don't see any more questions. I'm here the rest of the day. I might pop out real quick to the brand bouquet for the Climate strike, but I'll be I'll be there for for the evening event as well if you have any questions Cool Alrighty, thank you