So thank you very much. It's a tremendous pleasure to be here, so I'm delighted to come to yet another party, lots of birthdays this year it seems. So I wanted to follow on actually from a really nice talk that you just heard with a kind of a different view of sort of a broader class of Pilaritan systems and just to sort of lay out the talk. I thought what I should do is to start by saying a little bit about a microscopic model for Pilaritan conversations in quasi-equilibrium theory. Just to give you a bit of background, because what I really want to talk about in this talk is systems which actually have other interactions in them beyond the interactions that we classically think about with Pilaritons. Nicely described. And so in particular there are many systems that one can make condensate itself potentially in effect, some which have been achieved, which involve quite strong coupling to the electron-phonon interaction. And there's a question about what that does. And in particular can you condense into systems which actually have a phone on side dance, so if you have a strong coupling there what would that mean? And then I'd like to spend quite a bit of time talking about cavity-coupled Rydberg atoms which have the interesting effect that in the excited states the excitons which you've created or the Rydberg states of the atom have strong and long-range interactions. And because of long-range interactions there's going to be competition between the kind of superfluid or Pilaritan state that was just talked about in the last talk and actually some kind of quantum crystal and what might that look like. So quick reminder, without any worries about Copenhagen and other statistical interpretations, the kind of systems that we can eventually talk about are typically 2D. You have excitons which are typically confined formally to a quantum well or some array of quantum wells. They're pretty much dispersionless on the scale on which the light modes in the cavity move. You have a confined 2-dimensional like photon mode and by tuning the mirrors correctly you can bring this photon mode more or less into residence with those two. And that of course means that as quantum mechanical objects these two things, if there's a dipole moment to connect these to mix these then there's some splitting here which is the ruby splitting which can be quite substantial in some systems. And as you've just heard, at least formally ignoring inelastic processes, decays, whatever. And I will say assuming the rotating wave approximation which is actually critical because it means that there's actually a conservation law built into the Hamiltonian. So assuming that and that's generally a good approximation but not exact, then you have this kind of picture. The polariton wave function is a mixture of exciton and photon with that mixture depending on the detuning of the level so it varies enormously across here. Furthermore the nice thing about the system is you get to look at the occupation of these states because the mirrors are not perfect. Eventually the component will tunnel out and you'll be able to observe an emitted photon and if the mirrors are flat the transverse momentum is conserved and therefore the angle of emission can be converted into a momentum in here. And the reason that, no this goes back a long way, but the reason that this is interesting from the point of view of condensates is that down here you have a branch with a very light effective mass, typically around 10 to the minus 4 of the electron mass, you also have roughly the characteristic coherent scale, scales inversely with that. So relatively dilute gases down here can in principle have quite strong on sets of coherence. So what's been seen run through some experiments, indeed this is momentum distribution that you get to see by looking at the light so this is a paper going back in 2006. If you look for the, if you look now for what happens to the dispersion of that lower polariton branch when you go into a putative condensed phase you begin to see something which looks quasi linear as if a buggy-lubar spectrum. You can look at real space images and demonstrate coherence seeing fringes and you can also see vortices, in fact you can see half vortices because there's a polarization degree of freedom here as well and that's also locked by the transition. Because it's 2D you might expect to see power law correlations and you can find those, you can find ways of setting up condensates that move and here's a condensate moving and there's some kind of object in the way here and you can see going from super flow around the object to ejecting vortices and and there's solitons and then you can do fancier experiments by pumping pairs of condensates and having them coupled and looking at the dynamics of how that happens. So there's been quite a lot of work much more than this actually which has happened really over the last decade. So for what's useful I want to move a little bit on to theory space about how I like to think about these and I will say right at the start that I don't like to think about polaritons as particles and I'd like to explain why. And not for the same reasons you heard in the last talk although I agree with that also. It's actually the fact that there are very strong renormalizations of the exciton and photon component in this and I'd rather like to think about models which actually have both degrees of freedom explicitly. So let's imagine just as a theorist that you have a bunch of dots coupled in a cavity and each dot can have an exciton or not. And if you're in a regime where a double occupancy doesn't happen the natural model to describe this is actually a spin model where an empty state is spin down and a singly occupied state is spin up. And of course you flip the spins by absorption emission of a photon. So this is the simplest Hamiltonian you can write down for this. This has a single cavity mode at a frequency omega and principle there's a dispersion as a set of modes. There should be a K index on here but it doesn't matter. This is the energies associated with these spins. Typically there's very little dispersion in here. There may be quite a bit of disorder so it's often useful to think about this in a kind of a site based index. Here is the Rabi coupling between this and this is the interconversion term as I say written in the rotating wave approximation. And the consequence of writing it in the rotating wave approximation is that the total number of excitations is conserved. If you go outside the rotating wave approximation and that actually begins to be important in very strong coupling then you need to be a little bit more careful with that. There's a parameter in here which I call n but it really is simply associated with the fact that the wavelength of light is very big and the size of an exciton is of all to the bar radius or whatever the trap it's in and that's very small. And so typically there are a large number of exciton states which are available within a wavelength of light so n is a very large number. So as a theorist when you see this Hamiltonian when n is large you know what the ground state is. It's a coherent state. It's just a large n limit of this and so here is a generic coherent state, a coherent state of photons. This is a coherent state of spins. You know if you turn spins into fermion pairs you get something which looks exactly like BCS and you can also go through the standard BCS style argument with all of this and you get a transition temperature which depends on the coupling constant and generically on the density of states. So this is what the mean field theory looks like. You can follow that mean field theory, do that mean field theory to find out doping and so this is concentration. This is the total number of excitations. We don't know whether the excitations are photon like or exciton like. You start off with a lower enough polariton branch as you add particles to the system. We're here to find out temperature. Here's the chemical potential. Chemical potential rises. At this point it hits the bottom of the band and at that point the spectrum renormalizes. The upper polariton starts to push up. The chemical potential clamps. You get coherent light emission associated with this. This is a conversate appearing here and then there's of course a ghost branch that comes down here. Built into the model by the way of course is a parameter which we tend to call detuning which is the difference between the photon frequency and the exciton frequency. Here I've had them tuned together. It's worth discussing what happens when you tune the photon mode way above the exciton. Then intuitively what happens when you do that is that you're going to have excitons coupled by virtual transitions through the photons. As that happens you actually get a model which is kind of more familiar from Bose Hubbard. You get the appearance of mottlums. Let me just go through this rather busy fugue graph about what happens as you detune. Here I'm just plotting the chemical potential which sits in the polariton band again as a function of pumping. If you look down at the bottom I've broken the wave function up and because this is by the way a coherent state. This is entirely classical. I'm in the coherent state limit here at least in the theory. Here what happens is that the exciton computation is here and the photon curve is up here. What you see is that they run in parallel but after a while of course you can't put too many excitons into the system. So it saturates but the photon number is allowed to grow without bounds. As you start detuning what happens of course is that the ground state becomes more exciton like and less photon like. But interestingly if the detuning exceeds a critical number which is actually 2 something odd happens. The exciton number grows and grows and grows because of course the photon is far above it and you're filling mostly exciton states. So it's really an exciton cognizant there's very little photon occupation. But you notice interestingly that the exciton cognizant grows all the way it gets to a half in these units so this is fully occupied. So this is just a field band and at that point of course you can't have a cognizant because there are no excitations. And the photon occupation goes to zero and then you begin to get a photon cognizant growing on the other side. So this is just the appearance of a modeler because you've got you've you've spin half is like a Bose Hubbard model with infinite u. And so in terms of a detuning parameter there's actually a mod insulating phase which is actually in the model at least it's hard to see experimentally. And then there's a super radiant phase which is sort of outside. A bit more background that was mean field what happens if you go beyond mean field going beyond mean field. Of course what does it mean you actually have to include the finite k modes the boggle above modes and to incorporate all of those. So if you look at what happens this is the dispersion in the normal state in the you know at this leading order one loop level. This is what the dispersion curves look like now so these are the boggle above mode this is the lower and the upper mode is still here. And as you go through this line this the spectrum changes you can actually do this with disorder it doesn't matter I'm worried so much about that. But of course those fluctuations then are normally populated thermally and that would be in the low density region the dominant method of producing decoherence when you go to a finite temperature. Because when I do mean field the transition temperature just depends on the rubby coupling it doesn't depend on the effective mass at all. And I told you that the reason for believing that we should be using polaritons to make condensates was because their effective mass is very light so where did that go. And it's quite clear that the let me just run through all of this. So the mean field transition is a function of density scaled by something like you know the characteristic size of an exit on the ball radius is going to be some curve that comes up like this. And if you went and did however your standard theory for the cost of a stylist like theory for a transition here of course you get a BC transition which would be roughly linear as a function temperature as a function of density. So down here in the dilute limit of course you're dominated by phase decoherence associated with the acoustic modes but once that transition temperature exceeds the coupling constant you're dominated by decoherence associated with thermally populating the upper polariton line. So where's that crossover? Interestingly in these experiments well this line is very steep because the effective mass is very small. So the scale is roughly determined by taking the binding energy of the exit on the ratio of that to the ridboat coupling so this is a number which is of order one. Sometimes it's bigger than that sometimes it's of order five but then there's an effective mass ratio so this number typically turns out to be about ten to the minus four. So only very dilute systems are in this region and it's actually very hard to do experiments in this region. Most experiments are in that range. So that's an argument by the way for a theory which is of course based on long range interactions which is then very stable essentially close to a coherent state. And taking that one step further that provides a justification for what everybody does in practice which is to use Gross-Piedevsky in these systems. And you can add to this of course decay processes which allow things to decay into the bath and things to come back in. And what that does in the end is it adds lifetimes to the quasi-particles and gives you a damp driven Gross-Piedevsky equation to describe the dynamics of all of these systems. And that is often the most effective way to describe real experiments. So how am I doing? I started a little late so I'm going to keep going. I'll keep this piece in. So that was really just intended to be a bit of background. And now I want to talk about some recent stuff we've done on two separate problems. One some work actually with Jonathan Keeling's group on what happens if you've got systems with strong electron phonon coupling. So why might that be interesting? Well let me just describe the model I want to talk about. I'll call it Dickey Holstein. So it's the same one we saw before. There's a cavity, there's an exciton, there's a ruby coupling. But in particular I want to think about systems which may be made with Frenkel excitons on organic molecules. And there are many systems which are like that which can also be put into cavity, couples. Organic molecules typically can have very large ruby couplings. But they also typically have very strong local phonon modes. And so I've added in here just a local Holstein coupling on the mode. And the parameters that describe that are of course the frequency of the phonon and some dimensionless coupling strength. And as you know, if you have an exciton coupled to signalling the photon, the effect of this is that the exciton will develop side bands associated with adding and subtracting phonons. So rather than have a single set of excitonic modes, I've got excitons with side bands. And I've got all of those side bands and the question is can you have conversation into a phonon replica? So the way we've treated this is to do essentially a mean field treatment for the photons arguing that that's a reasonable thing to do because of the very long wavelength and the long range coupling. But we do numerical diagonalizations on the phonon. So you can break this up and you can just diagonalize the phonon states and work through that. So what happens? So the first thing that's interesting if you look at this. So this is now a phase transit boundary with temperature and an excitation number. I've gone of course to very high excitations. It's not particularly physical to go up here. Just to notice that one of the effects of doing this is because you can actually get re-entered mottlopes. So the blue curve was the one I showed you before. If you add electron phonon coupling in here, actually quite strong electron phonon coupling, I would say S equals 2 is pretty big for a Hoang Ries parameter. Then what happens is that you shrink this regime in response to the stabilizer if you like the mod-insulating states. But if you look carefully, and in particular if you look down at the bottom of this band, so this is raising the chemical potential just beginning to fill it, you get some re-entrance associated with this. So in particular if you sit here and raise the temperature, you go from an empty state to something which actually wants to go into a condensed state, and what's happening here. And that's really related to the problem I told you one should worry about, is I've got a whole bunch of phonon replicas. And of course at a finite temperature I have phonon replicas which exist below the exciton line, because I can steal a phonon, a thermally occupied phonon from the system and use it to dress the exciton. So I can go on the anti-stokes side at finite temperature, and that's actually what's happening here. So another way to look at that curve is, you know, plot the density along this axis, so coming from zero now, not from a half, but so there's no excitations along here. This is the chemical potential rising at some finite temperature. And now we've plotted all of the phonon lines together with the replicas. And so this is zero phonon, one phonon, two phonon, three phonon, four phonon lines. And, you know, you notice that something odd happens here, because of course there's in principle an infinite number of replicas going all the way down here at any finite temperature. You know, they're bosonic in some sense. So the chemical potential seems to however to go through them. So what happens, and actually the interesting thing is that there's kind of a conspiracy, is for most of these level crossings, what happens as the chemical potential goes through, it turns out that the photon component of the wave function vanishes at this crossing point. So that means that the excitons, of course, can't be multiply-occupied, they really spins. The photons could be multiply-occupied, but if there's no phonon weight when the level crossing happens, there's no transition. And so on the right-hand side, what you're really looking at is what happens to the photon number as you raise the temperature. Not surprisingly, when the temperature is small, you can only add photons. You have to go on the stokes side. But as the temperature grows, then what happens is that there are photons around in the system that can actually be absorbed. So the line effectively shifts down. You have a thermally populated state, and that thermally populated state, at least at the mean field level, can condense. So that's a proposal. It's not being seen, but one can look for it. I think there's actually some interesting data from Stefan Keener-Cohen on this, but I would show something similar, but my current interpretation of that is that that's probably dynamics, and I think you would agree with that. So now I want to finish by moving to a different problem where you don't have local interactions with phonons, but you actually have interactions between excitons. Of course, the basic model I talk about has on-site interactions with excitons. You don't put two of them in the same place. But you can have systems where the excitons do actually have long-range interactions, and one way of thinking about that is with Rydberg atoms. So when you make an excitation of an atom to a very large n-value, the atom itself gets very big, and it interacts with other excited interactions on a long-range effect. So that produces what people like to think of as a blockade effect, and that's certainly been seen in Rydberg systems, and there's work under foot to build, to put Rydberg atoms and to put them actually into microcavities. So you can also make Rydberg atoms in solids. This is some really lovely work on the classic textbook exciton material, cuprous oxide, which has the famous yellow and green exciton series, and here you can see it, you go from n equals two, and then you look up at the end here, and you can see the next piece of this. This goes all the way around to about n equals 25 or so, where you can see these in this really lovely material. And here's the blockade effect that as you increase the laser intensity, if you look at these large n-levels, what you're discovering is that the absorption cross-section goes down, because the presence of states which are already occupied moves them off resonance. So this is not work done in a cavity, but there's I think work under foot going to try and put cavities around this, so it looks plausible to be able to do this. So what about a model? So the model I'd like to propose again is to take the previous one. So this is same Hamiltonian I've been showing all the way through. The thing that I've done in here is just to write this, I'm sorry, not as a spin model, but to take the spins and represent them as two fermions with a constraint. This is mathematically equivalent. And so if you like A and B are the lower and upper levels, but you can only have one site occupied by as a result of this constraint. So this is SZ, this is S plus, this is S minus. And then what we're going to do is that we're going to add an interaction in the upper state. So the upper A state, if it's occupied, then there's interaction. There's no interaction if the system is not occupied. And because I'm not smart enough to solve the general problem, I'm going to mostly simplify by talking about a lattice model with nearest neighbor interactions and then generalize it to long range. But the kind of things that you would think about here if you just look at this is now it's quite clear that there's a possibility of an instability of the superfluid state because the superfluid state, super radiant state assumes that the excitations are spread out across all of the system. And there's an alternative state which would take the same number of excitations and put them one at a time per blockade radius and produce some kind of crystal. So there has to be some kind of transition about this to try and understand this. A few technical points. If you actually take a long range repulsive interaction one over R to the sixth, that has some unpleasant behavior because you can't get to the final answer from a weak coupling instability. As you know from any kind of theory of fermionic gases, if you have a in Fourier space, if you is always positive, then there is no weak coupling instability because the compressibility is positive at any queue. Same problem you have if you just want to get to a vignette crystal of the electron gas starting from weak coupling, you actually have to start by having a magnetic instability creating spins and then going through that. There's another one which is kind of interesting here and you will see is that of course if you look at the super radiant state of these two things, it has of course amplitude and phase modes. The phase modes are neutral and therefore they don't construct with density. The amplitude mode is also neutral and it doesn't mix with density and all of the action only happens at finite queue. So what do we do? Well, here's the machinery. You construct an action, it has in it excitons and photons. What we would like to do is we would like to integrate out the excitons, leaving the photons there but also introduce collective variables which you can really think of as density weights, I'll call them phi, psi for photons, phi for the density weights. At the mean field level you find the stationarity of the action, then you look at fluctuations about that and we'll see what we get. So let me remind you again of mean field. You've seen this already, no interactions, what happens well depending on the detuning. There's a phase diagram, his temperature between condensed and normal state or if you look at it in a way that you may be more familiar, chemical potential rising up here, then the condensed state is up here, this is increasing coupling constant, this is the mott-lobe with nothing in it, this is the mott-lobe with one particle per site in it and here's the condensed region which sort of enters down in this space. What happens if I add just a uniform repulsive potential at the mean field level, then what actually happens is that you split the mott-lobes. So this is really just a trivial effect because what happens is you have the possibility of using a blue shift produced by interactions to shift the excitons closer into resonance with photons so then you get a regime out here where even for weak coupling you can go down to that. Now let's look at the fluctuation spectrum, let's look at the excitation spectrum. Here's in the normal state again, the upper and lower polariton branches. These come as we know from the interaction between dispersionless excitons and the photon modes. In the coherent state depending on the detuning you get these two branches and they don't actually cross here, there's just a very small gap at this point. So this again is no interactions, what happens with interactions? Again at the mean field level when it turns out in the normal state the excitation spectrum is completely unmodified and the reason for that is that you can't couple q equals 0 superfluid fluctuations to q equals 0 density fluctuations. However if you go into the condensed state and now I'll just, this is actually just in the nearest neighbor model on lattice then what you find is that in the condensed state then because you can actually have the mixing of the amplitude, then you get a phase and amplitude modes and out here at some large q which really corresponds to q equals pi in units of the lattice you can have an instability. So here are the two Mott lobes now and now what you find is looking in this regime here there's a condensed phase out here but there's an instability line so this is associated with a soft mode and inside this instability line it looks like you're developing density fluctuations at finite q. So this is just saying that the condensate is unstable to density fluctuations. As you can diagnose that instability just by changing the bandwidth this is probably not too relevant. Let's now go to the next stage of doing a calculation where I look for a ground state which can have this kind of staggered mean field. So in this one so then I look for the, so now I've again done mean field but I've got a mean field which is staggered so as well as having a Mott lobe at 0 and a Mott lobe at 1 this is a Mott lobe at a half, so this is a half field lattice. And at that level if you look at that what you discover is that this state its energy cross is relative to the condensed photon like state along this green line but this is a first order transition this is level crossing. But of course the next thing you have to do is you have to go and look at the normal state the so-called normal state here this is a Mott state and look at the phonon dispersion curves of that and figure out whether there are any instabilities there. And then you, so then the dispersion requires an extra branch and that extra branch of course in the Mott state there is a gap but then there's a new mode which appears in here which is actually associated with the, eventually with the super fluid order and that softens along this green line. So what have we got? So at this level we have a mean field analysis for the normal state, a mean field analysis for the condensed state and they seem to be stable in here. Along the green line you have a point where the Mott state develops an instability to a photon super fluid order and on this purple line coming in you have a point where the photon coherent state develops an instability towards charge order. Good, thanks, I'm close. So at this level of approximation one has in between this some kind of super solid. Just to check that there are no particular artifacts about looking for the order you can do a generalized variational Monte Carlo to allow the order to develop on different points and that more or less confirms this. The next thing one might do is to start adding long range interactions and as you add longer range interactions you expect to get on a lattice well, you expect to get some kind of devil's staircase and indeed you begin to reproduce that. So you get a self similar branch appearing in here. This is yet another piece of a Mott lobe which is appearing down here but this is one half, this is two thirds and in principle there's a whole bunch of things which are sort of down there. So it seems that at least on a lattice one has a possibility of a phase with both super fluid and charge order. There are now three acoustic modes in it. There's two sound modes and one bogey-luboff. It has two amplitude modes. It has an upper polariton and a charge density wave amplitude mode. The amplitude modes mix and the sound modes don't but it's not a gauge theory. There's no Higgs-ish stuff here and maybe this is a Cold-Atom version of Nibium-Dicelenide which is one of these things which has a mix of this but I don't know whether this phase continues to be present without the lattice and I think there's a sort of big open question about whether that really makes sense. This is very much built in to the lattice model. So the last slide but two because I have some small things to say. Just to thank, over the years all of the people who worked on this and many colleagues that we've collaborated with, particularly the work that I just showed you was done by, unless you have lost them, Jonathan Keeling and Justina Quick who I seem to have lost off this list. That's embarrassing. And Sahin Orreja on the organic materials and most recently on Rydberg Atoms with a graduate student in Chicago, Alex Edelman. So I'm done with this piece of the talk but I have a remark or two because this is a birthday. So, Ariel, I'm afraid you've seen this. Ariel knows the punchline because I used it on him last year. So there's a very famous speech or many famous speeches in Shakespeare but one is about the Seven Ages of Man and I think there are really seven ages and there's the infant show and tell as an undergraduate, the school boy he was sort of dragging to school carrying his satchel and that's really, of course, the graduate school. Then there's the lover, presumably a sister professor at that stage. Can imagine somebody as a lover at this rate. Then, of course, there's the soldier, the full professor charging forward, bombastic all of those. And, of course, what happens after that? What happens after that, of course, is the distinguished professor. So as you said, the distinguished professor is the fifth age and actually what did Shakespeare have to say about this age? The justice in fair round belly with good cape online to somebody who likes their food happens to all of us. Eyes severe, well it is in this one, not so much in that one but certainly a beard of formal cut and full of wise sores and modern instances. So I wish you a very extended fifth age because sixth and seventh aren't really so great. But there's nothing about how long this should last and so indeed I wish you another 60 years of great contributions to science. So thank you very much. Thank you.


back to energy justice in Hawaii. I think we're on episode four now, joined today by three special guests to tell us a little bit about a recent development in a project in West Maui in which a community group secured enforceable community benefits associated with a solar project in Kahana. I'm gonna start with asking my three guests to introduce themselves since some of them I'm just meeting for the first time today and then we'll dive in a little bit to the content of the project. So, Ryan, can you start us out? Can you introduce yourself in a couple sentences? Sure, my name is Ryan Hurley. I'm a local attorney here in Honolulu and I practice mainly in the areas of environmental law with focus probably on utility and energy. Awesome, glad to have you, Bianca. You're up next. Yeah, much like Ryan, I'm Faisaki. I have a private local practice in Honolulu focusing on environmental and Hawaiian cultural rights. Awesome, so glad to have you. And last but not least, Lance, can you give us a little couple sentence intro? Sure, my name's Lance Collins and I'm an attorney based in Maui, private practice and I also have worked for the West Maui Preservation Association or Wampa for short for 16 years and I serve as their spokesperson and at times their representative. Awesome, it's so great to be joined by the three of you today. Lots of legal knowledge in the room, the Zoom room. So let's start with a little backup rewind for folks who maybe are not as familiar with the Kahana Solar Project and Wampa. Can one of you maybe, Lance, give us a little intro about what is Kahana Solar and how did the community, how did Wampa and maybe the broader community first become aware of the project? Sure, so Wampa has been around since 2004. Actually the people who founded it started, were involved in some community activities before 2004 but actually incorporated in 2004. And so Wampa's been involved in the number of sort of hot button community issues related to development. I think the one that folks maybe in Hawaii are most familiar with, is that Wampa is one of the plaintiffs in the injection well lawsuit that ended up going to the Hawaii Supreme Court. I mean, excuse me, the US Supreme Court two years ago and that was decided against the County of Maui last year. So but with respect to the Kahana Solar Project, you know, the Wampa heard about it because people in the community had basically some people had heard that there was a new solar project that was potentially going to be occurring on former pineapple lands of Maui land and pine. And a lot of people in the community were very unhappy with the Kuia Solar Project which is in Lahaina town and it's above the bypass. And basically if you're in Lahaina town and you look up, you see the L and now you also see the Kuia Solar Farm. And I think people were very unhappy with the lack of community input and the way it was cited and basically total disregard for things like runoff and visual stuff. So when word started spreading about this, Wampa started looking into it and then eventually the consultants, the marketing consultants for Interjects contacted Wampa and we set up a meeting with them. So that's how it started. And as people found out that we were interested in trying to help address community concerns, various people from different parts of the community started communicating with Wampa about their concerns because many people are afraid to speak out on a project that may possibly just go through regardless of what you say. So people were careful but there were a lot of concerns and Wampa tried to address as many as could be addressed. Awesome, that's a great background. I just wanted to flag for those who might be hearing about it the first time, Interjects I believe is the name of the solar developer that is proposing this project. And the project is around, I think I read eight megawatts of solar and like 20 megawatt hours or maybe it was 40 megawatt hours. I forgot that number for the battery. I think that's about reverse a little bit. So I think 20 megawatts of solar, a little bit larger than I think. I forget what the exact numbers on the battery are. Okay. That's something like eight megawatt hours. There's something like that. Maybe eight megawatt, I'll have to look later. Yeah, it was 80 megawatt hours. Great, thanks for confirming that. Sorry for not looking at a pre-show. So folks, we're hearing about it kind of drawing a parallel between a project, a past project that had already been installed that didn't match community values and some of the same issues were being raised to Interjects' kind of community outreach consultants around I'm hearing like a visual and maybe also some environmental benefits or environmental impacts. Were there any other community concerns that you guys recall Wampa or the broader community bringing up in those sort of early days? Well, I think one of the main things actually was, well, I mean, overarching all of this, right? It's a large foreign corporation that's going to be profiting. So there was concern about who is this new neighbor that's going to be in our community that's going to be making money in our community and be physically there for at least 20 years. So that was a concern, but it's sorry. As far as like other concerns or as environmental around, especially runoff, I mean, the project is going to be placed on gulches across several gulches. They call them table tops because it's right across them. And those gulches go straight to the ocean. So that's going to have to be carefully monitored. And I think that's one of the things that was scrutinized also fire because this area is fire prone. Specifically this area that's, I think even in the last year or two years ago there was a fire there. There was concern about land rights because a lot of these parcels that were taken up by the previous, well, by the plantations going up through Mollie Lennon Pine which is the landowner for them. Like there was concerns about whether or not the land rights were there. I'm sure there's others missing. Those are some of them in English. And I'm curious at that stage, well, actually I'm not curious cause I listened to the snippets of the PUC hearing that Ryan sent to me. So I have a sneak preview but what was the response that Wampa and the community was receiving from interjects with these concerns being raised at that time? Well, I think the, I'm sorry. I think for interjects, the, their focus was on getting this thing approved by the PUC. And so there wasn't, what Wampa has even in the injection well case what Wampa is used to is basically engaging in some sometimes difficult negotiations but negotiating with the developer or in the instance of the injection well as the county to come up with solutions that everybody can live with as best as possible. And it just seemed that that was not part of interjects strategy at the time. And unfortunately, those things can happen. And so we ended up in having to do a contested case because there was no negotiation. And having to do a contested case, what does that mean from Wampa's perspective? Like what does that look like for a community to have a contested case? I guess maybe I'm talking specifically about like procedure. How does a community contest a case? Is that in front of the PUC? Is there somewhere where you log a complaint with the PUC? Oh, sure. So a contested case is actually, it's a legal term of art that refers to an agency engaging a trial-like process where it hears evidence and it basically weighs the evidence and makes legal determinations based on whatever the legal standards are. And so when Wampa intervened in the docket, it became like a party and the PUC identified what the legal issues were and basically signed posted. If you get to the hearing part of the contested case, this is what you have to prove. And so the process that we had basically went through that. And so the contested case hearing was at the very end of it and you have to prepare and of course Wampa had two attorneys representing them, Bianca and Ryan. So communities don't need to have an attorney to do a contested case but it definitely makes things a lot easier. Great. Yeah, Ryan, go ahead. I was just kind of at the time in about that. I just wanted to add a point, I think that that highlights kind of one of the challenges with getting the community involved is if the developer is not really open and welcoming of the community, then the community has to take a much more drastic step. Wes Maui is very lucky that it has organizations like Wampa to represent it and hire people like us to do this. But there is an expense to it and I'm not sure it's totally necessary if there's other ways we could handle this but I think right now given the current process I believe it was important for us to do that and it does just show one of the challenges is actually doing the contested case is takes a lot of effort, takes a lot of resources and months and months of work. And as we've seen in this case now if we could have hopefully done this prior to a contested case and save resources for everybody else. So that's one of the challenges I think we kind of hope to show in this and encourage the utility and the commission to look at ways to better incorporate the community so that they don't have to result with contested case all the time. That's a really, really good point and I wanna touch on that in a moment also but I would love for our audience to kind of get a sense what was the result of the contested case and the PUC hearing once that was formalized? What was the output of that? Well, it was kind of interesting I haven't seen this before at the end of the contested case and perhaps because of the testimony both of I think of all the parties and especially the consumer advocate which is a government agency for the Department of Commerce and Consumer Affairs. Everybody noted that there are concerns and we've said this even before we got to the contested case that could be remediated through mediation through settling it because nobody was specifically opposed to the entire project it was just there was details about it and so when the commission heard that message they ordered us into mediation. So that was a novel statement. That is interesting. So tell us a little bit about mediation. Well, I don't know if that's that a secret thing that you can't tell us about or are we allowed to know what mediation was like? Negotiation between the parties that's structured by a we've chosen mediator agreed upon by all the parties. The mediator talked to all the parties and kept trying to find compromises and points of weakness and strengths that then just talked everybody through and that's essentially we did it on Zoom as everybody is doing now. And there was a result that the mediator I believe I read that it was a retired judge who helped mediate between the two parties. What were some of the outputs the community benefits that were decided upon and how does Wampa feel about that output? Perhaps someone else wants to talk about how Wampa feels about it. Good? You all feel good? Okay, go ahead. Yeah, well, you know, the PUC proceeding is relatively limited in terms of what the PUC can and can't do. And so I think what Wampa was able to get through the mediated settlement was by far probably the best that it could have gotten in the process. And so, you know, for folks that aren't familiar with what Wampa got, the developer will be funding $55,000 a year of community benefits. And then they will pay at least 80% of their non-supervisory workforce. The equivalent of prevailing wage, which is basically a living wage, they'll commit to hiring locally first from West Maui, then County of Maui, then State of Hawaii, and then only if they can't get their labor needs met locally, then they will look elsewhere. And then also they'll commit to a decommissioning plan, which is very, very important for the community. So, you know, I think those issues, which are things that the PUC can deal with, are things that it should deal with in a more systematic way for those projects that meet the minimum requirements for a community to be okay with having these kinds of projects occur. And so, you know, in the same way that the PUC has decided as a matter of policy, that these independent power purchases will occur through a competitive bidding framework, they can just as easily require that the independent power producers commit to 100% of its workforce being paid the equivalent of the prevailing wage. They can also require a standardization of community benefits and so forth. And so I think Wampa's big hope is with this settlement that this can guide both the Public Utilities Commission and also the utility in standardizing some of these types of benefits and policy preferences so that it doesn't require community groups to have to hire attorneys and engage through a contested case process. I mean, that's Wampa's big hope, I think. That makes a ton of sense. And that feels like some amazing points and suggestions from the community that I think are wise business-wide. So that's amazing. Oh, I also wanted to make a plug as a good think tech Hawaii host. I get to ask most of the questions but if anybody's listening now and wants to ask our guest questions, they can email them to questions at thinktechhawaii.com. And if we get them in time, then I will ask them to our guests. But in the meantime, I get to keep asking the questions. I'm curious about process. Like this was such a burdensome process, many hoops to go through for Wampa to get this end result. And I totally agree that those community benefits that Wampa asked for should be a great starting point for all future projects like prevailing wage. What a wise idea. And I'm wondering for specific projects that might have like a site-specific community benefit or maybe a slight tweak on those things, have you guys, through this experience, like this is probably not on you guys, but do you see a better process forward for folks like Wampa who are gonna be the neighbors of future projects to get this outcome without being an intervener in a docket and going to a PUC hearing and having a mediation? Or do you think this is kind of what we're signed up for for the next couple of decades of energy development? Ryan, I know you have thoughts. Sorry. I don't wanna cut you off, Bianca, but yeah. No, we've talked about this a lot lately. So, and I will say, first of all, I think, and I know that the three of us have discussed this, is we do think this was the best deal we could got for Wampa at the time. That doesn't mean that this is a ceiling. We believe this is the floor going forward. I personally believe that 100% prevailing wages should be required for any project going forward. As Mr. Collins discussed in our hearing, one of the big selling points about large utility scale, industrial scale, solar was the jobs and the good paying jobs for that. So, I think that's absolutely critical. And the fact that we had to fight for something like that, I think it's telling of the times. I do think there's a lot of things that can be done differently and that should be done differently. Just in terms of getting, and I won't go into the details about all the different ways it could be done. I know that Ali, you're working on different community ownership models. I think there's a lot said to that. I think focusing on the distributed ownership, distributed resources first. Personally, I'm a big fan of we really need to start to understand our load and our future loads. As we were 10 million tourists in 2019 and are looking to exceed that in the coming years, I think we need to understand that that's gonna be our driving load and that those tourists equal solar panels on agricultural lands, right? And so that in certain areas, we're gonna have to understand that. I don't have a short-term solution. I do think there's gonna be a need for more intervention at the PUC. I hope it's not for the next couple of decades. I hope that with some good intervention, I think that we're making a lot of headway now in terms of, I'm seeing, and I know you're aware of this, Ali, how the commission and Hawaiian Electric is responding to some of the community benefit, to the community renewable energy projects, I'm trying to get some more input there. I will say too, I think the speed, right? As we're seeing it from everywhere, both the utility and the commission, is this time is of the essence that we need to get this done yesterday. And I think that does pose a challenge for us, for community groups and the community in a whole. And that might not lead to the best reality for us on the ground and how these projects have developed. So I think that always slowing down and taking your time, especially in the early procurement phase, if we can get more community benefits prior to the procurement, I think that not community benefits, but more community input prior to the procurement, that will go a long way. I know Bianca has some stocks in this too, so please Bianca jump in. Oh no, I just wanted you to go more into it. But I do like the language of this is a floor and like Ali said, these are starting points. Because I mean, you think about projects like Napua Makani and like no amount of community benefits was going to like remediate that. And so yeah, I mean, community inputs. And I mean, but like taking a step back, like why is the onus always on the community to like show up on a tent on 10 a.m. on a Wednesday when they have to work, which explain why a project's bad, when it's pretty obvious that it's, well, you know, that very, very often it's just a way of like extracting resources from an area by another corporation as opposed to keeping the benefits there. And you know, that's like you guys have been saying that's where the crux of the issue is. So we're just like finding band-aids at this point. All the way, you know, settlement was good. I'm going to add my other two cents in this. I really feel like at this phase we're at the putting the fires out stage. There's a lot of forward thinking, planning and discussing that we need to have. But the truth is that HECO has 15, 16, 17 extremely large renewable energy projects, 600 megawatts, 3,000 plus acres of utility solar going in the ground in the next two years. And there's been a very, very, very little community input. Add on top of that a layer of COVID-19. A lot of these public meetings have been held via Zoom. I think one of the interesting things we talked about in the Kahana Solar Docket was the differences in how developers are holding community meetings online versus how the commission holds their community meetings online. Are they doing it in a way that's encouraging community members to talk amongst themselves and be open and forthright? Or are they doing it in a way that limits kind of community input? That was one of the things I think we noticed. And this is not solely on Kahana Solar. This is in a lot of the developers I think and the projects going forward. But I kind of consider what we're doing right now is putting a fire out. We're trying to make sure that the community gets some decent benefits. They get the best that we can do for them. But I do think that in terms of, you know as we look to 2045, 100% renewable energy this is gonna be a massive, massive undertaking in terms of a number of projects and the size and the scope and the costs. And I think that the community really needs to lead in this. I think all of that makes a lot of sense. And I also, I loved your comment about like the speed and we need to move quickly. And so we're just, we're moving quickly and you guys, we'll deal with you guys you guys being community later. Like we're just focused on this. You're our second choice. And I think one of my realizations about seeing projects that had that perspective was that they are not gonna go any faster than projects that might slow down earlier. Although I wouldn't say that we've seen a lot of evidence of the slow down earlier yet. But hopefully they are seeing projects like this seeing the output, the community benefits that were allocated through this project. And hopefully they take that and incorporate that early in their project design going forward bumping that up to 100% prevailing wage. Lance, do you have anything else that you wanted to add from the perspective of like iterating this process to be maybe less cumbersome upon community groups like Wampa in the future? Yeah, well, I think one thing that I just wanted to follow up on what Ryan was saying is that, you know, as I understand it, a number of independent power producers are making donations to nonprofit organizations and calling that community benefit or the response to their community outreach. And, you know, the problem with it is that if it's corporate marketing department deciding where to donate money, it's a very cynical way of looking at community benefit. So there really is no reason why the PUC can't say, look, projects that the community support has to have a community benefit component. It's gonna be this percentage of gross revenue or something and that it has to be, you know, who gets that funding has to be determined by an independent third party, whether it's something like the Hawaii Community Foundation or a separate nonprofit is set up. That, you know, maybe that would have to be decided after further study by the PUC. But yeah, I just, there's a concern about, you know, that the fact that it has the effect of benefiting the community doesn't make it a community benefit with respect to the project if it's really just an extension of a developer's marketing. And so that, you know, and so as you see in the Nupua Makani instance, well, that one's a little bit different because I don't think any community benefit that could be given would have ameliorated the problem there. But I think in instances, you know, you can see from the Nupua Makani where that kind of problem can occur because you don't have a community. I mean, even what they're saying is community benefit actually isn't going to the Kahuku community. It's going to another nearby community that basically is pitting neighboring communities against each other. So that's actually like the worst case scenario what could happen. But by degree, there's all sorts of less worst case scenarios that could occur and in some ways undermine the purpose of having community benefit. So I think that that's one issue. The other being, you know, community outreach, the purpose of community outreach is to affect the proposed development. And if community outreach is really just trying to convince the community how great the project is without there being some change the other way, it's not really community outreach. And that's an issue that Intergex has faced in some of their other projects in Maui as well. So. I can add on to that. Oh, sorry. Can I add on to that just a little bit? Yeah. Yeah, we've got 30 seconds. Go for it. Okay, no, I just, well, for this settlement, I think it made sense. Community benefits package made a lot of sense because Lager's was going to go into the grid to Maui, but whereas like, I mean, there's also the consideration of like making sure that you get the lowest rates. But I think in this case, because it wasn't necessarily going to benefit the local community, that's one of the important reasons that you would want to have all of these clauses in there. Yeah, yeah. Really good point. Thank you for adding that. And I'm sorry to rush you guys. This was an awesome conversation. I'm so glad that you guys were able to join. And I hope that maybe we can have you back in another couple of shows to talk about what the future of community benefits packages are in a new process. And maybe we could even talk about community ownership because that's a topic near and dear to my heart. Thank you for joining us, audience, on our fourth episode of Energy Justice in Hawaii. I'm Ali Andrews, your host. If you think of anything that you want to come on the show and talk about energy justice lies, please let me know. We're always looking for guests. Thank you beyond Ryan and Lance, and we'll see you next time.


While the Russian aggression in Ukraine rages on, refugees, most of whom are women and children, have found comfort in Europe. Local and regional leaders have stepped in once again, this time to provide a safe place for Ukrainian children through camps organised across European cities and regions, an act of solidarity that offers a sense of normality to some of the millions of Ukrainian children whose lives have been shaken by the war. The Bavarian Youth Exchange Foundation, in cooperation with the German Youth Hostel Association, Landersverband Bayern and the Association of Russian-Speaking Youth in Germany, Yunost, Landersverband Bayern launched a programme called Language Encounters, which brought together Bavarian and Ukrainian youths. In total, eight summer camps were organised, with programmes of each camp including sport and creative activities. This enabled the young people to get to know a new language through a fun method. Melanie Hummel, Bavarian State Minister for European Affairs and International Affairs, and Chairwoman of the Board of Trustees of the Bavarian Youth Exchange Foundation, spoke to us about the aim of the camp. In the whole of Bavaria, summer camps for young people from Ukraine take place. I think it is so important that young people from Bavaria and Ukraine can meet, that they can exchange with each other, that they can learn the language. We also spoke to Thomas Rudner, Managing Director of the Bavarian Youth Exchange Foundation, about the programme of the camps. We didn't know that there were any major educational events but that there was a lot of value, that young people had time to relax, because their situations and what they did are often quite traumatic experiences. We hope that we can contribute so that young people can get better. The German Youth Hostel Association Landersverband Bayern provided accommodation facilities in Bavaria. We spoke to Winfried Nessensohn, Managing Director of the Deutsche Jugendherbeer's work Landersverband Bayern. We are happy to offer the young people here in Bavaria the opportunity to offer a refugee refugee a language meeting here in Ukraine, to offer a refugee a language meeting here in Bavaria. We also have the youth support and psychosocial support in our work. We are very happy to be able to do this here. Victoria Chetverikova, Project Coordinator of the Summer Camps from the Association of Russian-Speaking Youth in Germany shared with us the valuable work of counsellors and language animators throughout the camps. This is a great exchange for the children, for the young people who had very difficult last months. We also have a great programme. We do a lot of creative activities, sports activities and have a lot of fun together. The team, the team that we have, I find it very engaging. We do everything we can to give this opportunity to the young people, to be happy, to find friends and to dream again, to enjoy life again. EPP-COR members have already organised summer camps in Nîmes, France, Maramuresh, Romania and in the regions of Vielkopolska and Pomerania in Poland. The costs of the camps are footed by the EU municipalities and regions participating, together with public, private and civil society partners. Meanwhile, cities and regions remain committed to contribute to the reconstruction of Ukraine to give these children a better future.


Good afternoon. Thanks for coming to this session and My name is Christian. I work at Los Alamos National Laboratory and this is Nicholas Bock working at SUSE Linux as a software developer and today we are gonna tell you a little bit about our experience on running a scientific application on an OpenStack Club and I'm gonna tell you a little bit about the code that we developed and which is our case study. So working for some reason. Well, this one is but not the screen up there. So as a matter of introduction What is scientific computing? What do we do when we do scientific computing? Essentially, what we do is to develop is to is to write a code, a very specific code to solve a particular problem and what we usually do is to try to optimize the code as much as possible to be able to essentially to profit from the computational resources that we have available. And that is why this is called high-performance computation and essentially the computational resources that we have available is basically a cluster of computer, right? And so scientific computing spans through many different scientific domains. It can vary as widely as going from social science, let's say to material science, going through a lot of different domains and depending on the problem we want to solve and depending on the on the scientific domain, we will have different computational requirements. One of the scientific disciplines that requires a lot of computational resources is computational chemistry and in particular computational quantum chemistry. And the reason why is because in computational quantum chemistry, we need to solve what is called the electronic structure of a molecular system, which involves operation linear algebra operations with extremely large matrices, which sizes could go up to tens of gigabytes. So that means that we will be often memory-bounded by the problem. But also there are in-medic operations that we have to do scales really poorly as order n cubed, where n is the dimension of the matrix and it's often related to the system size and in quantum chemistry, the system size is essentially the number of atoms that we have in the system. So again, it requires a lot of computational effort. It scales as order n cubed and we are very often memory-bounded. So what is the operation that the main operation that we do in quantum chemistry? Essentially what we always can construct from the system is the so-called Hamiltonian matrix and from the Hamiltonian matrix we can construct something, another object that is the density matrix and the density matrix give us all the electronic properties of the system. With the density matrix, I can ask the system, I can compute any property of the system. So this matrix to matrix transformation is the one, is our bottleneck essentially, that scales as order n cubed. So what is the technique? What we are essentially doing? We do something that is called molecular-dynamic simulation and what is molecular-dynamic simulation? Molecular-dynamic simulation is essentially a technique that allows us to follow the position of each and every atom in the system. So basically what we are doing is to track the position of each and every atom to be able to compute useful properties. So how this is done? This is done essentially by integrating the Newton equations of motion for the atoms in the system. The atoms are interacting among each other through interatomic forces. If we have the forces, we can essentially update the position of the atoms by integrating the Newton equations of motion. So what do we have? What is the result of doing this? It is essentially that we get a collection of vectors for each and every time step. So we have the time step that is discretized, the time that is discretized in different steps. So we have the vector that are dedicating the position of the atoms for each and every of these time steps. And that is something that we call the trajectory. And again here, we are generating a lot of data. We are generating a lot of I.O. and writing a lot of things into the disk. So this is another, also another story. So once we have the trajectory of the system, we can compute useful properties such as thermodynamic properties, or also we can see a particular phenomena that is happening at the level of the atoms. Here I'm showing a polymer that is moving due to a heat that is trapped on the system. And here the red and white sticks are warred. They are solvating this polymer. And we can see that we can see that we can essentially track what is happening with this polymer. And this is coming out of the molecular dynamic simulation. So this is, this technique has a tremendous predictive power and it's very important for material science, material discovery. So but what happens if I want to see reactions happening in the system? What happens if I want to see what happens if I mix A and B in the reaction pod and let it react? So remember that reactions, chemical reactions are nothing but formations of new, of new bonds or breaking of those bonds. And essentially, if I want to see reactions happening in the system, I have to go beyond a classical molecular dynamics. And I have to do what we call quantum molecular dynamics. So now we will be solving the electronic structure of the system at each and every time step of the simulation. And again, we will be doing this Hamiltonian-to-density matrix transformation. So this is a technique that again requires a lot of computational effort. But the benefit of doing this is that if I can predict reactions happening in a molecular system, this will open up a lot of possibilities in material discovery. So it's very important to be able to predict reactions happening in computational chemistry. So these diagonalizations scales as order and cube and that means that after certain system size, the calculation is going to be unfeasible. So how do we solve the problem? How do we go to a very large system? And this is an idea that another star from Los Alamos National Laboratory had and his name is Anders Nicolason, and he came up with this idea of using the underlying connectivity of the molecular system to be able to partition the molecular system into small pieces so that now we can, let's say, distribute it into ensemble of computing nodes, for example. So and how do we get the connectivity of a molecular system? So we can get it from the density matrix, for example, or from the Hamiltonian matrix. Essentially, the density matrix is going to tell us how the atoms are connected in the in the molecular system. And if we have this density matrix, we can think about the adjacency matrix, and the adjacency matrix is a mathematical abstraction, a mathematical representation of a graph, right? So if we have this adjacency matrix, we have the graph, and if we have the graph graph theorists know how to partition this graph into small, into different communities. So now if I can partition my system into different communities, now I can construct this Hamiltonian matrix for each and every community and what I can do is essentially to to send all these communities, all these Hamiltonian matrices, to distribute them to across them, all the computing nodes. We can run our we can compute the density matrix for each and every node, and then what we can do is to gather everything together to reconstruct the electronic structure of the full system. So that's how we do in order to scale to go to a larger system. So this is a very simple idea and which requires almost no communication between the nodes. So when the nodes are computing their own electronic structure, they don't need to interchange any data. The only thing that we have is a null gather at the end to reconstruct the full density matrix of the system. So it's a very interesting technique and this is where the idea of being able to allocate computational resources come out because if we have an extremely large system, we would like to have as many nodes as we as we can to be able to distribute this calculation. So again just to to summarize that the way we do this is to partition the system into very small pieces, construct the Hamiltonian for the for these pieces, send the Hamiltonian, distribute the Hamiltonian across all the nodes. These nodes will compute these portions of the density matrix, and then we will reconstruct everything because we know the connectivity of this full system. So we will reconstruct the electronic structure of the full system. Here I'm showing a picture of how this this looks like. So at the left here we have here we have a molecule which is a dendrimmer. It has a sort of fractal structure and in computational chemistry, in quantum chemistry, atoms are usually represented by orbitals and what you you see here as red dots, those are orbitals that are connected with these black edges. And what happens is that after partitioning we get these communities, those are the communities that we use to construct the Hamiltonian, the pieces of the Hamiltonian matrix and send them, distribute them across the nodes. So here we see a problem that we can run into and is the fact that the communities are usually unbalanced, meaning that we can have large communities and small communities. So here we have to be careful and try to send to distribute the communities across the nodes such that all the nodes are more or less the same workload because we don't want nodes sitting there waiting for calculations to arrive. So so here is an example of system that we usually use and it's it's called a water box. It's just water molecules in a box and here you can see one of the partition and this is more than a hundred thousand atoms and the problem that we have here is that this system is very large and it takes about one minute to do an amolecular dynamic time step to basically update the coordinates of the system. So this is a lot. I mean we want to we don't want to wait one minute to update the coordinates of the system. So we need to be able to use more for this particular system. We need to be able to use more nodes and so that we can basically lower the time to something less than a second because what we want to have is an MD simulation that could be practical that where I could extract useful information out of the MD simulation. We need simulations long enough to be able to see phenomena that are happening in the system, right? So we need to think. We need to go as large as we can for system sizes and we need to go as we need to increase the time scales that we are able to simulate. Why do we need to go to very large systems? Essentially because for example in biophysics, we have systems that are composed by millions of atoms and for example here I'm showing a full virus. This is the mosaic of virus. This has million atoms and it's very useful to be able to do the quantum molecular dynamic simulation of these systems because there are a lot of interesting questions that that we will be able to answer. For example, the possible interactions of drugs with the proteins of the virus or chemical reactions that could occur at the level of the RNA. So these chemical reactions could lead to mutations. So there are a lot of questions that could be answered if we are able to push this technique towards many million atoms and very long time scales. So now I'm going to let you with Nicolas Bogd that is going to tell us a little bit more about details of this, the porting this code to the ECCLA. Hi, so again as Christian said, my name is Nicolas Bogd. I work for SUSE and before this I worked at Los Alamos National Laboratory as a computational physicist with Christian in the same group. Is it all the wrong direction? There we go. Okay, so let me first give you an overview of what HPC Atlanta typically looks like. This is a slide showing you some specs on Trinity. Trinity is the largest HPC cluster at LAML right now. It's a rank number 10 in the top 500 as of November last year. It is capable of more than 40 petaflops of performance, has more than two petabytes of memory and more than 19,000 compute nodes. So that's sort of, I mean, that's a very large cluster, but that's in the supercomputing centers around the world. That's a typical sort of setup. From a user's perspective, you run a resource manager such as PBS. That's kind of a classic example on a cluster like this that takes care of provisioning resources and also ensuring that the cluster itself is operating at full capacity all the time. So the queue, there's a queue that you submit your job to. The queue is not processed in order necessarily. PBS is able to rearrange things there to fill gaps in allocations, for instance. The provisioning is done bare metal in a sense that once you have an allocation, you're able to log in to these nodes. They're not re-imaged or anything. So you just get shell access and then log in and run your code there. That means, of course, also that the OS and the software libraries are not something you have control over. You have to work within the software environment that these nodes come with. And then the last thing I want to mention, a lot of applications. Obviously, if you have more than one node, you have a distributed memory problem. So a lot of applications use MPI, the message passing interface library to work with that. This is essentially a software library that allows you to send data messages in between nodes. So that library, though, adds a requirement to our setup that I'll mention in a little bit. Just keep in mind that we have that MPI here that we have to do something about. So then you write a shell script if you want to use this. That's a typical example here. In the first few lines, you instruct this PBS, this resource manager, what kind of resources you're expecting to use. Those are upper bounds. And PBS then uses that to schedule your job. So the tighter these bounds are, the more likely your job is to get started earlier. If there's a resource gap somewhere, I mean like an allocation gap somewhere, then PBS may put your job in there and maybe put a bigger job further back in the queue. So then once you have the allocation, the script is run on the nodes you get. So you then proceed with loading some software modules to set up the library paths and all that, and then run your application in line eight. You submit such a script with this command line down at the bottom. And so overall, this is a very simple and straightforward process. Now coming into this, not just from a pure user's perspective, as an application developer, I mean Christian and I, we wrote the software that we're running there. There are some drawbacks to this approach that add, let's say, some friction to your workflow. The first thing is that typically the OS that's running on the cluster is not the same OS that is running on your laptop. I mean maybe your laptop runs a different OS, entirely different distribution, maybe different OS. You have probably different libraries and a different tool chain. So your development though mostly happens on your laptop, that's more convenient. So if you then go to the cluster and compile your code, sometimes that doesn't work. So this all adds some obstacles to the workflow that you have. The second thing is that when the cluster administrators decide to update some libraries, that sometimes leads to breakage on your end, maybe the API changes of some library. I mean HDF5 is a good example. At some point, the API completely changed that forces you to rewrite parts of your code.  So the quality of the software that you typically write is probably not quite up to the standard that you're used to seeing in a project like OpenStack. So there's probably less abstractions in there and refactorizations and all that. And then the last thing I want to mention is that although these clusters are very large, they also have large numbers of users. And so typically the queue is very full. That either means that there's a long wait time before your job starts or the administrator has to limit the wall time that you have available. But if that happens, that's typically what happens at LANL.  And checkpointing is sometimes not so easy to add to a code. Like in our case we use dense linear algebra. We just call a library to do that for us. If that library doesn't checkpoint itself, it's almost impossible to add checkpointing yourself unless you change that library somehow. All right, so when I started working for Zuza and with OpenStack I asked myself the question, isn't there a better way to doing this that resolves these drawbacks at the same time and at the same time gives a researcher the same sort of performance. So I mentioned that to Christian, we discussed that and we decided to just try that out, to actually go ahead and try to run our code on an OpenStack cloud. All right, so I'm going to tell you how we proceeded doing this. So the first thing we wanted is a custom-made image so that we can control the software environment to make that match what we have on our laptop. So we created a server based on OpenZuzaLeap. And then we put that... Oh, what did I do now? Okay, so then we put that on a public cloud and now the first time we did this, we realized that within an hour we got hacked and because we picked a weak root password for convenience, we didn't want to bother with SSH keys and everything.  So obviously afterwards we started securing our service using standard procedures immediately. Then we went ahead with creating a normal user, SSH keys for MPI. MPI has to talk internodes using SSH, so you need that. Then we installed all the software libraries that we needed, built our code and then created an image and uploaded that to Glance. So that was actually fairly straightforward after we fixed our security problem. Okay, so then to run a job, all that's left to do now is to create a cluster with P servers using that image. Now I want to circle back to this MPI problem I mentioned earlier. MPI in order to run has to have a list of IP addresses of all the servers in the cluster. Okay, so we don't have that list. We collected a list of IP addresses that we got from Nova and uploaded that list into a node, logged into that node and ran the application. So at this point, everything seems to be solved. It's kind of like the same thing we do on an HPC cluster. It's just much better because we have a custom-made image. Well, after using this approach a few times, we realized that there's some rough edges there that maybe are not so convenient. So for one, there's this question of resources. Now, in an OpenStack, a public cloud with OpenStack, you pay by time and by flavor. So what you want to make sure of is that your flavor, the flavor you pick is large enough, obviously, otherwise your calculation wouldn't run, but that it's not too big. You don't want to overpay for resources. And an HPC system such as Trinity, on the other hand, you typically have so much memory per node that you don't have to be overly careful with how much memory, for instance, you're using. So that forced us, that realization forced us to more carefully profile our application to understand better how much memory we're going to need and then pick an appropriate flavor. There was sort of quite a difference to what we're used to. The second thing we realize is that manually creating a cluster with P nodes doesn't scale. You can do this with 10 nodes. At 100 nodes, this becomes kind of tedious, and at 1,000, it's kind of ridiculous. You don't want to create 1,000 servers by hand. And the same, of course, is true for creating this list. You don't want to do that by hand either. Okay, so we looked at how can we improve that process, this workflow. And the first thing we did is write a heat template that creates this cluster. The input is, you know, how many, this P, how many nodes we wanted, how many servers. Unfortunately, we weren't able to get heat to also generate this list of IP addresses. So we then ended up writing a second script that we run after the cluster is provisioned that asks the OpenStack API directly for all the addresses that we need, compiles a list, and uploads that list into a node, the master node. So that step then, the deployment step, the deployment step is fully automated that way. And the second thing we wanted to address was the creation of this custom image. Obviously, that's somewhat error prone if you do this by hand. And let's say you want to update the environment in that image for some reason. You don't want to do this over and over again. So we wrote a script that automates this image creation, but obviously there's other, you can do this through many other ways, SUSE Studio, Kiwi, Disk Image Builder, or whatever else you like to use to automate that. Okay, so at this point, this workflow is just as convenient as using PBS on an HPC system. Okay, now let's go over to some performance results. So we ran our code with a test system on, we wanted to run it on a public OpenStack cloud, so we picked Rackspace because they're listed on the Marketplace OpenStack web page, and also because they're offering ironic resources because we wanted to make, I mean we already assumed that we would have to have some bare metal deployments to get the same performance as on an HPC system. And for comparison, we ran this code also on Darwin. Darwin is an experimental HPC cluster at Lannell. That was going to be our, or is our gold standard HPC performance, so to speak. That's what we expect, or that's what we're used to seeing, and we would like to see in an OpenStack deployment as well. We looked at three flavors that Rackspace offers.  This on metal flavor at the bottom is an ironic bare metal one. Because our job is mostly compute bound, we paid attention to the CPUs that come with these flavors. I mean you'll notice that they're different, of course, but their performance is pretty comparable. So based on these benchmarks there, we wouldn't expect any significant differences in performance simply because the CPU with these flavors is different. So that's good. That means that we can directly compare performance results between these flavors. I don't know what I'm doing here. I'm clicking. Oh, okay. Anyways, okay. So now we ran this code for a few MD steps and measured its performance by the number of MD steps per minute that it gets us. So this is kind of like a speed measure of the code. So the first thing I want to point out is the results on Darwin. That's our HPC sort of gold standard. That's the line in black. And on top of it, the line in blue. That's the ironic bare metal run on Rackspace. So you see there that the performance and scaling is basically identical. So that's great. That means that if we do that, we don't lose any performance moving from HPC to OpenStack. On the other end of the spectrum, let's focus on the line in red. That's a general purpose flavor using Nova. Obviously it has a lot less performance. So we weren't too shocked or surprised about this. I mean, that's a general purpose flavor. It's not optimized for workloads like this. So we wouldn't expect the performance of that or with that flavor to be equivalent to something like on metal or an HPC system. What surprised us a little bit was the performance of the compute flavor. That's the line in green. That's a flavor that's optimized for CPU intense workloads. But obviously as you see here, the performance is not much better than the general purpose flavor. So let's look actually in fact, let me mention this first. We went to the one node case to remove all network communication issues that potentially influence this result there. And even there, we see about a factor of two performance difference between the on metal and the compute flavor. So we wanted to understand better why this gap is so big. So we took a closer look at the compute flavor that we were using. So this is now taken directly from Rackspace's documentation. So this is a flavor optimized for CPU intense workloads. And the optimization is that it uses reserved virtual CPUs, meaning that on the host, there's no more vCPUs than physical threads. But we found out that the host uses hyperthreading. And I don't know if you're familiar with how this works, but in hyperthreading, you have two threads per core. And in Linux, often they show up as two logical CPUs. But the way this is done is that basically only the thread state is duplicated in hardware and everything else is shared between these two threads. From a computational point of view, especially for our workload where we use the floating point units most of the time, which are not duplicated, we don't get any performance gain by using two threads on one core. So really what this means for us is that we're mapping eight vCPUs onto four cores. So we really only have half the number of CPUs that we're expecting. And actually, in fact, what comes on top of this is that there are some studies that show that hyperthreading for scientific applications because of their particular workload can sometimes be bad for performance.  Okay, so that was kind of interesting. Let's go... Okay, Jesus, I keep clicking on the wrong thing, sorry. So let's go over to our performance results again. Now, this is the same graph I showed you earlier, just slightly differently formatted. What you typically do, I mean, you're not really that excited about how many MD steps per minute you get. What you really want to do is you want to run a simulation for a certain amount of simulation steps to capture an event in some system, the system you're looking at. So let's say we run it for 2,000 MD steps. I mean, typically you run for a lot longer, but that's just an example, which corresponds to one picosecond simulation time, and this graph shows you the wall time, how long you have to wait for the result to come in. And so, as we've seen on the previous slide, the on-metal flavor is by far the fastest than the compute and then the general-purpose flavor. Okay, so being on a public cloud, we pay for resources, so we can also tell you how much these calculations cost. So this is a graph showing the cost for a one picosecond simulation depending on the flavor. So the nice thing here is the big takeaway, I think, is that the on-metal flavor, which is the fastest, is also the cheapest if you measure it like that. So if you're interested in running a one picosecond simulation, you should really go with the on-metal flavor. Interestingly, the positions of the compute and general-purpose flavors have switched now. I mean, the compute flavor is faster, but also more expensive. And I guess it's more and more expensive than it's faster, so the position of these two lines flipped. Now to put this number, these cost numbers into perspective a little bit, we wanted to understand better what the alternative, actually running an HPC system on-premise looks like. We didn't get, or we couldn't get numbers for Trinity or Darwin, so we can't tell you how much these calculations at LANL cost. But we went to, Amazon has this TCO calculator that you can use to estimate TCO for on-premise. So we used that, assuming an eight-node cluster with an eight-core CPU and 32 gigabytes of memory, which corresponds to the on-metal flavor. It's the same specs, essentially. And Amazon thinks that this costs around $5,000 per node per year. I mean, there's some assumptions in there about cost. This is not just the hardware. This is other things like power and networks and blah, blah, blah. On the other hand, on rack space, this is how much rack space would charge us for the same kind of compute time. It's between $2,500 and $6,500. So let me make some... So the first thing we see is that these costs are actually pretty similar. So it's not actually more expensive to move through the cloud. That's good, right? And the second thing I want to mention is that these costs, they depend a little... They depend on your situation individually a little bit. Let's say you're at a university and you have a shared server room, then your on-premise costs may be much less than this because maybe you can freeload on the power and cooling and the network or whatever in that room. So all you got to do is buy the hardware, right? So this number is probably lower for you in that case. And for rack space, obviously, if you don't run these simulations nonstop for 247.365, rack space would charge you less, right? It charges you only for time you actually use the cluster. So basically you have to individually decide which option is the more cost-effective one. But now in conclusion, I think we've convinced ourselves that the performance is equivalent between an HPC system and OpenStack using an ironic bare metal flavor. And it's also a cost-competitive option to go to an OpenStack cloud, the public one. So given those two factors, we think that OpenStack can be a very useful alternative and additional resource for researchers, especially for smaller or medium-sized problems. Because it has... I mean, there's a lot of public cloud providers out there, so you don't have to wait really to get these resources unlike on an HPC system. You sometimes have to wait. I mean, you just rent these things and they show up, right? And from a usability point of view, it's actually more convenient to use this because you can define your software environment yourself. You run on a custom-made image. That doesn't change unless you want it to, right? You don't have to update anything in that image if you want to. And you can make the image look like the software environment you have on your laptop to make it identical to your development environment. The only thing we found lacking was usability. I mean, initially, when we started, it was definitely not as easy to use as something like PBS on an HPC system, but adding a few scripts made the usability essentially the same. So if you gave a researcher scripts like that, it would be just as easy to use for them as PBS's. Okay, so before I open the floor for questions, let me just thank the awesome SUSE team. And in particular, Johannes Krasler, Robert Warwick, and Bernard Wiedemann, and Roman Arkea for their support. And our collaborators at Los Alamos, Susan Miesinski, Mark Hockwell, Mike Wall, Enrique Martinez, Tim Gurman, Risto Giedjeff, and Anders Niklasov. Thank you. Did you try... When you said that we're running on the hyperthreaded chips, did you try cutting the number of threads in half to see if the performance? No, unfortunately, we realized that sort of very late. I mean, I added that slide on the flight here. Yeah. I mean, but that definitely would be an interesting thing to look at. On the Jetstream cluster, we ran the HPCC benchmark suite. And for Limpac, which is a CPU bound, we ran it 97% of bare metal. Now, the other suite, we were somewhere down around 65 to, I mean, between 55 and 75% performance, because there you're getting into your memory access or networking layers, but that's something, yeah. Right, right, right, right. Yeah, this is, as Christian mentioned already, because of this graph partitioning, the communication component is really small here. So this problem is really mostly compute bound. So, yes, I mean, obviously for different applications where communication is more relevant, you may see different scaling results. Yeah, we didn't try to overload our cores. So even though our system is hyperthreaded, we never ran more than the number of cores we actually had. Right, right, right, right. So we had 24 real cores per chip, and we ran, even though it was hyperthreaded, we ran 24 threads. And like I said, we saw 97%. So we really didn't take the hit with the modern hyperthreading like you used to see it. Yeah, right. Yeah, I think it's way better nowadays. Yeah.  Yeah, yeah, that's right. So one feature that you didn't really call out here, but is actually really useful in OpenStack is that you can, if you do virtualization, you can live migrate the instances, and you can actually persist across maintenance periods. You can persist across interoptable sessions and everything. So, you know, there is some benefit to doing the virtualization over the bare metal. You might consider that as well. No, I think that's a very good point, actually. And I think that one thing we wanted to try, which we didn't really get a chance to, is if you have control over the hardware, you could, I mean, you basically turn off, you know, hyperthreading and actually try that directly. I mean, we suspect that what you're saying is exactly going to happen. I mean, everything I found is that hypervisors like KVM or Xen have an impact of a few percent at most on the CPU performance. So, yes, I totally agree. I think virtualization would add a couple of features that would be great, actually. You talked about the additional work that was required above PBS for this one application. And I'm sort of curious how much of that would be, have to be redone for the next application and the next application after that, this is a one-time cost. Because we didn't do any, I mean, so this heat template is completely independent of what this image looks like, for example. All it does is it spawns P servers. And, of course, that script that collects the IP addresses is the same thing. I mean, it talks to OpenStack API and just gets addresses and populates a file. The only thing that's somewhat specific is the image creation itself. So obviously, if you have a different application in there, you'd have to modify that a little bit. But if you use Kivi or whatever, you know, some automation method like that, that's not too hard to do. Is that something that you think the average researcher can handle, or is that something somebody like you has to do? Yeah, well, I mean, the average researcher I'm, I know, probably would be a little overtaxed with that kind of thing.  But to write a Kivi script, I think it's a bit over the head of your typical computational scientist. Are there any plans to take a look at computationally, not computationally, communications bound or communications balanced workloads in this environment? Yes, that's, yeah, we would like to look at that as well. I mean, like this code is maybe not optimal for this because it has, I mean, built into it very little communication. But we're thinking of using other codes like this project is also part of another MD code called LAMS out of Sandhya National Laboratory. Yeah, and so they have a different communication pattern and one thing we would like to try is run that code on a cloud such as this one and see what the communication impact is. Actually, another thing that would be interesting would be containerization itself. That's, you know, as another way of creating a cluster. You know, through Kubernetes or whatever. We haven't looked at that at all but that would certainly be another option. You mentioned that you had a problem with the MPI interface and you had to build a host file on the fly. Yeah, yeah, yeah. So does this mean that we need some sort of a DNS system for MPI? It probably would help. I mean, if you could automate this. So the way this works with PBS is that PBS takes care of this behind the scenes. It allocates your resources and it generates a variable that has this list and then MPI reads that, right? It gets put into that environment. So if neutron or nova or some, you know, some component like that would generate that, that would greatly help, actually. Any more questions? If not, I guess. This was small. We had 1,032 atoms. But we know this was just for expediency. We didn't want to wait forever. But typically runs are much bigger. I mean, what Christian highlighted, if you want to go to a more realistic system, you're talking about tens of thousands, maybe 100,000 atoms. Like the largest simulations to date, they're more than a million, you know. So it's, I mean, biological systems are very large on that scale. And to get a realistic simulation, you need to go to large numbers of atoms. Yeah, the question was what the network fabric is. And actually, I don't know. I think the on-metal one has a 10 gigabit per second fabric. I don't know if it's, I don't think it's infinite band, but maybe it's just a bonded gigabit or something. These other flavors, they have bandwidth caps. They're much lower. But the fabric they're running on, I don't know. It's probably the same, I'm guessing, to the actual underlying transport mechanism. Yeah, yes. So the question is whether the open MPI library is abstracted above the actual transport layer. Right, basically. You can run on an infinite band or on a normal network interface card. I mean, there's many different ways you can operate with this thing. So there's compile time options, basically. That's why it's nice. I mean, it's not exactly a very high-level abstraction, but it's good enough that it works with any kind of supercomputer, basically, or a cluster at home. Right, if there's no more questions, then I guess I'll close. Thank you.


how to calculate percent in under one minute. The first part of the calculation is to understand the difference between the two numbers. To do that, I'm going to do an equal sign, open brackets, and value number two minus value number one, close brackets and hit enter. There's a change of five between 15 and 10. What we want to now do is work out what five is as a percentage of value number one. Go back into your calculation, add the divide symbol, select value number one, hit enter. We have the value of 0.5. If I copy this down, convert this to a percentage, we can see that for 10 to turn into 15, it has to increase by 50%. In our second example, we just want to compare the two values. In order to do that, all we need to do is go equal sign, select value number two, divide by value number one, and hit enter. Copy formula down, convert to a percentage. 15 is 150% of 10. Alternatively, value of 10 is 67% of 15. If you enjoyed that video, please don't forget to like, subscribe, and follow for more videos.


is going to be stunning. I look at the fireplace which I'm now going to pull out and put in a new fireplace. Now what do I see? I know that this used to be a wood burning fireplace. Look at the beautiful brick that they closed up. It tells me underneath, this is a chased one the way this goes, this is the old bent style for the wood burning fireplace that allowed to take that heat back from that fireplace and force it out. So I'm going to rip this apart and see what I find.