The study presents a global data set of seasonal and diurnal anomalies in urban surface temperatures relative to rural surroundings, and uses satellite observable parameters in a simple model informed by the surface energy balance to understand the dominant drivers of present and future urban heating. If city growth remains unchanged, it is projected that urban populations will live in areas with daytime summer temperatures that are 3.21 degrees Celsius warmer than surrounding rural areas by 2100, resulting in a significant increase in the number of people living under extreme surface temperatures. However, the study also finds a significant potential for mitigation through optimizing vegetation and surface albedo in 82% of all urban areas, which could reduce urban daytime summer surface temperatures by an average of minus 0.81 degrees Celsius. This article was authored by Suzanne A. Benz, Stephen J. Davis and Jennifer A. Burney. We are article.tv, links in the description below.


Welcome to Think Tech on Spectrum OC16, Hawaii's weekly newscast on things that matter to tech and to Hawaii. I'm Michelle Daigle. And I'm RB Kelly. In our show this time, we'll take a look at the beginning of a brand-new medical and business phenomenon in Hawaii, the sale of medical marijuana. We'll visit the Steep Hill Testing Lab, recently opened at an undisclosed location on King Street to see how they are testing for antigens in medical marijuana. There are many applications of cannabis as a medicine to provide palliative relief from glaucoma, cancer, AIDS, arthritis, rheumatism, and inflammatory disorders. Its use is on the rise in many parts of the world and in a growing number of states around the country. That includes Hawaii, which is now rolling out the Medical Marijuana Initiative adopted two years ago by the state legislature. For this show, we visited the Steep Hill Medical Marijuana Testing Lab, recently licensed by the Department of Health. It's on King Street. For security reasons, if you don't know where it is, you won't be able to find it. Because of the risk of robbery or burglary of the marijuana or the cash for it, the lab is and must be a very secure place. We talked to the owners, Wendy Gibson and Marilyn German Carter. Wendy is a medical marijuana educator and Marilyn is a registered nurse. They showed us how their lab works and how they do their testing there. In fact, the lab had already started doing its testing. The supply chain is from the grower to the lab, from the lab to the dispensaries, and from the dispensaries to the public. Each part of the chain is designed to assure that the product is fit for human consumption. They showed us what they do with the marijuana after it comes to them from the grower. They weigh and test it for contamination, pesticides, heavy metals, mold, and other antigens. The lab has millions of dollars of testing equipment. To be sure that the medical marijuana you buy will be pure. Each testing station had different technologies, testing for different antigens, and each was staffed by a technician, skilled in testing for those antigens. We walked around the lab, talked to people, and looked at the testing equipment. Contaminants and molds are some of the most dangerous things that you can have in, yeah. What happens with the mold? Well, a little bit is probably going to be found in all plant materials, but it can cause respiratory problems, it can cause fungal infections in the lungs, and that is, if you have an immune-compromised patient, you might kill them. That's one of the contaminants that they're testing for, absolutely. Yeah, we need to know. Where's the plant grown? For the medicine, the medical cannabis, everything that will be available in the dispensaries has been grown by one of the licensees, and it has to be grown indoors. Well, each dispensary is allowed to have two grow sites with up to 3,000 plants. They can have two manufacturing sites, and they can have two dispensary, the retail outlet stores. What has been legal since the program's inception in 2000 is patients are allowed to grow their own medicine, and now they're allowed to grow their own up to 10 plants. And once the dispensaries open, then they will have access to the dispensaries. They'll be able to buy the bud, the flowers. They'll be able to buy concentrates, lozenges, and I think there's a couple of other things as well, but they'll be able to buy up to four ounces of the usable product, the flower, every 15 days. There are many forms that you can use it in. Edibles will not be one of those. That is not something that is allowed in the law, but patients will be able to buy concentrated oils and tinctures and be able to take it orally that way. They'll be able to vaporize or smoke the flower or the bud, but none of the smoking materials like vaporizers or pipes will be available in the dispensaries. The two main classes of chemicals that are in cannabis are cannabinoids and terpenes, and they'll be testing for both of those. You're probably familiar with the cannabinoids, some of them THC, CBD, but there are others as well that are therapeutic. And then the terpenes also have some therapeutic value and also are responsible for some of the side effects. My education is in analytical chemistry, and so I was approached to help set up this laboratory and get it functioning and certified and accredited and everything goes along with it. How long was all that process to get it set up and accredited? How long did that take? It was a two-year process, you know, from beginning to end. We've been in this location now for a little over a year. We receive samples from the dispensary, okay, and they usually we have to go do the sampling to make sure we have a statistical sample, and then we bring the samples back to the laboratory here, and there's a tracking program that the dispensaries use to track how much they gave us, and we get here and we put in that information saying that we receive that amount of the cannabis here at the laboratory, and then we start to do the test. This is where you would, this would be the first step where the sample would come in. You would weigh the sample and put the data into the computer. Everything's here is cloud-based on the on the limbs, they call it, laboratory information system, and that would give you barcodes for each of the tests that you need to do, right, and the tests that we do are the tests that are required by the Department of Health, right, which is standard cannabinoid profile, tells you how much of your different cannabinoids are in the cannabis, and we have to do pesticides. It's an Oregon list of approximately 60 pesticides. We have to verify that they're not in the material, and then we do heavy metals, four heavy metals, and let's see, and then the whole biological aspect of it where you're looking for yeast and mold and certain biological organisms that could affect people with compromised immune systems. We, the laboratory personnel, there's a few of us here that have the background clearance to go pick up the samples. You go pick up the samples, you got to pack them, do some manifests, chain of custody to make sure, you know, like I said, everything that you pick up and sign for when it gets here is the same and it's accounted for, and so, yeah, it's all the laboratories actually going sample, picking a statistical sample, bring it here to the lab, and then dividing it up so we can analyze it for all the different tests that the state is requiring. The people that are growing it have no affiliation with us. They just bring it to us so we can test it to assure that it's medical greater, etc. We can test edibles and oils and basically any marijuana product, but the state doesn't allow the dispensaries to sell edibles, right? But if you're a card holder, a 329 card holder, and, you know, you take the cannabis and you make it into edible so you can ingest it, they can bring those edibles here and we can test those edibles to tell them how much of the cannabinoids are in the edibles. So they can have a dosage, understand what the dosage is, what they're taking, so it's not just kind of like trial and error. The dispensary can't sell edibles because it's against the law, so you could buy the product and then go home and make your own edible. Where is the dispensary? Well, there's one located about a block away here on King Street, which is Aloha Green, and I believe there's one that's not open yet over in Kapahulu, and there's another one that's going to be opening shortly right here on Young Street. We test immediately, basically, because as soon as you get the sample in, you want to test it the way that it was given to you. So the faster you can test it, then there's no chance of something contaminating it or some microbials growing on it. So we test it as soon as we get it. Will there be labs on the neighbor island? Yeah, currently there's a laboratory that's attempting to get their certification from the Department of Health on Maui. It still needed to be worked out how these outer islands are actually going to get their samples to Oahu to be tested without breaking the law. Cannabis, of course, has many unique properties. You have the terpenes, the flavonoids, and the cannabinoids, and so that's why cannabis is such a medicinal medicine. And from here it goes to each of the tests throughout the lab. So, I mean, this is pesticides. Every time for the heavy metals, it'd be another sample he would prepare for the cannabinoids, it'd be another sample. So that's why we need a little more than just this. We have the bag full, so we have enough to do each of the tests that are required. This instrument separates out the pesticides and then runs the pesticide through a mass spectrometer that looks at the size of the molecule and the fragments and then identifies the pesticide by how long it takes to come out, right? Go through the machine and come out to the detector, as well as the ion fragments that it produces as it's being smashed apart in the detector. This machine can run a hundred samples at a time. It's all automated. So, for example, as Caleb was over there preparing the sample for pesticides, basically if we had 20 to do, we would prep all 20 samples. This is where the samples would be loaded. And then through the software that controls the machine, we tell it what to do, whether to test for pesticides or cannabinoids. If we were to find pesticides, there's a certain level that are allowed. It's one part per million per one gram. So, we would test a gram and it would determine if it was above one part per million, then that marijuana or cannabis would not be sold. We're searching for heavy metals inside a cannabis. So, what we're doing here is we're basically taking the sample and we're throwing it on a graphite furnace and then we're lighting that graphite furnace at 3,000 degrees and basically it's exploding that cannabis substance and we're able to shine a certain element light through that. We're able to see how much amount of arsenic lead or cadmium is inside that cannabis matrix. So, there's levels that the state requires that. We'll say that it's safe for human consumption. So, basically, if it passes then we would have to alert the dispensaries that that batch has over the state limit and therefore it's not safe for human consumption. You can have some heavy metals in there, but you reach a certain level of heavy metals and then it becomes considered unsafe for consumption. And it's the same thing with anything like the pesticides or the microbiological yeast and mold, bacterias, everything. It can be present, but it has to be in a very small amount. Once it gets above a certain level then it's considered unsafe for the consumption of people because a lot of people that use the cannabis are using it because they're for illness, for some medical purpose, and sometimes they have a compromised immune system so you want to make sure that they're not being harmed by the cannabis. When we test the samples, we take the sample results and put them into Biotrack. And Biotrack, the dispensary on the other side is able to see what we put into Biotrack as well as the Department of Health. So, everybody knows what's in that lot of cannabis. And so, if it fails any of the tests above a certain level the dispensary has to destroy it. They can't sell their product if it fails. So, everybody's incentivized to make the cleanest product possible. During our validations, all the hoops, we had to jump through in order to get certified Department of Health. You know, a lot of times they wanted us to do these certain experiments. So, we needed clean cannabis that didn't have any pesticides in it. So, we could do the experiments on the pesticide. So, we would go around and get product off the street and test it. And that's when we realized that, you know, 90% of the product on the street has pesticides. I mean, a lot of it's at lower levels it would pass, but still it still has the pesticides in there. Well, since 2000, we've had a medical cannabis system that had no dispensary attached to it. And patients have always had the right to grow their own medicine. And we just passed a bill so that they can now grow 10 plants instead of seven. And now they also have access to the testing labs so that they can find out what's the chemical composition of their medicine. And like he said, whether or not there's some pesticides in it. We started working on getting a dispensary system back in 2014. I was part of a task force that looked at the feasibility and recommendability of having a dispensary system in Hawaii. We had to look at whether or not we actually needed one. And then we looked at other states to see what they had done to try to take the best methods from each of those. And if you want to read all about that, that's all online at the Public Policy Center of Hawaii at UH Manoa. The recommendations that came out of that task force became the dispensary bill, House Bill 321, which passed and became Act 241. And so you can read all about that if you read the bill. You can also learn about Hawaii's medical cannabis laws if you go to the Drug Policy Forum's website. And you can read this booklet that they produced. And this outlines the laws. It was made for physicians and patients specifically, but also for law enforcement. Many of our patients are interested, however, because the process of getting a card, they don't have a lot of knowledge on cannabis. So it's pretty intimidating for them. And a lot of them feel like, well, by the time I apply for a card and get my card and figure out what's going on, the patient could probably expire in that time. I would like to see a shorter path for some of our patients. I've been working on helping create that path through its Act 230, the Legislative Oversight Working Group. And we have identified that need to expedite hospice patients' applications. And the Department of Health doesn't have a system to expedite those yet. But we're working with them to try to create that kind of a system where we can just have a check-off box on that application so hospice patients' applications will go through quicker. How long does it take to get a card? It's been fluctuating. It used to be 10 working days turnaround time. And now the Department of Health has gotten it down to five. A lot of physicians who don't want anything to do with it, it's not prescribable five milligram pill. And that's all they're comfortable with. There's a difficult time finding a doctor who will assess you and get you a card. I have found a few doctors that I know do it and I've been leading patients that way. The Medical Cannabis Coalition of Hawaii can help patients find the providers. They just have to email me us at info at mcchi.org. And we will send you a list of providers that allow us to give out their contact information and a letter explaining the process and links to the Department of Health's website to help patients navigate that. If a patient is sick, can their caregiver help? Caregivers can grow for patients and they can also visit the dispensaries I believe to pick up the medication. We've been talking about cannabis and hemp and hash for what 10,000 years or more. Tell us about the history of the plant itself. I think they found some cannabis in some of the tombs in Egypt, you know, when they when they unearthed some of those tombs. So it's been in use by humans for several thousand years. We have records going back to up to 10,000 years, China, Siberia, Egypt, and of course India. It has always been a medicine. At one point it was legal and then it wasn't legal. What happened? There's a lot of history and mostly starting in the 1930s and it was largely due to racism and industrialists who were who changed the name from cannabis to marijuana and got rid of marijuana. What marijuana is what a Mexican name? It came from the Mexican immigrants, the racism against them. And so it was largely targeting, I believe hemp and because that is another form of cannabis sativa. Those are both cannabis sativas, the medicine and hemp. You had one that you were talking about. Yes, with Hearst and Hearst, the publisher? Yes, he at that time they were using hemp for paper and he owned Forrest and a newspaper and he wanted them to use trees and basically in his paper he started the marijuana issue. He blurred the lines between hemp and marijuana. He came up with that and then attributed it to Mexicans and African Americans saying they were smoking it and raping white women and people slowly turned. He also had a son-in-law that was in law enforcement and that's how they started all of this craziness we know today. Schedule one, that was actually the final nail in the coffin of the medicine was in 1970 with the controlled substances act and that was also based on racism and targeting hippies and blacks who Richard Nixon did not care for. So they created this scheduling where all medications or non medications, all drugs are placed in this controlled substance act and schedule one drugs are those with high risk of abuse and addiction, no medical value whatsoever even under the supervision of a physician and we know it was very unscientific. It doesn't meet any of those criteria, it does not belong in a schedule one drug category but that's the most highly illegal category you can and effectively shut down research to find any beneficial effects for it and started the mass incarceration that we see today. We would like to get it out of the schedule one drug category, there is some legislation Congress has introduced that would do that it would get it out of the schedule one drug category at a federal level which is what we need to do. It would help with a lot of issues including banking. How are they going to pay us right because we'd rather be paid by check or you know some way to keep track of all our accounting issues and so some of the ways people are getting around it is using the cryptocurrency so everything is you know you do a transaction let's say with bitcoins and then you just convert your bitcoins into cash in your bank account so there's no actual schedule one illegal transaction involved. That's one way people are getting around it or I mean I know in the mainland some people are using community banks you know that aren't regulated so much by the feds but I know here in Hawaii some of the banks already have issues. People are used to getting pharmaceuticals to fix their illnesses so I mean that's it's hard for them to accept that maybe this plant that they've heard that was you know the gateway drug and all these things bad things about it is actually a medicine and it takes them a while to get past that. Is it a gateway drug? Oh I don't think so I think alcohol is the gateway drug for everything you know there's a lot of people that that use cannabis that don't use it to get high they're using it just to help them help their body and help themselves get better. Yesterday's paper there was a poll that said did you approve did you not approve or did you not understand the majority was in the middle they didn't understand the whole about medical cannabis what it does what it doesn't do. Right and that's typical because most people think of weed as or cannabis or marijuana whatever you want to call it is something that you know people use to get high they don't see it as a medicine and it's it's I more people are are starting to change their mind on it and but most of the time it takes a personal experience or somebody has a family or somebody they know that's going either dying or having something that only cannabis is helping with or helping a lot more than the pharmaceuticals and that usually is what changes their mind. Younger people have a different attitude and older people usually don't change their minds until like he was saying until somebody needs it. Anybody who wants to learn about cannabis as the medicine please contact me in my 30 years of working in health care I have never seen a medication that has such a great safety a broad margin of safety and a great therapeutic index. I'm a little distressed to hear that a lot of public don't understand how the system is going to be working and I feel we've been lied to for I don't know how many years 80 years about the the true dangers of this medicine and I would really like to help people understand what the true dangers are and it's not what we're being told. Now that we're doing business for the legal production and sale of medical marijuana what issues or obstacles will emerge and how will the path to recreational marijuana be affected. This is an important story it will affect many of us in one way or the other think tech will follow the story and let you know the sea changes going forward. If you want to know more about medical marijuana in Hawaii you can follow our think tech talk show on the subject it's called the cannabis chronicles and it plays bi-weekly on thinktech hawaii.com on alternate Tuesdays at 11 a.m. hosted by your own Marcia Joyner and now let's take a look at our think tech schedule of events going forward think tech broadcast its talk shows live on the internet from 11 a.m. to 5 p.m. on weekdays then we broadcast our earlier shows all night long and some people listen to them all night long if you miss the show or if you want to replay or share any of our shows they're all archived on demand on thinktech hawaii.com and youtube for our audio stream go to thinktech hawaii.com slash radio and we post all of our programs as podcasts on iTunes visit thinktech hawaii.com for our weekly calendar and livestream and youtube links or better yet sign up on our email list and get the daily docket of our upcoming shows thinktech has a high tech green screen studio at pioneer plaza if you want to see it or be part of our live audience or if you want to participate in our programs contact shows at thinktech hawaii.com if you want to pose a question or make a comment during our shows call<PHONE_NUMBER> help us raise public awareness on thinktech go ahead give us a thumbs up on youtube or send us a tweet at thinktech hi we'd like to know how you feel about the issues and events that affect our lives together in these islands we want to stay in touch with you and we'd like you to stay in touch with us let's think together we'll be right back to wrap up this week's edition of thinktech but first we want to thank our underwriters that wraps up this week's edition of thinktech remember you can watch thinktech on spectrum oc 16 several times every week can't get enough of it just like rb does for additional times check out oc 16.tv for lots more thinktech videos and for underwriting and sponsorship opportunities on thinktech visit thinktech hawaii.com be a guest or a host a producer or an intern and volunteer to help us reach and have an impact on hawaii thanks for being a part of our thinktech family and for supporting our open discussion of tech energy diversification and globalism in hawaii and of course the changes in our quality of life you can watch the show throughout the week and tune in next sunday evening for our next important weekly episode i'm michelle daigle and i'm rb kelly aloha everyone


Everyone, Basel Chapman, this is the Tiger Financial News Network market update on this Friday, the 18th of March. We're looking at the Dow down 136.34,342 after such a spectacular three days, you just, you need a bit of a breather, right? So we're sitting right on the 200-period moving areas. That's the importance of this 200-period moving areas. We were above the Chapman Wave Insight Track propellant zone, it is a repellant zone. It's now a propellant zone. We'll see how we get into Tuesday. This is a by-mode in the, in the Dow, theoretically, based on the Chapman Wave methodology, we still should go to a leg D, make a peak C, and then a leg D sometime next week, and then we have to reassess. We're looking at the S and P not as strong as the, a little stronger today, but not, not as strong as the, the Dow and the chart formation. It is only now just pushed above the 200-period moving average. We're watching this closely to see if it turns this by signal into a by-mode. The QQQ, the NDX100 is trading right now. This is a, this is good action. And this is what I was talking about in my show, the tide conditions are a few moments ago, that the old, the stocks that were so fantastic going into the end of 2021 and then just collapsed, they're finding some support now. And they might not go all that far, but they're finding some support as the stocks that had late rallies and held up much better are starting now to digest those gains. We're looking at the IWM, the Russell 2000, also a nice move up, up 20 cents. That two or five, it's just sitting on a resistance line. Most importantly, gold, gold is trading. It was up sharper early in the evening, and now it's up, it's down seven. It's 90, it's just digesting huge gains. You're looking at Crudeau, it's the same thing. It's up, it was up now, it's down 89 cents at 102. Just huge gains is digesting them. And the TLT, that was our focus this week, and we'll make it our focus again next week. That is bond. The yields are holding in the higher range. Will there be some pullback to have the bonds ameliorate some of that extension to the upside with much higher yields? So I'm going to hand you over to Larry Pazavento. It should be a wonderful show. It's always with Larry.  Have a wonderful weekend, everyone. And yesterday, I think I forgot to say Happy St. Paddy's Day, I'm sorry. Have a great day today and we'll see you on Monday.


Good afternoon everyone hope everyone can hear us if you can't just wave your hands crazily, and we'll do our best to speak up Okay, or just wave him anyway My name is Jonathan LeCours. I'm the vice president of cloud at dream host my background is Development engineering, and I spent the better part of the last two and a half years learning the plight of an operator And operations that brings me to Jeremy. So I've been with dream host 14 years Start out strictly system in been doing more and more development as kind of have to do once you get into this open stack thing We're gonna spend some time talking today about exactly what's been going on there and the pitfalls and the Bonuses that we've been getting out of everything. Yep All right, so first I'll give you a quick introduction to dream host itself if you've not heard of us We've been around for a long time in internet years. We're kind of like Grandfathers, you know so 1617 years old we have over 400,000 customers at this point hosting 1.2 plus million domains On our hosting platform. That's kind of our bread and butter and something we've been doing for a while And we're really proud of it and now we're doing cloud services as well We have a big open-source focus All of our work over the years and our hosting platform has been as a result of the fantastic work That communities have done on things like Linux and Apache PHP WordPress We have over 750,000 WordPress sites hosted at dream host. That's a lot of WordPress That's a lot of PHP. We like to say we kill trees with PHP because of all the CPU that it uses up So we're also the creators of seph our wonderful friends at ink tank Just now over at Red Hat stage while the creator of seph was one of our co-founders So we are very familiar with seph and we're users of seph Big contributors to salamander neutron Oslo all sorts of other things even myself the VP I created a framework called the con which is used by a lot of open-stack projects now For the air API service So we're being an open-source and we're an open source or an open-stack foundation member so we're very committed to open-stack and I'm gonna gonna give you some caveats here because I'm about to be very candid in this presentation and so is Jeremy and and we're gonna say a lot of things that Talk about some of the troubles we've had with open-stack But I want to be very clear that we're very committed to open-stack and we're excited about the community and all the things We're talking about are from our experiences growing and learning and we are contributing a lot of fixes back upstream And we want to share as much of this knowledge with others as possible I think public cloud is a use case for open-stack that isn't talked about nearly as much as private cloud and it has different kind of requirements and So we're gonna try to bring those to light here and help other people help us make open-stack even better for public cloud So in spite of any constructive criticism, we're as committed as ever So first I want to talk about the fantasy right two years ago when we embarked upon this great journey of Getting into open-stack. What what do we have in mind? What were our assumptions? What were our goals? What was our vision for this product? So the goal was to create dream compute a public cloud built on open-stack We felt that the world could really use a public cloud from a smaller player like ourselves We have like I said hundred four hundred thousand customers We felt that we could offer them a cloud service that they could really use and enjoy And we had this great sef thing that we had been working on that We really wanted to get out there and use the block storage That is provided in sef for our cloud So the ultimate vision Was to be a truly open cloud We are very open about everything that runs dream compute including what the switching hardware is What operating system we run on the hosts what hypervisors are? Every single thing about it all of the code that we use and generate in the cookbooks in the monitoring We try to open source we try to give everything back and go upstream So we really wanted to really create a really open cloud We wanted to use sef for block storage We are the first public cloud that I know of that is using sef for block storage and that provides some really interesting Features that we'll talk about later. We also wanted to get ahead of the game and Jump on board virtual networking. This is important for a number of reasons for us Primarily the security that comes with layer 2 isolation for tenants every tenant and dream compute is going to get their own Layer 2 domain completely virtualized and isolated from every other tenant and open stack We wanted people to be able to create these networks whenever they wanted to Be able to completely program the network and we also wanted to do that with IPv6 support from the ground up This is something that was very important to us as a small provider We don't have millions and millions and millions of IP addresses anymore And getting them now is actually very difficult. You have to go to auction and buy them for a lot of money And IPv6 is here and it's not enough here, but we want to help push that forward So that was another critical part of our vision, right? And we wanted to have integrated authentication We wanted that to be a part of the dream host kind of family of products We wanted people to be able to use some of the credentials that they already created use the same sort of management ability To be able to Connect to their open stack to dream dream compute. So this was the vision This is what we had in mind and now Jeremy is going to walk you through the operation side and the kind of physical side of that vision So you can see it's nothing super crazy We actually use VMware NSX for the network virtualization and you probably can't see this diagram But don't worry. We're going to talk you through it and publish the slides as well So that would be the top row of the main compute pods you see Everything is controlled by a separate hypervisor or control pod Basically three racks in our environment It's all running strangely enough on a VMware cluster All of our endpoints for the API stuff for the open stack various service nodes that sort of thing we didn't want to start with open stack on open stack Yeah, we wanted to start with you know something else and then we would eventually the goal is to transition off of that Exactly when we're when starting your first open stack cluster having open stack via dependencies not exactly what you're hoping for So we've decided to go with the tried true easy, you know, just buy it get it up and get it working solution So that runs, you know staff monitors. We've got chef nodes all of our central services the API endpoints for Cinder and for Glantz and Nova and Neutron That's all running in that control pod Back over to our compute pods It's basically a giant stack of storage boxes on the bottom Del R5 15s if you really care 12 disks each three terabytes And then a bunch of super beefy AMD hypervisor nodes Everything's got 64 cores 192 gigs of RAM It lets us over subscribe only two to one on the CPUs, which is pretty great Better than most the other public clouds that we're aware of That's over 3,000 cores for those of you doing the math It's a lot of course Also AMD And then all of the networking gear we decided to Try something new the network Layout is terminating layer three at the top of every rack.  They're all ODM switches just white box stuff running chemilis linux 48 10 gig ports and for 40 gig ports for interconnects between them in a partial mesh It's pretty awesome. You end up spending 60 bucks a cable instead of 600 from one of the other guys, which is Majority of your savings in the long run you get fewer stickers though. You do get fewer stickers big thing fewer stickers Sometimes you might get the wrong sticker. Yeah, which is also true And yeah, that has actually been one of the most rock-solid bits of our Architecture so far. It's it's almost a surprise really how well it's been working for us And and all of the architecture decisions we made were really driven by the fact that we're a small team very small team We have Two operations engineers today working on very tired operations engineers working on dream compute And that is an incredible feat and a testament to them and some of the decisions we made but they all worked Necessarily the easiest decisions or the right decisions. So we're going to go through some of the nightmares now So clearly, you know, it's open-sac. It's a giant complicated problem Project or both So clearly not everything went the way that we wanted it to from the beginning the compute side in particular is What I like to consider an onion not only are there layers upon layers, but you end up crying a lot You've got Nova compute sitting on top of Libvert sitting on top of QEMU 99% of the time they talk together great and then that 1% of the time you're wondering Why is there a QEMU process that nobody thinks should exist and it's Causing, you know some set volume to be unable to be deleted. It's causing ports to stick around in neutron The most bizarre things that you never think should happen all tend to happen all at once To combat that we've kind of written our own suite of software utilities Nova auditors neutron auditors Cinder auditors Auditors for the auditors. Yeah, that's very important as it turns out. In fact, maybe we should just name the project IRS Yeah, it's a good idea and bundle it all together So that's all up on GitHub. I think I think we posted the link later on. Yeah So continuing on Nova Network is no more the future is neutron the future is always great and shiny and just a brand new projects always work It is always true So we were pretty excited about neutron particularly because we're doing a lot of layer 2 virtualization and all this white box stuff and it's all bleeding edge And so, you know, you got three parts bleeding edge. You might as well out of fourth Because why not? That's always smart and sexy and exactly what you want to do Neutron While it was designed for this sort of thing from the ground up it kind of wasn't at the same time They suffer in the same way that a lot of other open stack projects suffer where they are developed by developers who've never run anything and There's a lot of outside influence coming in from various vendors. So you've got and I'm a developer and I've done that and I love you guys I'm an ops guy. I do that when I'm developing something right we can say the same thing about you crazy people So you end up with situations where Feature parity between drivers is not it's almost there and it's there enough that you always expect it to work And then it doesn't This is a hard hard problem. So I want to be very clear. We're huge. We're deploying neutron now We run it and we've got it to work for us But it is not an easy problem and when we first adopted neutron it was pretty early So a lot of progress has been made. Yeah, and we've been pushing a lot of code there So and in no way is it anything that's you know neutron specific This is this is something that happens in any complex distributed system. So it's to be expected You just need to know what to look out for So so software to find networking was so great. We were so excited and there was all these great vendors I mean, there was one at the time or two or three maybe Most of them in stealth mode. We're talking about three years here, right three years ago So we talked to a bunch of different people and we ended up selecting NYSERA who is now VMware Which is gonna be it's I'm still not used to that but it's pretty cool great for them We're very excited for them about that and they provide just rock solid L2 network virtualization So we knew that was the case They were the they were the people we selected because they provided the best of the basic, right? And it worked really well But there was this whole need for you know L3, right? What if you can't actually route anything? That's kind of a problem. So we talked to them about that and at the time They didn't really have anything since then they have developed some things that that provide L3 and that's great and fantastic But we didn't have time for that. We needed something today.   Yeah, they didn't get it and they didn't get open source either So this was a real problem for us So, you know, we did what any Crazy person would do and a lot of developers and operators have done and said Because do that ourselves. Yeah And deal. Yeah, that's that's always the right move So what we decided to do was start up this project called a conda It is a software router runs as a service VM So we get on mag distribution. Thanks to Nova and all this other stuff It's actually just running open bsd and pf. I think Open bsd 5 4 we're testing 5 5 so we're we're able to keep up pretty up to date with all the latest releases Routers run one pretend it you can It's not great. I'll get to this later. You can add multiple networks on The process is not super great. Thanks to open bsd They also run a restful API service so you can push firewall rules You can push DHCP your metadata all of that it gets configured on the fly as a push service In order to Maintain all of this. We have what's called the rug which I think was Mark's yeah name for it Yeah, one of our former engineers Mark McClain named it and it's because it ties the whole room together So if you've seen the big with Big Lebowski, that's why it's called the rug It it is the service that essentially Glues everything together. It spins up the routers the orchestration of all of our layer three services basically It watches on various message message cues for events like I've created a new VM or I've spun up a router I've added network and then it does the appropriate things the API running on the the economy image Now There are some problems open bsd Unfortunately, we found out just very recently does not actually do network hot plugging It does PCI hot plugging and then the PCI devices sit there and don't get configured Which means that we can't simply add new networks to them without a reboot Which is something we're trying to figure out how to overcome right? It's rough, but right now most most use cases you don't actually use more than one network So again, we're in the nightmare section right now. We're gonna get to the dreams in the It's not so depressing. Yeah, the whole talk is not gonna be a downer open in the kimono here And all the all the bad stuff so some of the interest interesting issues We've had had with it neutron is not super happy about handling really rapid API requests Particularly when you're running NSX on the back end There's there's a lot of layers there the last synchronization has to happen and it gets a little bit bogged down Race conditions it turns out are a huge deal and synchronizing state between all these virtual devices in the central process It's it's a hard problem to solve we recently I think probably ironed out the last of the major issues, but it's been years Literally years. It's not an easy problem The service VM updating is hard that's actually one of the more expensive processes because you basically have to initialize all the state So if we're doing You know, we've got a hundred routers running and we have to reboot them all to do an upgrade to the image Which you know, isn't just no s update. It's also upgrading the appliance Application that's running there or anything else It basically means initializing state for all hundred routers as rapidly as we can Which ends up being quite taxing on pretty much everything, you know, it's no great way to not run right over just yeah Kick it over And DHCP management is one of the things that we actually thought would be super easy and it turns out it's not DHCP and metadata even in bare neutron suffers from some pretty heavy delays what 30 to 60 seconds, I think is about what we were seeing and It turns out as we'll get to later that our own home belt system is actually surprisingly good at that compared to What everybody else is using and all these delays that Jeremy is talking about this impacts our vision Our vision one thing I didn't mention was to have 30 second boot times 30 to 40 second boot times So that's a big ask. That's a very fast And that's including the OS everything yeah, not just until the the image starts to boot but up and running and you can Ssh in 30 seconds is the goal now Seth Because we built it ourselves Partially to be able to run stuff like this and partially you know for the hosting side which is pretty obvious We we try to use it as extensively as we could it's backing all of our block storage and dream compute It is behind our dream objects Seth s3 and Swift product all of our images stored in as well inside glance Which is great because means you know, you've got one cluster storing all of your data It's a copy-on-write operation in order to boot from an image into a volume Remember the fast super duper quick. Yeah sub-second volume creation from any image in our cluster now We did have one Incredibly amazing issue with this if you remember from the slide before we've got about 1,100 discs in our cluster At some point I think because we started with a cluster at such an early version has been through a lot of major upgrades Our crush map, which is what distributes all of the data or maps out how the data should be distributed in your cluster Got into a state where our I want to say 20 terabytes of data at the time ended up on 10 hard disks Not exactly a massively distributed data store We had like all of these empty discs and like 20 just like every yeah So the funny thing is we didn't notice because it actually performed pretty well still But the problem was when you're distributing that much data off of only 10 hard disks It doesn't happen quickly. No, it was angry three weeks of data redistribution because it was all coming off of 10 spindles We have a solution to this as well. Yep, so you will not have this happen to you learn from us Okay, so cinder so, you know, noble volume is no more. Let's let's cinder is gonna be great. It's the future It's another new project. It can't be bad, right? We've heard this before so cinder is actually great. I I don't want to Again tell you that it's a bad thing, but there are some interesting things about it and it was fork for Mova so it's a great idea you can Get a lot done with it, especially if you're doing some very specific things like using local storage or LVM whatever If you're using something else say, I don't know sef Things get a little bit different because you can't assume that every operation is going to be hyper fast, right? Deletes and sef in particular are a little slow Creators super fast, which is kind of what you want problem is cinder volume is currently designed to run a single process And so if you get a whole bunch of operations queued up that are deletes in front of your club creates You know, we actually graph and chart every single we spent up about 6,000 vms a day and destroy them And we graph and chart all of the boot times because we want to get it down down down We'd see things spike from the 42nd point up to 120 hundred and 40 seconds because cinder was just sitting there going delete Delete great great great great great Delete so this is a problem So we have a sef and cinder war stories panel tomorrow that I'm on that we'll talk and Jeremy as well We'll talk a little bit more about this and I'm going to get to in the next section about how we fix this problem So sick scaling cinder big it beyond kind of private cloud scale. It's not easy And it's putting out that it's also it makes sense Given where cinder came from because absolutely it was designed with the idea that once you create a volume That volume belongs to the node that it was created on because you're dealing with ice fuzzy things There's one-to-one mapping between the cinder volume host and the storage mechanism with sef any cinder volume process can handle any Volume in the cluster because they all talk to the same thing and that's a use case that really wasn't considered when cinder was created right and Make sense because that didn't really exist when it was created and nothing competitive was really out there either So let's talk about keystone So like I said, we wanted to do integrate off and this seemed like a great idea to us at the time So we were gonna use the keystone plug-in infrastructure, which is there for just this purpose, right? Well turns out no It's not exactly there for this purpose and no it doesn't actually work particularly well for our use case Also, it happens to change from release to release in breaking ways So this is kind of on us.  We'll get there So now the realities it's not all bad news Yeah, we we've been working on this for a very long time And we feel like we've finally gotten over the hump and things are starting to work the way that we're hoping they would Image building It's actually a surprisingly interesting problem we've Come up with a methodology where I literally just throw reboot into RC dot local and let them cycle It's kind of amazing the sorts of bugs that you uncover in the boot process by letting you do that In particular a bunch of precise we found there's all deadlock in Mount Hall About I think five percent of our boots were just deadlocking Saying it couldn't mount some random thing like temp or something else and it's a bug that is Well, there are two bugs one of them was flagged for backport and forgotten and the other one was apparently just ignored By the way, we are having conversations with the appropriate people about this to get it fixed Yeah, we actually had a thing that you turn up when you do this kind of testing So we had a pretty good conversation with them today And I think we finally talked to the right people and things are gonna great. That's awesome get all ironed out So It's not out is yeah at the GitHub This is where we've been storing some of our tools the Nova and Cinder auditors and I think I've got a keystone thing up there Basically every time we encounter a problem. We write a script either to fix it or at least to report on it And toss them up on GitHub now they'll layer 3 and network virtualization We've discussed this a bit in the pains, but it actually turns out that once you deal with it for a while It's starts to work We We hit a bit of a milestone about what two months ago when we decided We weren't really sure if we were gonna get the rug working as well as we needed it to So what we decided to do was talk to you VMware Get the cluster set up just a small one a test scale on one of our hypervisor boxes and just Put like a deathmatch together between later layer three services with all the bare bones neutron stuff Against our layer three services with a condo in the rug And we ran for two days three days something like that like over a weekend And just compared times the same boot time tests that we do for everything else where we find out How long does it take before cloud and it launches until cloud and it finishes and then you can SSH in and It turned out that we were actually like two to three times as fast We've talked to the neutron guys apparently they are quite aware of some of the timing issues Particularly with a metadata service and neutron and I think Aaron said Juno was gonna see a lot of fixes there and looking promising another point on this front This is not a statement that the L3 services from VMware are good They're actually great, but there were a couple things that were specific to us They didn't make sense one no IPv6 support, which is something we said was one of our tenants that we really wanted to have We were open two months ago to just saying well, we gave it the good old college try and IPv6. Bye But you know, there's also some other things. There's DHCP the way that you have to do DHCP and the all three services for for NSX are very different than the way that we're doing them and we actually end up Preferring and being able to scale ours better. So yeah, we ended up we were surprised here actually we were expecting to be defeated But we didn't yeah, we also One interesting thing is as a bit of a workaround for those times when DHCP does lag a little bit We configured all of our images to try using config drive first Which it turns out Nova just kind of does for you even if you don't tell it to mm-hmm because there's an option to always use config drive and Basically as far as I can tell what that does is disables DHCP But config drive is actually always running even if you don't tell it to so using cloud in it It will just look try to mount something with the right label config dash to remember correctly, which is kind of odd. Yes, and then it works and It actually allows your image to boot up a little bit faster might take a little bit well a while for the networking Just to actually get packets in But it means that you don't have that two minute timeout or whatever it is for DHCP, right? We've ended up being just a few seconds slow So as Jonathan said earlier Our testing consists of basically spinning up about ten six thousand VMs a day Tearing them down instantly and just graphing at every point where we can determine something has completed An important stage in the boot process We graph it as a data point and then if anything fails if at the end We can't SSH in or we see in the log the cloud in it wasn't able to do its thing We graph that as another data point so we have failure counts success counts and timing info for all of it And this keeps us up at night We we lead off every stand-up meeting every morning looking at what our current success rate is how fast are our boot times How many fails you do get why? So this has been all of our conversation lately, which is good Yeah, our QA guy actually wrote a little little script that uses phantom.js to Download all the graphs and just email them every morning. It's pretty great. It's a nice Great when you get back to the ninety nine point nine nine eight percent. Yeah success. It's not so great when you get the eighty five percent Now Seth I don't know if the CERN guys are in the room, but apparently I just yes Apparently we're tied for the biggest running operational stuff clusters in the world at three petabytes And I'm sure as they'd be happy to tell you it's not always great Any cluster that size no matter what workload it's running you're always going to run into some problems Just like with open stack after we had that data distribution problem I basically spent two days just writing Nagios plugins to check every single last thing that I could figure out We could have caught if we had known to look for it Those are also up on github under my own repo right there because to be fair Most of these problems are caused by human error that you you should really be monitoring for I mean If you're gonna monitor one thing that's the most unreliable in your cloud. It's the people who run it. Yeah And honestly, we don't know how long it had been in this state We don't know if it was somebody who fat fingered something or if it was an upgrade gone bad And that's that's that's on us and now that's what all this stuff is supposed to catch and Even if you're a small guy and you can't you know afford ink tanks professional services. They're on IRC They are very mailing lists You'll be hard-pressed to find any time day or night where you won't get a response within a couple of minutes on something That's gone horribly wrong or horribly great if you feel like just or you can get Sage's cell phone number Which I figured out and that's a really great way to get things going I'm not sure you'll be able to do that, but So let's talk about scaling Cinder mo workers less problems, right?  It's senseless So, you know, we tried a bunch of different things, you know we tried some some of the stuff that we heard the CERN guys were doing and What we ended up doing and the thing that we're working on lately is we actually implemented an RPC driver That will intelligently send certain operations to different topics And then we made a small patch to Cinder volume such that it could bind to only specific topics in this way We can create a pool of different, you know Different pools of Cinder workers a bunch of Cinder volume workers that can process only specific things So I can have a pool of Cinder volume nodes that are designed purely to make fast operations happen as quickly as possible And a separate pool to relegate the slow operations to so that they don't Slow down the fast ones and this helps even out the lumpiness So what we were seeing in our boot times is like this spikes and it's now Starting to go like this and I'm hoping that with this RPC change, which we were talking to the Cinder devs about Well, if I remember correctly that one patch one line patch to Cinder was actually a bug fix Yeah, it was it was a feature that they were supposed to send or yes work right right note to Everyone if you configure if you configure Cinder to lift Cinder volume to listen on different topic it just won't But now it will yeah, we're gonna upstream that so All right Keystone so We don't always have the answer to these problems where we say here's the smart trick We did to fix this problem in this particular case We met with some of the keystone devs and talked to them a lot about our use case and Talked to some of people on our security team and did a lot of thinking and decided to give up And we went back to vanilla keystone and you know what life is much better And at the end of the day we actually prefer it this way now that we're there because We actually have the ability to create a bunch of different users in a very you know nice way We can integrate with our control panel This is where failures are actually kind of good sometimes because it makes you rethink why you're you know using some feature or developing something Because after you've been developing it for a year, you know circumstances change. It's good to go back De scoping is okay We did that in a couple other places that I haven't mentioned Especially on the networking side and if you have questions about that we're gonna have a couple minutes left We'll talk about those as well All right, so on to the dreams What did we get right in the end? Fewer slides in this part guys. Yep So packaging and testing We like I'm sure Probably half the room here do we run our roller-roam packages We use Jenkins Jenkins job builder FPM package everything inside of a virtual and in that packages because everything's running on a bunch of precise and talks for dependency management basically, right? Thanks Jenkins job builder. We were able to pretty easily target any particular patch set or Tree branch in in a git repo so we can pull in new stuff. We can run our own patches on top of something if we decide to fork And it ends up being really flexible and we're liking it a lot. It works really well Any packages that get built through this which for most cases happens on a nightly basis or as it detects changes going into upstream github They automatically get rolled out to our staging cluster Burn in for a little while before a manual upgrade into our promotion into our production cluster whiteboxing whitebox switches as I mentioned earlier with cumulus and who are great partner They've been public for how long now six months. Yeah a while. Yeah They're pretty awesome. Basically you end up with a Linux box running essentially Debian in our particular case they show up with just 48 ethernet interfaces that show up as 10 gig and then 440 gig that we use for interconnects You can I think on the gear we have if you want do QSFP on those and get four 10 gig ports out of every 40 if you really need some more density The newer Trident 2 stuff. That's just starting to trickle out now. You're looking at I think 3640 gig ports or four times that number of 10 gig ports, which is just insane The other nice thing about this is when you're buying whitebox switches Wow, do you save some cash? The the list price on one of these switches is about six thousand dollars if you're buying one If you're building a cloud you are not buying one You are buying a lot of them and it drives that cost down a lot and cumulus Linux is actually really inexpensive as well Their pricing is also published no games really really great Yeah, it's oh and they're doing demos with us at our booth Throughout the week so a couple times a day Through Thursday, I think well actually ask station to one of our switches and you can yeah see us poke around on it Actually the live cluster yeah One of the crazier things which I actually didn't realize until I was putting together a talk about six months ago As cheap as the switches are we really save is in the interconnects we run Just copper for the 10 gig because we're doing pretty short runs But even with copper Other vendors were going to be charged like six to eight hundred dollars a cable Which is absurd and we're at least a year ago last time we had to buy some we're paying like 60 bucks a pop The at the end of the day, I think our costs were about 10% of what we were using for our commodity or our Hosting infrastructure using you know the normal players in the game All right, we're running close to our end, so we're gonna speed it up a little bit Oh good. This is the last slide before the end. So yeah On this the Seth stuff It's it's probably one of the better storage platforms that I've used and I'll say that just because we developed it You know being in the hosting business for 16 years.  Yeah comparatively Lightning fast boots the open-stack use case is pretty incredible for it Because you literally upload an image in a glance preferably raw image But with cloud in it, you know, we can build these images that are like three gigabytes. So it's not super huge It gets Copied into a volume the moment you start up an instance it's three-tenths of a second to do that and then Cloud it's yeah, it starts booting cloud in it will actually just automatically resize that volume to whatever size you want to be Which is almost just as instantaneous and you can run an 80 gig instance off of a 3 gig image and You know, you're ready to run a website in like 30 seconds my favorite thing about Seth as the person who runs the business unit is that I can operate it with a very small number of People because it's self managing self healing that doesn't mean that these guys aren't you know Sometimes pulling their hair out and working really hard on it But you know, it is enabled us to be doing something that would normally take a team ten times that large minimum To do with a couple people. Yeah, unless we're doing an upgrade because you know, it's a cluster of roughly a hundred machines And it has to be fairly well orchestrated because it's particularly with major version upgrades You have to have a very set regimen of the order in which you upgrade everything because disk formats change and all this other stuff But outside of those instances, yeah, we sleep well. Yes, nothing really goes wrong thankfully As long as we everyone just everybody If it does it's usually our fault. Yep So what does this all mean? Well, good news the the beta is expanding on May 27th We're looking forward to adding in more people. A lot of you have already signed up We're really excited to scale it up now and and we really think that we've got this thing working So we're looking forward to your feedback, especially from you guys in the open stack community Because if you do see things that crop up, you'll help us find out what the problem is and get it fixed So the future is bright. We're out of the weeds Why aren't we going straight to GA? Well, we want it to be perfect No, no, we don't want to be perfect. We want it to be really good, right? And we feel like it's good now, but it's not quite where we want it to be So come help us out make that happen come by our booth and see a demo of cumulus linux running on our white box switches and in our actual cloud and Come see us tomorrow in the seph and cinder war stories. Okay, 1150. I think so. Yeah, thanks Questions, I think we have maybe time for a couple if you could go to the mic that would be great What what hardware vendor we're using with the for the ODM switches? I think they've changed names now You have now they're Agama. I think yes Agama Agama. And what was the previous name? Delta networks Yeah another question you mentioned you guys are doing a Vcpu-to-cpu ratio of two to one is that two vcpu to one hyper thread or actual core actual core Yes, so you mentioned about performance. So do you use something like rally to test performance and You also mentioned a second question for that is like 6,000 VMs booting at an instance. So it's like Currently keystone runs in an event-led based model. So do you use something like Apache or something like We use Apache for keystone and As for using rally we do not We have our own test suites that we use okay, but we are planning on you know We've done some tempest stuff as well Now with ref stack coming, you know, there's some other things we're planning on doing there So testing is important to us all the things bring them on. So right now. You're not using rally not yet Okay, but you package say Apache and all those in a virtual and kind of things not Apache. No, it's basically the open stack code And a few other extra things, but all of the Apache and other system services. We're just using a bunch of packages Thanks You mentioned that you created a not just plugin for our safe. How do you deal with monitoring of other parts of Open stack? Well, we're not doing a whole lot of monitoring of the rest of open stack just yet Mostly it's of our performance tests If any of the endpoints go down they're gonna catch it we will eventually Explicitly monitor every endpoint individually. We're just not there yet Yeah, we are using log stash Grafana Graphite, you know, we've got a lot of things in place and I guess we just haven't written every single monitor. Yeah Yeah, basically everything's being looked at it's just it's not alerting anybody So you mentioned that you guys gave up on some changes during making the keystones Maybe I missed it, but what was not vanilla about your initial keystone? We were we had implemented a custom plug-in to keystone that would middleware middleware or plug-in I'm not exactly Rosario. You can probably help me answer this question It was at a plug-in. Is that what it was? So it was a plug-in and it was Hooking into our off to make it possible to use those credentials And because of both the way that the keystone plug-in mechanism was implemented And the fact that it was unstable meaning it was changing in a breaking way from version to version We couldn't get it to go fast and we felt it was going to be an a burden to maintain So we just said forget it And so just one more question on the stuff you guys have a shared storage across all the nodes They see the same storage. Mm-hmm. Yeah, that's it for time. Yeah, I think we're maybe out of time come and talk to us later Yeah, well, we'll be both at the booth. Yeah quite a bit in the next couple days. Thank you You


Beast is the newest film sent to theaters to die starring Idris Elba and a pissed-off Mufasa. Let's talk about it. For the five of you planning to see this in theaters, I will say this is gonna be a spoiler-free review. I can only ask that you subscribe to the channel Adam Does Movies as I post tons of movie reviews, content, sometimes I'll sprinkle in some TV material here and there each and every week. The star of the film is obviously Idris Elba, who can make even filing paperwork look interesting. Charlotte O'Copley's in this too. Haven't seen him in a hot minute. Nice to have him show up here. I wish he had more screen time because as it stands, most of the film is Idris and his two girls, and I have to say the girls, they're trying. They're cute little things, but it's a struggle. It's a struggle. And not so much because the girls aren't believable, although the younger one is having a hard time convincing me there's a line after them, but it's more so that they're obnoxious, the writing that they have to work with. Oh my god, some of the decisions in the script are perplexing. The movie clearly takes some inspiration from Jurassic Park. At one point the daughter even is rocking a Jurassic Park T. They have scenes where the girls are left alone and the lion's skulking in the background and you're thinking, okay, this is the kitchen scene from Jay Park 1. You know, but done much worse.  It's not worth it. The older daughter actively does the opposite of what the dad says, constantly getting in harm's way or screwing things up, and it's just not fun to watch that play out. There's a scene in the Jeep where the dad's looking around taking stock of the inventory, and he picks up a couple bottles and says to the kids, look, this is all we have for water. We have to conserve this. The girls instantly start bitching him out, like, ah, this isn't fair, dad. That's all the water we have?  What are you thinking here? Like, he can just run to target quick. Like, they're not in a horrible situation where a lion's been attacking them. It's just so tone deaf in the moment. I'm like, these kids are garbage. Throw the bottle of water at one of them. So, outside of the frustrating cast, obviously, we have the lion to deal with, and I will say the CG lion, the beast, very well done. I'm sure the budget wasn't too insane for this movie. It's clear they put some dollars into the lions, though. They also spent some budget on CG flies.  Well, the lead characters are there, having a little convo, taking it all in, the scene around them, and there's flies everywhere, but they don't once go like this, or like, wave one off. It's almost like the flies were added in post-production. The film is rated R. I think it could have got away with a PG-13. There's not that much here that made me go, oh, whoa, that was a lot. That was definitely deserving the R rating. I don't know. I don't see this movie doing very well. It was an all-right watch. I could sit through it again. I like the message. You know, it's very simple premise.  He starts killing the poachers and anyone else that's in his way.   Obviously, this will draw comparisons to Ghost in the Darkness. I would watch that over this any day of the week. That's just a funner movie in my opinion, a little bit more scary. I don't know what cameras they're using or if they just really bumped up the brightness in post. Everything in the film feels about 15% too bright. There's not enough contrast. And the night shots, oof, they look rough. They look like you're watching a VHS copy of the movie. There is artifacting everywhere. There's snow all over the night shots. Very, very ugly. I don't know what happened. Alright, I've said enough. I'd like to hear from you though. What do you think about the movie if you saw it? Or are you interested in seeing it now? Let me know in the comments below. I personally think this is one you could wait to stream. I wouldn't rush out to go to this movie unless you're really bored. It'll entertain, but you will also be pretty frustrated at the same point. Make sure to subscribe if you haven't as I post tons of movie and TV show related content here all the time. It's a good time. It's a lot of fun. And hopefully I'll see you around. Since you made it here to the end of the video, you're probably a subscriber looking for ways to help out the channel. One easy way and free is to join me on TikTok. I'm trying to make a presence over there as well. Just cutting up these videos into shorter segments, giving people a little taste of the flavor they can find for me. That sounded, that sounded sexual. It wasn't supposed to. Anyway, check me out there. You could become a Patreon as well.  Not much though. There's a dollar tier, a five dollar, a ten a month. You just say, Adam, love your content. What can you offer me now?  These are good videos too.  No, they're edited, they're polished, and they're just for you.


Good afternoon. This is your host, Guillermo Salvatlliet, on Perspectives on Energy, and welcome to today's show. Today, we'll be discussing inverter-based resources and new NERC regulations. So, once again, I'm the Director of International Services for HSI, the Health and Safety Institute, Industrial Skills, and we recently covered this. He's not with us today. My colleague Jim Stanton and I covered this earlier this week with an earlier webinar, but definitely want to discuss what has happened with these changes. They've been coming for a while. It was some of the new standards that NERC is putting up where they've lowered the threshold for some of these renewable resources, namely solar or the wind. In the past, it's always been 75 megawatts or below. Usually, you see them running at 74.9 and they would build them to that spec or they would just limit their output to that spec. And really, all that was was to avoid, for example, compliance, violation, exposure risks. Now, we're going to discuss that today as far as when they make a business decision with the wrong perception of what that risk is. And we'll talk about what the kind of help that exists out there. We'll talk about what the steps are. We'll talk about there's no need to be afraid because it's not that hard and there's plenty of help available. Inverted-based resources for regulatory changes, but we'll talk about the efficient and cost-efficient strategies for these NERC compliance programs. I mean, we help quite a lot of clients doing that, but there are many others, one of the only ones actually do that. I've been working on NERC compliance for a very long time as well, my career. I mean, NERC standards have been around since after the 2003 blackout. But before that, they were called NERC policies. And yes, they had audits and audits, but they never really implemented those standards. They went from policies to standards. And yes, they had violation penalties where they could be as bad as a million dollars per infraction per day. But the truth is for these smaller generator owners, generator operators, have never really seen that kind of violation and much less seen that kind of penalty imposed on a generator. So just why again, I emphasize, it's not as bad as people first think, and it's terrible when they make a business decision just limiting their output, merely for the idea that they have the risk is just too great. That just tells me that they don't have a full understanding of what their risk is, or they have zero understanding of what their, what their compliance requirements are, or whether they'll be able to meet them or not. So hopefully there's many ways to do that. All right, so standards, right? Is it risk, routine, or both, right? And one of the things we need to look at it is when the most important thing, and I'll skip down to the bottom, right, is preparation is key, right? So you have a lot of advanced warning, right, before any standard becomes effective. And how they apply to these inverter-based resources, well, you know, for example, the ones that are applicable to each standard, right, they're well, it's good news is that now there's enough of you out there on the grid that you're creating, at times, a reliability risk. So that sounds bad, but at the same time, it tells me that you have saturated the market full of generator operators that now NERC has his answer for. Granted, the more entities they have to keep tabs on, and they have to follow up and look to see that your filing compliance, the longer it'll be before you actually get called up to do that. But at the same time, it's understanding what your obligations are as these standards become applicable to you, right? So mind you, NERC has been around since right after that 2003 blackout. So they've been around for a while. And they normally apply to larger utilities, co-ops, that sort of thing. And then usually entities that had facilities a hundred KV or above or 75 megawatts or higher. Well, now this is all changed, right? When it comes to these, these types of renewable resources. So the threshold's gone a lot lower. And so as a voltage, so now you're going to find yourself right in the situation where you are now subject to standards. If you already have an existing asset, now it's, now it's going to wrap it. These standards will now involve you. So how do you manage this, right? Cost and effectiveness, right? As I said earlier, preparation is key, right? And a lot of that involves having all your documentation, which I think for the most part, you already do. At some point or another, you had to do interconnection agreement with your local transmission or transmission operator or the balancing area. You probably have to do a lot of submit. Well, do carry out and submit a lot of system studies. You have to submit, for example, all of your line impedances, all of your protection settings. I mean, you do have equipment in your, that's going to protect your side of the asset, right? You already have relaying and sort of protection that you have in place to protect your asset. So in a lot of cases, these already meet compliance anyway. And if not, there's a possibility, right? You can change that to fit these new right through capability requirements. And if not, if you are stuck with something you bought before these things changed, all you have to do is make a declaration that, hey, I don't have the ability to make settings that allow for a right through. But, and that's fine too, right? So these are just examples, right? The other thing is I'm making sure you have documentation and you're making sure you have, are your evidence and audits, for example, don't need to be as scary. So here's what you don't want to do in, when it comes to these like compliance requirements and resources, right? You don't want to limit your facility in order to avoid these standards. If you have a facility that's rated 80 megawatts and I've seen entities that will run it at 74.9 because they didn't want to, they were willing to forego those extra five, six megawatts of sales revenue just to avoid this perceived risk compliance, right? So these wrong approaches, right? It's, and I understand they're making a business decision based on risk perception, but in reality, that perception is probably incorrect, right? The other thing is, for example, your OEMs and all your EPC contractors, right? So when you install these sites, you're going to have contractors that do all of the facility ratings for your conductors. You're going to have protection and control contractors that are going to install and test all of your relays. You're going to have an engineering firm that's going to actually do a thought study a lot of your equipment, right? So that's just a switch chart. On the actual generating facility, right? On the inverter, there's going to be settings in there that is already, is for the most, for the most part, right? A lot of them are going back five, six years, they have the capability to do what they call write through, whether it's a frequency excursion, voltage excursion, or even being able to write through a certain fault that's happening far off in your system, right? So these are the things that you need to understand, right? To be able to set that up. And it's not hard to do, right?  Okay. So the other thing, you don't want to try and figure it out later. If you're going to become online, you're a new facility, right?  Don't in your rush to meet that deadline to come online and separate those megawatts to then decide, no, we'll find all that compliance evidence later. No, you need to think about that before you energize everything. And the reason being is that it's easier to delay a project and make sure you have everything in place rather than finding out during an audit, right? That, because one thing the auditors don't like is, is to know that you made it the deliberate decision to ignore compliance just to make, just to like make revenue. So they are more willing to probably, in my experience, probably, like, let it slide with a warning if you just had an oversight, as opposed to then finding out that you decided to, to forego the compliance process just to be on schedule for something. So that's definitely not, not a thing to do, right? When it comes to compliance. So don't try and figure it out later. Get that figure out ahead of time before you're commissioned or energized for the first time, right? Again, my second bullet, when's the facilities have been running before focusing on reliability considerations? No, that's, that again, that's something you don't want to do. You want to have that already managed and looked at way ahead of, way ahead of energization, right? Usually in the very beginning stages, right? I've actually like scoping this out, doing some of the engineering studies. When you have your settings, when you're testing these relays, that you're going to need to, for example, run, run this, the site or whether it's a switch hard, or it's the inverter based resource protection, right? There's settings that go into that are specific requirements on writing out of disturbance, whether it's a frequency excursion and voltage, or writing out during, during an issue with a line fault. So, right? So they're not that bad, but at the same time, you have to make sure those settings are in there and tested and you have evidence of those tests that they were successful. So the way ahead of energization, right? What's next? Assume your viable resource status contributes less reliability than other technologies. So here's the thing, right? Maybe five, six years ago, that may have been true, but right now there are so many renewable resources and so much variability that you are impacting the grid in such a way that NERC is now definitely looking at what's happening, that they have to change their, they have to expand their scope of standards and applicability, which now includes these inverter based resources. So, yeah, you're, the good news is there's a lot of you now. So, take that to her. What's next? Keeping reliability considerations a side item to be addressed when the time is available. No, once again, compliance is important and planning is key. You don't want to do this at the, as an afterthought or then try and catch up later because believe me, you're going to have problems. Whenever you try and do this after the fact, you end up not finding evidence, which is a bad thing because, and then the auditor will know that you just didn't do this up front. And you're better off, you're better off getting all this ready ahead of time before you try and go back and buy out what we've got and just don't ever, ever try and create evidence or write at the station saying, I hereby declare a test that doesn't comply it because it's going to be an issue. So, again, most of the contractors that you hire to actually commission all these sites are going to be pretty well versed on providing documentation and carrying out testing that is according to these NERC standards, PRC-5, PRC-25, FRCA. All these standards are really, really important and they're often violated. But again, these contractors are really, really pretty well versed on this and if not, then you need to find different contracts. What else? Be able to integrate reliability considerations into the routine process. So, part of that is the, they're the operations, right? And you're operating procedures in your, in the way you conduct business, in the way you operate this molecular asset, right? Reliability is a crucial aspect and the reason that these standards are in place is because they want to ensure reliability. They want to make sure that your asset along with thousands of other assets out there are not going to cause a major system disturbance that will eventually to a blackout, which means you will be able to produce revenue anyway if you're blacked out. So, that's the other thing that they want to make sure that you don't do. So, do not fail to integrate reliability considerations, right? So, and again, that is something that you can implement, integrate, right? With your procedures. And when you write a procedure correctly, it doesn't have to be that long, you already can bake in compliance already in there. So, you want to make sure your operating procedures are compliant. You don't want to try and create a compliance procedure that you're going to force your operating personnel to try and use. So, remember, operating procedures should be compliant, not try a compliance-friendly procedure to kind of retrofit it to make it friendly to operators. All right. So, here's the thing, right? Now that these standards apply to you, right? Being a registered entity, right? The good news is that now you have the ability to be able to vote on some of these standards. You have the ability, you have a saved, you have, there's value, for example, and being part of that registered valid body, right? The other thing is, you know, there's also value in your contribution as an entity, meaning a generator owner, a generator operator. So, you're going to have the ability to actually contribute to whether these standards are, so most of these standards go through like a process in which, you know, they several years, right, where they vote on it. And at that point, you have the ability to make changes or to even vote on some of these changes, right? So, remember, standards are part of normal operations, right? And part of that is, for example, cybersecurity is something that is very important, something that you're probably already doing, right? Cybersecurity is a threat to everyone. Some of them are a more juicy target than others.  And one of the things that's the biggest risk of all, really, it's your cybersecurity, right? The other thing that's important, and it goes hand in hand to cybersecurity, is protection system operations, right? So, a lot of these relays are network connected. So, that's one of the things that also comes into play. So, again, understanding that type of like cybersecurity risk is something you're going to do anyway with business, right? So, it's a matter of like paying attention to how you do that, making sure that it's compliant and that you have evidence, right, that you did that. So, again, not that bad. The other interesting thing about this is that when everything is transparent, right? You're not going to have any, there should not be any surprises. You're going to know years ahead of time when there's a change or a standard, a requirement, a criteria, and they'll even give you the rationale behind why they're making those changes. And as a stakeholder, right, being part of the ballot body, you will have a chance to vote or write comments on some of these. So, that's another really great benefit of being a registered entity. So, you have a lot of like buy-in and you have input into how this happens. So, again, with NERC, there's nothing hit, right? Once you're registered entity, you get to see everything. Also, there's another resource, right? The North America Transmission Forum, which is made up of all your peers, all of your GOs and GOPs, and all of you help each other when it comes to dealing with NERC compliance. Nobody, there's no vendors and there's no NERC personnel allowed in that forum. So, it's just U.S. peers. Now, when I used to work on the utility side, I was a rep for my organization, right, my utility. And I got to have a lot of important information on there. I had peer reviews sometimes where the peers would come in and kind of like review what I was doing and get me ready for an audit or they would critique my process. So, again, it's another great asset to have, right? So, you're not alone in this and you can get a lot of help. Ultimately, there's entities like us, right? They can come in and help you with a complete anti-incompliance service or there's others that can help you with advising here and there when it comes to NERC. Next slide, please. So, bearable burden, right? So, here's the thing we talked about. As a registered entity, right, this is not that bad. Evidence documentation, you're going to be able to, it's just, when they ask me how many hours a month it takes, once you got it up and running and you have your compliance program up and running with your facilities, right, or your assets, or you have your system, from what we've seen, it's about eight to 12 hours a month, maybe six to eight hours a month on your ongoing compliance duties. And that's really, it's about making sure that you have your compliance documentation or your evidence properly stored, labeled and ready. Also, keeping an eye, looking ahead, looking forward on that compliance landscape, right? And you get notifications all the time that we're saying, hey, there's a meeting this month regarding, you know, this new standard that may come into a five, three years from now. So, you get plenty of that information. The other thing is, of course, being ready, right? Being ready for audits. Ideally, if an audit is announced, you're going to have maybe eight to 10 months, you know, or six months at the worst to be ready for that audit date. If you run your compliance program correctly, an audit can happen tomorrow and it shouldn't matter because you have everything up to date, all of your compliance evidence is up to date. And they gave you usually an audit period of like maybe 18 months to a year, meaning from this day looking back, right? So, ideally, you should have had that all that ready. And so, a good compliance program, you'll be updating everything basically from a day-to-day basis. So, an audit could, you could have a spot check today and you'll be fully compliant if you did your job right. The other interesting thing as well is that there's a form called the reliability standards audit worksheet. So, in our soft. So, whenever you have an audit, usually on a standard, a standard has a whole bunch of requirements. In an audit, they may audit you with just one or two of those requirements in that standard. So, it's not, it's not going to be the whole standard either, right? So, this R-soft basically tells you the requirement, right? It tells you, it asks you to explain your narrative of how you comply. So, you explain yourself.  One thing to keep in mind though is just that, hey, as standards change, right, within a three-year period, you have a standard that changes version. Well, it's up to you to keep up with that version up to that date. Or if you update a procedure, you made a significant change, well, that piece of evidence now has to make sure, you got to make sure that evidence, you know, you show both versions and they fall within that compliance period. Not a hard thing to do if you're spending your eight hours, 12 hours a month looking, just looking at compliance, right? So, all right, next slide, please. So, again, there's different things to make sure, right? So, you have day one compliance documentation. That's when you're finally, you know, you've done all this work in preparation to the day that you energize that device, right? And you're in service. You want to make sure that on day one, you've got everything ready to go, right, documentation and scope. So, you have your contractor, you have all that information, that's the facility ratings, the relay testing, and all of that evidence that goes along with that, right? Interconnection requirements, everything's your operating procedures. Also, for example, all of your personnel training. It's another thing that's important. We'll talk about that a little bit more later. But having your personnel onsite that are going to be operating, that, for example, are going to be receiving calls from the TOP or the BA, well, they need to be trained, right, on how to properly communicate with your TOPs and BA. So, but that training is simple as another example, right? The other aspect here is include documented evidence, right? So, things that are, that see, things like voltage right through and frequency right through capabilities, that's PRC 024. And all these standards, right? For example, all of these you get from the OEM or from the equipment manufacturer, and you also get that from, it's verified through the contractor that did the relay testing. The settings, for example, engineering firm as well. So, all of that, you know, comes together, and before you energize it, they have to test all this. And once they do energize it, they got to go back and verify it. So, all this documentation, right, is an important thing to keep. So, that's one thing that you need to make sure you get on your contractors to have this documentation with them and they present it correctly.  And then they stay around for available for a good 30 days or two months until you get all this compliance evidence and spread away and properly document that is sorted, right? One of the things I've seen has been really, really bad is that we've had a couple of clients at some point that they had over 340 pages of prints, test records and all that, all scanned, you know, at one long PDF. And some of those documents were not OCR, meaning that they're not Optical Character Recognitions. You cannot do a control F or a control fine. And they weren't even properly, like, even on the margins, right? They were really bad scans. So, when you show that to your auditor, you know, it's going to present problems. So, again, not only must you ask for the right evidence, but the evidence has to be in the right format, right? Because they're really nice, legible, nice, nice files that is a property set up because that whole sticking all 300 pages into a scanner and then here you go as a PDF, it's the absolute worst way to do this. And then, of course, the interconnection study satisfies model requirements, right? So, again, that Mod2032, your interconnection requirements, right, on that agreement, before you connect anyway, you're going to have to have all these studies done. So, and then your interconnection agreement, right, the TOP, the RTO, the BA, all of those have those requirements that you have to adhere to. And for the most part, right, a lot of them also make sure that you're going to have that as evidence. So, make sure you have that properly carried out, which you're going to do anyway, right? So just, and most of them are already compliant. So just make sure you store that as well as that. Again, manufacturer data required. So again, they, like I said earlier, make sure you hold them accountable for all this compliance evidence. Like I said earlier, make sure that that's part of the contract. And then the other thing is negotiate a post commission timeframe, right? So you got to make sure that they are available for some time afterwards to make sure that they correct any issues, whether it's with the equipment or the documentation, because that is important evidence you need. You know, we have so many examples of, like only ever disappearing and offering no help to their client. Want to say, you know, commission that had given them all the evidence. So controls, right? A lot of that, it's just a matter of making sure everything in the compliance program. So when you set up your compliance program, there's a lot of different ways to capture all that evidence. Your procedures are controls. Training, of course, is important to make sure that you're within the compliance program. And then, of course, you want to test, right? That compliance evidence that's being captured correctly. Your procedures, for example, you're going to have either an operation procedure, it tends to be pretty short, or you have a training reference that explains more of a background as to why, right? What you don't ever want to do is you don't ever want to grab like the NERC standard and then copy it verbatim as your procedure. That is the worst thing you can do. And I see entities do that. And that usually gets the otters rolling their eyes and, okay, show me your evidence now and see what you've got. And usually they find a problem with that. Just show me your procedure, how you really do it. And then maybe as an appendage, you can show, hey, this procedure is written to comply with this evidence here, right? So training, I said that already, right? So consistent training. So your personnel, whether it's your high level leadership, all the way down to your mechanics or anyone doing the control center of these sites, training is important to make sure they understand what your compliance burden is, what your responsibilities are, and whether it's cybersecurity, whether it's communications. Very important. There are certain requirements specific to communications. They have to do it with talking with control centers. So that, they have to be trained on a certified before they actually serve. And there's many ways to live with this training, right? So for us, whether it's computer-based, there's a webinar, micro-learning, so we have all this different availability, right? One of the things that we do is we also do audit prep. We get your subject matter experts ready for these audits. And the most important thing at the bottom is new hire, right? NERC 101. So make sure that they understand whether you're about to be a facility, it's about to become NERC compliant or you have a new person, new personnel that you want to get them ready. This training is really important. And of course, I have to retain evidence for that. And look at that. I think we made it to the end, just under 30 minutes. But again, we really, really condensed this. This presentation was a little longer when I had the other webinar. But if you have any questions, feel free to reach out to us. Thank you. And thank you for the show.