Good morning guys from Tucson. I'm at Arizona Transmission and Engine Exchange. That's a tongue twister. Today we're going to see a teardown of the new MT643 with Christian and Richard. And I don't want to sound the alarm. Nobody needs to replace their AT545 unless you have an issue with it. So I'm only doing this because there's an issue with my transmission. There's a lot of brass in the pan and that is no good. Here is the transmission I got. This is a used MT643 and I have no idea what the condition inside of it is. And that's why these guys are here to tear it apart. So, let it rip. That's a torque converter. I'm going to take it apart and see how it looks inside. First the demonstration is when we slide the torque converter on, it's a slide on and off torque converter. We slide this off for inspections as we go through the model. So the main portion we're going to look at is here is this hub right here. So this hub is worn down and will need replaced. In the bushing area, if you feel the diameter is worn out, it's dipped down. Yep. Yeah, I can feel that. Very slightly. But that is beyond specs, so it will have to be replaced this hub. Okay. And so if I was driving this and I had an issue and this was the problem, what would that be called? It would be a front end leak, which would start to leak out the front end and this rides on the pump bushing area. Interesting. Okay. So this splines into your oil pump. This is your pump bushing area, which gets worn out under just high mileage transmission wear. Okay. Would that be a slipping part of something, or is that further inside? That's further in. Clutches are further in. Okay, perfect. That's one point there. Right here we're opening the torque converter for inspection of components. Lock up clutch. And this oil looks pretty dark, right? Is that correct? Oil is burnt. Burnt? Yes. It is dark and burnt, which means it's going to have clutch material, probably bad. And so is there any way to know without taking one apart if one is good or bad? I mean, if someone had one that was messed up and they just did a fresh oil change, would you be able to tell that without driving it that it was in bad shape on the inside? No. Interesting. So opening up here is the lock up clutch. This is your lock up clutch, which is worn out. It is. You can tell it right here. Oh, yes. Yeah, there is some damage right there. Yeah. It's coming off. Yep. Hardly no material left. And would this be metallic material that would stick to the magnet? Yes. Okay. And metallic, if you look at the oil, the metallic portion of it. This is all friction material coming off there. Okay. So as we go through the hard parts, which is the backing plate, lock up piston, which the lock up piston looks like it's cracked, which will need replaced. If you see this line right here. Oh, yes. They cracked through there, and then they bleed off pressure. Oh, yeah, I see that. I see that, yes. And that's your lock up piston. Okay. For the lock up torque converter. All the way around, all the way through here. Okay. So that whole part is going to be... So that piston will get replaced also. Okay. And it needs pressure through there. Now your clutch is not applying correctly. Okay.  That's all the material that is coming out of that. Oh, yeah. Look at that. Wow. It looked like my shake this morning. Shit. Oil pump and the forward housing is going to come out all together as one assembly. We're going to set it to the side for inspection. Assembly, all complete. We will disassemble, disassembly at the later point in time. We're going to open the oil pump. Okay. So we'll remove all the bolts and open it up for inspection. All this black right here is clutch material. All right. So definitely some wear and tear in this one. Yes. What are we looking at? Okay. So when we open the oil pump, we're inspecting the pump for excessive wear on the housing. This is the front support. Looks very good condition. We will pull the gears out of here and set them aside, inspect down in here, and look for edges on the side right here. Now looking at it, it looks very good condition. Good. The oil pump is very good condition. There's no sharp lips, no excessive wear, metal gone from anything. Okay. So that's a good thing, right? Yeah. That's probably an expensive part too. Yes, it is. Good. Every time you say, oh, that's going to need to be replaced, I hear a chaching in my head. Yeah. We're going to check here the seal ring areas on the forward clutch housing. Down here is the seal ring areas and the bearing area here to see if it's got any damage. Looks very well. The input shaft gets a little bit of wear just from bushing, but it's not damaged. Awesome. A little disassembly of the forward clutches for inspection of the forward clutches that get replaced anyhow. But we're checking all major hard parts. What we're checking here is we're going to be replacing all the clutches so they get replaced anyhow, but you can see that the darkness of the steels are discolored that they're burning. Okay. And that would be the burning smell when you're operating it. Yes. Okay. And some of these look smooth. Are they totally smooth because they're worn down? Yes. When they get smooth, they're supposed to have high ridges. When they get worn down, then they'll be almost flush. Like that one, huh? Yeah. Well, this is the metal. Oh, that's a steel. Oh, okay. This is your friction, which people consider a clutch, but it's the friction. Okay. Forward and forward clutches use the same diameter and same style friction. That's what we go for. Awesome. Here we go. And just for reference, this is my current AT545, and this is the larger 643. Slightly, slightly larger internals, as you can see. Inside here is a lot of metal from the clutch wear material. Down here, all the clutch material gets built up. Oh, wow. This is all your friction material that is built up in the housing. So that's an indication of wear? Yes. Okay. Yeah. High mileage, just normal wear and tear on the transmission under normal circumstances. If your linkage shifter linkage gets out of adjustment, which is very important because if it gets out of adjustment, worn down, it will burn your clutches up very fast because it's not in the detent all the way. Gotcha. We're done with disassembly of that. Okay. The piston will come out later for inspection and rebuild, but we'll continue on with our disassembly. Okay. So we're forwarding on to taking out the forward, fourth clutch housing, and this is third clutch snap ring. So we'll set that to size. We're going to go in here, grab fourth clutch housing, slide it out slowly because of very sharp objects. One of the important things we're going to look at is the main housing here. The main housing looks very good condition. Good. Otherwise, it would chatter and wear between there. And what happens is third clutch will not slide smoothly and grab and slip and cause more damage and clutch failure. Another thing here is another seal ring area. Very good condition. So this fourth clutch housing looks good so far. We're going to be removing frictions out of it internally there. Disassembling fourth clutch, fourth clutch vacuum plate, steels, frictions. Same with these. These are not completely burnt, but do have high mileage wear and tear on them. This one here, you can see it's cooked. I mean cooked, it's just starting to flake apart. Mm-hmm. That one's gone, see? Oh, yeah. That one's really smooth. See, that's a lot of times you could look at all these and it all looks good. You flip them over, they're coming apart or gone. It looks like sandpaper right now. Yes. Mm-hmm. Which all these get replaced on rebuilt any house. Third clutch, frictions and steel, they are burnt down metal to metal. Wow. Nothing left of the friction. So can you reuse the steel? No. No. So all of them need to get replaced? Yes, even the backing plate. Wow. The backing plate is the thick component. Look at this side, it's smooth. This side, it is completely worn out like a clutch disc. You know, it looks like disc brakes. Yes. You can see the grease. Yes, that's how exactly it is. Look at that clutch. That third clutch, completely gone. This transmission was slipping. So a third gear was slipping? Yes. The patient of you're having clutch or metal damaged internally when you take out the speedometer sensor and it has tons of metal on it. Wow. So if someone, so... If someone is replacing their speedometer sensor and seeing all this metal internally then they have wear and tear going on. They need to get the transmission repaired before they end up stranded somewhere in the middle of nowhere after their travelers. So would it be advisable to, if you were inspecting a new vehicle or, sorry, a used vehicle or a used transmission, if you pop this thing off and it's dirty, is that a good indication without stripping this thing down that it's screwed? Yes. Okay, good to know. We're taking the dust cover off and then we're going to pop the output seal off. At this point in time it just makes it easier to remove the seal. The output shaft, output bearing internally so far looks everything... We are removing the governor and we're going to be pulling the governor out. And that's the thing that regulates the RPMs? This is what makes your transmission upshift. If a person has a problem with an AT545 or an MT643 that quits upshifting or starts to upshift late in problems, this could be one of your main problems here. The gear gets worn out, the surface here gets worn out. Sometimes when there's tons of metal going on, then inside the bore gets worn out. And this is governor pressure as the weights move back and forth makes the valve body shift make sure transmission upshift. If the gear strips out completely, it will not upshift anymore. So we pulled the rear cover off. This is your output housing. This is your first clutch piston. And you just mainly inspect it, make sure there's no major damage going on. Sometimes the retainer will crack in half. This will come apart. The springs will fly in there and then you'll get metal throughout the whole back end which would be another major failure of the transmission. Okay, right here we're going to just slide the spacer. This is your speedometer gear with electronic. And your governor gear right here, they just come off the output shaft. We're going to just take those off, set them aside. Then we'll be pulling the planetary section out with the first clutch. Right here is your first clutch assembly. It means the clutches were slipping also. Dark means high heat. High temperature, yes. These are all frictions that get replaced on rebuild anyhow just as a demonstration of the clutch first clutch. So there's definitely some whatever vehicle came off of there's some intense heat in first gear. Well, the whole transmission is showing high mileage wear and tear. So it's not something specific that just burnt up a specific clutch. The whole transmission has high mileage wear and tear on it. Okay, so the housing has some wear on the back area. It gets stuck in there because it's just wear and tear. This is your first clutch backing plate. Still reusable, it's not worn down. So it's a good. Now we're pulling out the planetary section of the transmission. On this assembly, the main components that we'll be looking at for damage, hard part damage, is the spline area here which is starting to show a little bit of wear and tear. But still okay. Thrust washers good. Here's are still good. If you look down in here, it still looks good there. The planetary gears and thrust washers internally. This is your front planetary. Another important area here, this is the main shaft bushing area. Even though it shows a little bit of wear, the main shaft is still good condition to reuse. It will be polished up here. If it was worn down beyond specs, then it would get replaced. See the darkness that's starting to go on? This is just because the high mileage person doesn't replace their fluid when it should be. So the fluid gets very dark and stains it. Yes, it goes through the whole transmission. Interesting. That's your center planetary. It gets inspected for thrust washer play. If any thrust washers are missing, then this planetary gets replaced. Old times the planetary would get rebuilt, but due to cost nowadays it just gets replaced as an assembly. This other area here is bushing area for the bottom main shaft. It's good condition. And then one more area right here where the pilot bearing goes. That is good condition. The gear is good. This assembly is good. The planetary housing here that gets all your planetaries assembled. Everything looks good condition. The splines are good. They're not worn out internally. Other times they get worn like that. All this section and we check right here. The thrust washers, any missing? The thrust washers are there. If the gear slides up and down excessively, that means your thrust washer is gone either on the top or the bottom. But this is still good condition. We're going to pull out the sun shaft, which the main shaft goes through for inspection on other bushing areas. The thrust washer, center sun gear. Right here is bushing right areas. And these are the inside bushings for the sun shaft, which go on the main shaft. Everything looks in very good condition. Now to complete the disassembly, we're going to have to take the valve body off. So we've got to remove the oil pan to remove the complete valve body to get to the center support bolt and finish disassembly of the transmission MT643. So here we are, we removed the oil pan for further inspection. We know we have lots of clutch wear already.  Sooner than later. So we're here to the valve body and we're going to be pulling the whole body section off so we can get to the center support bolt and finish disassembly. The valve body, this is your whole valve body section here, this is your channel plate and as you can see all the black material which is clutch material just transmission high mileage and worn out, ready for rebuild. Alright. So is this thing fully disassembled then? No. Okay, one more thing. We're going to take the center support bolt which is here. Sometimes the center support bolt will snap, break because of just normal wear and tear or driver abuse depending on if they're going from reverse to dry without stopping the vehicle. The vehicle is very heavy. So when it impacts it, it moves the center support and abruptly breaks the bolt. If it breaks the bolt then the center support is not held to the main housing firmly and the pressure is bleed off internally and will burn your fourth and forward clutches up you know, immediately. It won't last long enough. Oh, okay. So that one came out just fine. Yeah. Cool. The center support didn't fall. The center support pistons, all the seals get replaced. The center support pistons is the tops, these pins break and doesn't hold the retainer down anymore so this would have to get replaced and this one is in good condition. This is washed up. Here is a sealing area right here where your sealings go. This is where your fourth clutch housing goes on to here and it gets applied through these pressure ports here. So, we got one more clutch to take out. The second clutch, Allison transmission. The greatest transmission in the world. You like it? Yes. This is 1994. I got the opportunity to go to work at Phoenix, Arizona training. First started out mopping the floor and turning the transmissions apart and I got to see the first transmissions being designed at Allison and Indianapolis. They sent me to school there and anybody wants to do it. Cool. Let's see how wonderful that one looks. Looks like grip tape on a skateboard. Totally smooth, huh? About to cut off all the teeth, almost. Wow. Great disassembly. MT643. Alright. Anything else? Now, inside the main case, we're going to go look right here and then down in here where the clutch is right, there's very little bit of wear you can see on the edge here. Down in there were first clutches. Many times the snap ring, the housing and then the snap ring will pop out. But everything, even though the transmission shows high mileage wear and tear on all the clutches, the main housing is in very good condition internally. And you just see all the clutch material laying in there. What are these scuff marks right here? Right here is machining from the factory. Okay. When they mold the case, they have to machine this out and get the burrs off for the components to go in. Okay. Interesting. So, we're looking good. Yes. Awesome. The case is good. Okay. So, now having taken this thing apart, what are you going to order? What's the diagnosis on this thing? So, this is typically, like every transmission that we rebuild gets all the clutches replaced. If the steels are bad, if they're good, we can reuse steels if they're in great condition. If they're bad, we replace them.  Alright. We look brand new. Cool. So, if anybody's in the Tucson, Phoenix, Arizona area, how did they get in touch with you to rebuild? Well, we're located at Arizona Transmission and Engine Exchange, and our phone number is<PHONE_NUMBER>. We have very, very competitive pricing, and we guarantee our transmission is 100% warranty. Awesome. For two years, right? Yes. Two-year warranty. On our automotive, we do automotive also. Okay. Automotive analysis. Awesome. Unlimited mileage. How long will... We're going to come back for the rebuild part. Will that take a day, two days? On a typical build, replacing all the bushings, clutches, pistons, seals, about a day and a half, two days tops, just depending on the washing of the components. Okay. Awesome. There we go. Alright, guys. We're going to check back in with Richard next week. And Christian. And Christian. Thank you so much, guys, for explaining what most people don't get to see when they drop off their transmission and, you know, what goes on in the inner workings of this thing. So thank you very much. Thank you. Here we are with the... going to be the overhaul kit for the MT643 Allison transmission. All brand new, genuine Allison components. Got the seal and gasket kit here with all your necessary components. Seals, gaskets, output shaft seals, pump seals, for converter, for converter shims, your stator springs here, and all, like I said, genuine Allison parts here for a complete MT643 rebuild. And so, right here is your fork and forward clutches. Wow. That's a big difference. And this is a big difference if you look at the ones that we took out of the transmission compared to the brand new. I mean, even the color alone is just completely different. The teeth look a lot newer and there's no smooth parts. Yeah, even I've seen transmissions come in that the linkage might have got out of adjustment and burned up a specific clutch. We've disassembled them before and the clutches looked fairly, fairly good condition to as new, which that's how they should be under normal fluid interval changes and so forth. There's the other clutch pack completed as you can see again. The design, color, the internal teeth, great condition. And that's why we give two-year warranty unlimited mileage. All right, awesome. And these are some of the parts that we'll be assembling when we rebuild the transmission. Next week. Yes. All right, cool. Well, I'll see you then. See you then. All right.


Hello and thank you for joining us. Today we're going to talk about improving access to hidden collections, a project that we've undertaken here at Washington University. My name is Bill Winston. I'm a GIS and data visualization analyst on the data services team in Washington University's Olin Library. Washington University is located in St. Louis, Missouri and is home to approximately 17,000 enrolled students and more than 4,000 faculty. The university was founded in 1853 and stands among the world's leaders in teaching, research, patient care and service to society. We are committed to learning and exploration, to discovery and impact. The project that we'll discuss today involves a collaboration between members of Olin Library's special collections, exhibitions and data services teams and Robert Morgan, faculty from the Department of Fine Arts. In our briefing, we will introduce John Izzelle and give an overview of his work and the materials in our collection that he has shared with us. We'll introduce the photogrammetry process and our specific processing workflow that we use to capture each model and create a digital version for access and preservation. We will also provide a preview of the outcome and share a bit about our plans for the future. My name is Robert Mark Morgan. I'm a scenic designer and I also teach scenic design here at Washington University in St. Louis in the Performing Arts Department. John Izzelle is someone I have kind of grown up as a designer knowing. I was a student at Webst University way back when in the late 80s, early 90s and John had been designing there for a number of years even by that point. In fact, there was one show production of Hey Fever by Noel Coward that I served backstage as a production assistant on and John did the set. In fact, I think you all have the model of that set and it's beautiful. Every piece of work he ever did was beautiful. So from afar, I've always been a long admirer and I'm proud to kind of call myself a colleague. John has done over 350 productions in his career. He's done theater work, New York and regional theater work. He's done some television work. He is well respected, well known as not just a designer but as an educator. He taught for a number of years at the University of Wisconsin-Madison until about the mid-80s and then after that he transitioned over to the University of Missouri-Kansas City just across the state from us here in St. Louis. And he ran the graduate program and design there at the University of Missouri-Kansas City. When it comes to using these models as a design tool, an instructional tool for students, it's really integral. When you think of every stage set, it is a sculptural piece that is inhabited by a live actor. Every model you'll see of John's of course has actors, cutout actors in the model. It's a sculpture around which is the playground of sorts for this play. And so for me and my design work and my teaching of students, they have to understand three dimensions from the beginning. I recruit a lot of architecture students for example from the Sam Fox School of Design and Visual Arts to take my class and essentially think about architecture for the stage. Think about a space where anything can happen. Rooms can float, there can be water on stage and it frees them up a little bit to be more I guess creative within the confines of that stage. That's the beauty I think of theater is we walk into a space to witness magic, to witness something on stage that cannot happen, not always, but cannot happen in real life. So John's models are really important instructional tools to showing not just how he sculpturally approached that play, and we'll show an example of death of a salesman in a moment, but how it is lit. Students immediately, and I think it's wonderful to see, get out their cell phone lights and begin lighting John's models because it doesn't really take life until it's lit, until there's lighting. So when it comes to your digitization of these models for my students, it becomes a really valuable tool for them to just be able to at a moment's notice look at something. At the moment, it's not a book, these are not books, these models, 250 of them, take up a great deal of space. And so it's not like they can go to the library and pull a book from the shelf and open it up and get the information they need or even necessarily click on a picture. What they're doing here is they're clicking on a three-dimensional model that they can look at from all sides. When it comes to a model like Death of a Salesman, two-thirds of the audience is seeing it from the sides. So that's important for them to understand, I'm designing not just for the good seats, the center seats, I need as a designer to make sure that the people on the sides get the same show. And so for me to use these as an example for them, in this case of a thrust theater design that's well done by a master of the craft is really great. I can send them a link and say, hey, I noticed you were working on a show that maybe as a thrust environment, you want to maybe look at John's model for Death of a Salesman as simply a reference point, an ideal for which they should want to reach perhaps. So it becomes a really useful, quick 21st century way for them to look at a model without having to physically go anywhere. So I'm Skylosert. I am the curator of the D.B. Dowd Modern Graphic History Library. I've been here for 15 years working in special collections. And so special collections basically is a primary source research facility that includes materials dating back from ancient times through the 21st century. And special collections has five units. They include the D.B. Dowd Modern Graphic History Library, modern literature, rare books, university archives, local history, and the film and media archive. So Johnny Zell has told me that he feels a really strong connection to Washington University and really felt strongly about having his collection and his life's work be preserved here and made available to the students. Currently, the collection is housed at West Campus, which is about a mile and a half, two miles west of main campus here. So there is a barrier of access because students and researchers have to travel to West Campus to see it. And we also have a hard time serving the materials because they are fragile, they are large objects, and you can't bring out as many as you'd like to see all the time. Another obstacle is that people don't know what they want to see. So in order to serve the materials, we can have them look at the scanned objects and then narrow down what they want to see if they want to come in and look at the real objects. For this project, we're using a process called photogrammetry to convert our 2D photographs into a 3D model. In this case, an object is photographed from multiple angles and those photographs are then stitched together to create a 3D model. The principle in use here is called motion parallax. This visual trick uses the apparent displacement of an object against the background to produce a 3D model and an estimation of the position of common points in the photographs. These common points are then stitched together to produce a digital model that we can then use to represent the original object. In order to perform the photogrammetry process, we use a software called Metashape. First of all, we bring in the set of photos that we've captured in our camera and load those into the Metashape software. Here's just an example of one of the oblique views of that stage scene. The software then examines each one of the photos and identifies tie points. These are points that it can find in common with multiple photographs so that it can identify that as a fixed position and use it to model the 3D object that it's trying to build. I'm going to close this photo and show you the sparse cloud that is initially created by the software. This is done with a cursory examination of the photos and the tie points that have been located on each one of the photographs. During the modeling process, the software identifies the camera position where each of the photos were taken from. In this case, our model is spun around and although it appears that the camera is rotating around the stage, in fact, the model is the thing that's being rotated. In this case, we generated four rows of photos at four different heights, capturing photographs completely surrounding the model. As I said, the software processes a sparse cloud. I'm going to turn these photos back off. Processes a relatively sparse cloud, identifies the camera locations, and then comes back for another round of more aggressive processing. In this case, it builds a dense point cloud. It looks almost solid, but in this case, it's made up of a bunch of points that are just tightly packed together. The coloring of each one of these points has come from the individual photographs. As it extracts a photo or as it extracts a tie point, it also extracts the color from the images. This generates a point cloud, but now in order to turn this into a digital model, we need to connect those points together. We do this by building a mesh. I'll show you the wireframe view now. What the software has done, and I'm going to zoom in very closely here, the software has captured or has taken all of the points and joined connecting points together or neighboring points together into a triangulated, irregular network. This basically builds a set of triangular facets with nodes, edges, and faces. This is a common methodology to represent a surface with changing elevation. As we have developed this mesh, we're now able to convert that into the final digital model. We do that by applying a texture to the mesh that's extracted from the photographs that then provides some color and more natural color to the final digital product. This is an interim model that isn't as detailed as some of our later models have been, but it points out some of the imperfections that might arise in this process. Areas that are very thin, for instance, the figurines that we see on the front of the stage here, these are basically flat cutouts that have been stood up on the front of the stage. The modeling software does not represent those very well without some more aggressive post-processing. Another thing is features that are underneath other features. These are sometimes not well represented because of the lack of a high number of photographs that represent them. As you can see, it's a little bit messy here in the interior of the model, particularly as we work backwards in the stage. The other thing I'll point out is this overhanging area here on the stage and the backdrop where we can see some missing points, basically. Those are made of a translucent material, and so capturing that type of material through photogrammetry is sometimes difficult and requires additional processing in order to adequately represent those types of features. But at this point we're done with the processing in the MetaShape tool, and we're set to export this out into a final three-dimensional model that we can then post up into a website or other source for sharing. I'm Jennifer Moore. I'm the head of data services at Washington University in St. Louis, and I'm one of the organizers of the Community Standards for 3D Data Preservation, or CS3DP. So WashU cares about this collection because Johnny Zell is an important collection that the libraries hold. It's one-of-a-kind. It's really cool, and people are always impressed when they get the opportunity to see it. The problem is it's also fragile, and so the opportunity to see it isn't great. Because it's fragile, we want to be good stewards and make sure that it has a long life of its own, the original object. But also the idea that it's a physical object and people can't necessarily come and see it anyway, even if they want to. So the idea of creating a digital surrogate is to provide the access that we as stewards want users to have while also protecting the original object, which is so special. However, creating a digital object is not enough. We have to take intentional measures to make sure that a digital object is preserved long-term. The way that we are approaching preservation now is a lot different from when we started. When we began working with 3D data, we didn't really know how to preserve it, and the existing digital preservation methods had gaps that 3D data didn't fit in too well. So the work we did with the community standards for 3D data preservation helped us work with others to figure out what the important pieces were when it comes to preserving 3D data. So one of the things that we learned through the CS3DP was about preservation intervention points. And you can read more about this in the best practices chapter of the CS3DP book. It's an opportunity for the creator to stop at critical moments in the model creation to make sure they're taking action for preservation. One of the first choices we made when we started the ASL project was to use photogrammetry instead of a 3D scanner. With a 3D model created on a scanner, you have less opportunity for stopping because a lot of the work is happening in an automated way by the software that controls the scanner. With photogrammetry, we had a lot of choices we could make ourselves about lighting, about cameras, about how many images we want to use, and how we want to process the model, specifically how we want to process the model. And every time we make a choice like that, we can make a choice about preservation. We might take many photos, the actual model that you see in a viewer might not have every bit of the model included, but the choices that we made along the way will make sure that that model could be recreated because we have all the parts and documentation and parameters that we used so that we could do it again. And photogrammetry is very cooperative in letting you make your own choices. Our strategy for long-term storage, long-term access, preservation in general, is threefold. First of all, we have our access copies that require no one to download anything. We're building a viewer for our purposes using the Smithsonian 3D team's Voyager platform that they developed, and we're going to put lightweight models into that so that anybody can come to the site, interact with the model, and walk away from it. They don't have to do anything else again if they don't want to, but they can still experience the model. In our digital data repository, we're also going to put data objects to make sure that copies of the data at different stages are accessible so that might be the images, the point cloud, for example, and those data will also be preserved using Archivematica, which is a separate tool that we will generate archival packages from our data repository and ingest into Archivematica to collect premise metadata to collect the data itself and any documentation in our curation process so that we have a relatively thorough chain of custody described in Archivematica. We also are keeping copies of these objects on our local servers, which are backed up to Tape Robot, and that includes our documentation, both the process information and contextual information. Some people call this paradata. It's the data that acts like a cookbook so that you could recreate the project. I truly believe that the definition of space has changed from when John built this to meaning virtual space and that in the not too distant future, we will have opportunities for audiences to click and show up in quotes in a virtual space to see a production thousands of people could across the world. The Royal Shakespeare Company did a Midsummer Night's Dream. They just called it Dream. I'll send you a link to the video. I think you did. Did I? Yeah. So they were kind of pioneering a little bit of this idea that what if our show was in this virtual space and our audience was anywhere in the world and they just had to show up at the right time, could be 2 a.m. wherever you are, but you could show up and see a piece and be moved by it. That's really exciting for me and to see these models be digitized is kind of like, well, maybe John's work can live in perpetuity. You know, this Macbeth could happen again. Yeah. In 2025 or something. Anyway.


In this talk, we will explain the concept of disk image builder, we will see some demo talk will be presented by Paul Belanger, who is the full time contributor to OpenStack infrastructure working in RedHack. Thanks. So introduction, hopefully you're here, you want to learn about image builds and particularly the way that we do it in OpenStack Infra and the tooling that we use. Before I start, I want to say thanks to Clint, he really primed the room for me here, he did a lot of good talk about Zool and CI in general. Didn't necessarily dive into the node pool side of things, this talk really relates to it. So while we did not coordinate beforehand, I could not ask for a better person to go in front of me. So who am I? So I'm a senior software developer at RedHack, I am ord under the OpenStack infrastructure engineering team. What does that really mean? Not too much, because I don't actually do too much internal facing things. I have the luxury of working 100% upstream in the OpenStack project, specifically the project infrastructure team. Where I am one of the, I think we're up to about 12 infrared root people, the people that have the keys to all the systems. However, anybody in this room can actually manage all of our systems, all of our systems are open and public, managed via Garrett and so on and so forth. I'm just on part of the team that has to sometimes go and reboot or restart Garrett because it's running out of memory or something. So what we're going to be talking about, we're going to talk a little bit about OpenStack infer in general, just to set you up on basically why we're choosing this image builder and Ansible. And we're going to talk a little bit about node pool because node pool is the scheduler or the piece of software that we use to front end disk image builder. So what is OpenStack infrastructure? Like I said, we're basically the team that manages all the systems behind the OpenStack project. It's done everything from the development, the testing and the collaboration tools. So for example, I'm sure people know of JJB for Jenkins Job Builder. If you're using that, we maintain that as part of our code base and day to day because we use it extensively. Basically, the mandate is everything is open, obviously not passwords or anything like that. Those are stored in a hybrid database, but any of our infrastructure is open with basically the purpose of having other people, other teams, other companies hopefully embrace that way of thinking and even start using our tooling. Like Clint was saying before, Zool and node pool are perfect use cases which grew organically out of the need to drive OpenStack as a project. And the super cool thing is that downstream contributors and downstream companies are embracing those technologies. So more specifically how it relates, this really is OpenStack infrastructure. So you can kind of see we have our Garrett on the left and that's the interface for the developers. These are our fleet of Git servers that we run because Garrett isn't powerful enough to do all of the load balancing of Git requests that we need on our whole right hand side, which is a testing infrastructure. So Zool, we're not going to really talk about that today. We have the concept of a merger and a launcher. So these launch the jobs, these merge the code for the jobs. These are all of our public clouds. The part we're going to talk about today is really this little box in here, which is node pool. Node pool is the piece of software that launches the VM on the cloud. And because we deal with so many clouds, we deal with the 12, I think we're up to 12 clouds now, 12 regions. We can't trust the images in the clouds because they're not uniform across all of our clouds. We started out that way, but then we started having to hack up our public deployment scripts to say, if rack space don't install system D because that image already had system D or something like that. So yeah, so what is node pool? So node pool is a system for launching single unit tests on demand of image builds with cache data. And basically node pool started as a way for us to attach a dynamic slave into Jenkins using the Jenkins API. And then once Jenkins accepted that slave, we would then connect into that slave remotely via SSH to run our tests. And then same thing. When it was done, results come back, node pool would tear it down or will tear it down and start the processes again. So basically there's three components of node pool today. There's the scheduler. I'm not really going to go into that. I just wanted to outline that's the loop that's actually going out and doing all the calculations of where demand lives. We have a concept of a launcher. This technically is still being designed now as part of our Zulu v3 effort. But this will be massively scaled out. Currently we really only support open-stack clouds. But moving forward, first-class citizens were aiming for Google, Amazon, Kubernetes, like anything, bare metal. All of that stuff falls under the new mission statement, I guess, of Zulu node pool. And finally is node pool builder. And that's really what we're going to talk about today. And the reason I put node pool builder in there, it's basically a very thin front end to disk image builder, how we use it. But it gives us the scalability of distributing builds. So node pool gives us the ability to schedule our distributed builds across our infrastructure rather than having a single machine create all of our image. Because there's not enough time in a day to build all of our images on one machine. So why? So why do we do this? So basically it's an ownership of an image build. This is important for us because images get deleted on clouds and there's no backup. And if we don't have that image, we can't test it and we can't reproduce it. It removes the dependency on other build systems. At the scale we deal with, we can't afford an outage of an upstream image being deleted or their build system down. It gets into immutable infrastructure. That's really what disk image builder was created for, the concept of an immutable build. We use it to a degree but other projects in OpenStack, which I'll reference quickly, really embrace disk image builder in that aspect. Obviously security, yay, nay, maybe. But basically it's a potential tool to do a single, a potential tool for single builds for both bare metal images, QCAL images for cloud images and potentially container images. And like I said, it's reproducibility. So very quickly, no pool, some image data stats. We've got 12 regions. We've got six images that we built today. Three Ubuntu, two CentOS, Fedora 25, and a Debian Jesse. These are the formats that we built. So QCAL 2, VHDs are needed for rack space. And then we have raw images because sometimes clouds just need a raw image for their glance back end. And then basically two dedicated builders, no pool builders. And this no pool builder concept is new for us. It's part of the Zool V3 effort and it really allows us to scale this up to, we could have six image builders or 100 image builders or whatever it is. It truly is distributed building now. Okay, so getting into the heart of it now that I've spent nine minutes on that. What is disk image builder? So disk image builder is basically a collection of elements. And these elements at the heart of it are basically written in bash. Now, love it or hate it, that's basically how we have it today. The concept of disk image builder though is it's really a phased approach of building your image. So you have the concept of these phases here. So a root, an extra data, pre-install and install, post-install, block device finalize and clean up. Because we're dealing with CH roots, depending on the phase, it's something that happens outside of the image and something that happens inside the image. The majority of these stages happen inside the image. Two that happen on the outside are obviously the root and the cleanup. And I'll explain a little bit more the way that I've chosen to kind of implement Ansible in the concept here. So some examples that are using disk image builder that may not be using Ansible. So triple O, the triple O project in OpenStack is OpenStack on OpenStack. That's the triple O. It's meant to deploy OpenStack using OpenStack. Initially started by HP, Red Hat and so on and so forth. Primarily now driven by Red Hat. This is the installer of choice for Red Hat. It is basically, again, all immutable infrastructure, image-based deployments, leverages disk image builder. Bifrost is a basically Ansible playbook that does bare metal provisioning, uses the ironic project in OpenStack. But the images that are used to boot the node the first time are generated by disk image builder. And they use Ansible to do the builds, but they don't have Ansible inside the container of the build. And then finally, us node pool for projects. So the OpenStack infra, we have our own elements that define how an image looks for testing. More specifically, in our elements, we actually run puppet inside of the image. This is for legacy reasons, because we are still a puppet shop. By moving to something like Ansible, hopefully, we can remove some of those dirty dependencies that are needed for your configuration management on the inside of an image, and it starts living on the outside. So basically, getting into the heart of it, what's needed to actually do this? So basically, I've wrote a very simple patch, and I call it the simple playbook element. It can't get any more simple than this. And what it does is it's really meant to launch Ansible playbook and define an inventory file to the CH root that has already been created in disk image builder. And the reason for this is it allows us to use a single command to build the image and then overlay our configuration atop of it. Another way to do it would be to create the image, save it, and then uncompress it into a directory and then have Ansible do it. So that, I mean, that's totally valuable, a total reasonable way to do it, but the amount of data and the churn that we're doing, this makes a little bit more sense just to do it all at the first stage. So what does your playbook look like? So very simple. This is a very simple playbook that I have wrote to build a Docker image for disk image builder. So I'm going to use disk image builder to create me a container so I can run disk image builder inside of it. Turtles all the way down, right? But the really cool thing about this is that if you're already an Ansible shop or I might have skipped over it. Does anybody not know what Ansible is? Hopefully everybody's used it, seen it. Okay, great. The benefit of something like this approach in my view is you're not pinning your management of an image to the tooling that's building the image, which means I can take this same playbook and in theory use the SSH connection of it to manage my actual production running node pool service. Today, we can't necessarily do that in OpenStack Infra because we have a different path that our puppet takes when we say if it's an image build, run this puppet manifest, but don't start any of the services in the image because you can't really do that and puppet gets a little bit crazy. So the whole idea is you have an existing playbook that you've written already for your production. Why not use that to create your immutable infrastructure at the same time? So yeah, so the connection route, like I was talking about, is Ansible connection CH route. Now very funny, there isn't really much documentation on this, so it might be kind of a surprise to anybody that you can actually do this in Ansible, but yes, you can. Monty Taylor was gracious enough to tell me about this neat little hack. I guess if you want to call it a hack, but it's not a hack. It actually works fantastic. Instead of SSH. Instead of SSH, yeah, it just puts it out to the existing mount that's been created by DiskImageBuilder and does its thing and then Ansible just stops working and then DiskImageBuilder does the cleanup and so on and so forth. Yes. Oh, I... Yes, I could, but I don't know how to do it. Yeah, unfortunately, I should have blown that up just a little bit more. I'm sorry. When I post these online, I'll make them bigger. I know it doesn't help the staff. Okay, so basically how do you run this command? Oh, I got five minutes, man. I thought I'd have not enough to talk about it. So basically, these are the syntax that we use, what we're setting up here. So these are... These did variables define the type of image that we want. So the top one is for Ubuntu, Tireball, of trustee. The key piece of information is this, the type. So the top one represents Tireball we're creating. The second one represents Kukau image. The third one also represents Kukau image, but it's a different flavor. But it's the same playbook. The beautiful thing is it's the same playbook. So if your playbook in Ansible is written smart enough, it's a single point of entry, right? Node pool builder. Yeah, so this is what it would look like in node pool. So to run this manually, this is the commands that you use. The cool thing about node pool builder is we can express it now in YAML format, a nice clean format, which opensack infra loves. Everything we do is YAML based because it's perfect for code reviews, right?  These elements are the elements that we want to install. So we want an Ubuntu minimal, which is a very basic Ubuntu image of VM things. It does some stuff with the kernels. Simple init is basically a replacement that we run in opensack infra for cloud init. And then some repos that we cache and so on and so forth. And then a fancy element that says some infra packages that we need that may not be in disk image builder. So from node pool's point of view, what does this look like? So we have a fancy command called dib image list. This is yesterday. These are all the images that we've created. So you can see we have... By default, we keep two images in the cloud, which gives us a primary and a fallback. If we upload a bad image, we can immediately roll back and not have to do a rebuild. We've got three going on here because we only delete the image once everything has been uploaded. And you can see on Debbie and Jesse, it's currently building. So like I said, the cool thing is this is totally distributed across multi-machines for image builds. It's backed by ZooKeeper. Yay or nay if you like ZooKeeper. But these are the actual images what they look like in the far end. So these are the images that we're uploading to the clouds. So you can see I couldn't fit it all in because I would have had to scroll like five pages because we have massive amount of images. So just here we got... I don't even know what that is. 20 images across all these clouds. So these are rack space clouds and regions. OVH, OSIG, you'll see multiple OSIGs because it's the same cloud, but different hardware profiles. Bluebox. And then for cloud, we have our own cloud and infrastructure now. Like the OpenStack infrastructure team is running a cloud on donated hardware. What else? So basically docker. So you could do this for docker. Really cool thing, docker import. So you're just padding the target ball so you don't import. And then you're running your privileged container thing. Kind of issue with this. I haven't cracked the metadata issue because in a docker file you express your entry points and firewall rules and so on. I'm not really a container-ish, but apparently there's a method to generate this JSON blob that represents this. I don't know how to do it yet. So if anybody knows how to do it, point me to the person. Thank you. I'll come talk to you. And finally I just wanted to say isn't there a little bit of overlap with Ansible Container? If nobody has anybody not heard of Ansible Container? So Ansible Container is an effort by the Ansible team to create kind of this concept. I've not really dove into it. I don't want to talk too much about it, but the main difference between that from what I understand is Ansible Container is leveraging a top of docker build infrastructure. So they're not doing the raw image build. They're not expecting an image to already be created and then overlaying their playbooks. Again, this approach that I'm discussing is end-to-end, we own everything and build it and so on and so forth. And I have, oh, right on time. At a time, questions? What's your question? These, obviously, places and locations of everything for both Ansible and Disk Image Builder. Yes. So Fedora, okay. So with Fedora we use it's Disk Image Builder and we have an element to use Yum Downloader and we express what the minimum set of packages are. So when you went back and seen the minimal element here for Ubuntu, Disk Image Builder expresses what that minimal image is. Now it's a static representation in the process of trying to merge some of this stuff upstream. We had kind of a discussion, a heated discussion of, well, Disk Image Builder wants to express this is the default because this is what the default is today. But in a container, you might not need Yum or you might not need any of this. So that's one of the, I don't wanna say pain points but that's one of the struggles as if you're going to go this route, you may have to create your own representation of a base Fedora that says I want Yum and something else and that's it. Right. Yes. So that's part of the things that we as an upstream project didn't really, we didn't support Fedora until very recently and the only reason we got Fedora is because Ian in Australia basically said we need something for Fedora and I'm just gonna throw some code into this and it magically works. We try to, Ubuntu is a little bit different, we use Deb Bootstrap, minimal build, right. Again we express we need, on the minimal image we need Python 2 or Python 3 or something like that but you're right, it's not exact duplication of upstream build methods and I wouldn't say you don't wanna do that. This is for you creating your images that you need for specific purposes and our purpose is testing, right. How you get kind of a nice separation line. This is safe stuff, this is not safe stuff. So what I should, what I was gonna do while we were all talking is actually get this process going after I made sure my actual builds worked. This is, there we go. So basically that, this is what happens in the background is the boring stuff. You can see if you go to nb01opensack.org these are the actual log outputs so after I was reading you can go and see how we do this. Sadly this isn't using Ansible but the one that I had on my laptop running is using Ansible so basically when we get back to it like it's running Ansible now it's kicked into Ansible and now it's doing Ansible things via this CH root connection which like I said the whole thing I was trying to express one minute, last question. Stop talking. Okay, hopefully you found it informative and yeah, thank you.


What's going on everyone? Sam here with late and sports guards. We are ripping 23 tops archives, six series, retired player edition baseball, 20 bucks, case break, number seven, random team style. Again, the way this will work, we have our 20 participants. As we do the break, Sean will be typing in a description of all the hits. Once all the hits have been shown, all 20, copy them over to a random.org window, random the names and the hits, same amount of times, match them up, that's the hit you'll get. Let's go look everybody. Jason, I don't see you. No, Jason. Jason B. Did you get an order number? Did you get an order number? All right, so random hits, she's good. Let's do it. Fail two is going to be a bot lobby tonight. I'm going to easy dubs in there. Oh my god. There's just walls. Not really. No, I've had time, dude. I've been dealing with all this house nonsense. Clinic. Good luck here, guys. Let's do it. Hit number one. Miguel Tejada, 30 of 42 for the Orioles. That's sick. 2010 Bowman. Miguel Tejada, MVP. Tejada. That's so nasty. We've got a nice 11 of 18. Paul Molleter, some Hall of Fame action there that is going to be 2016 Allen and Ginzer for the Brew Crew. Paul Molleter. So we've got nice. Another one of these, 12 of 50, Jimmy Rollins on a 2012 Allen and Ginzer. That's really cool. Some Jimmy Rollins action. He's like a potential candidate for the Hall of Fame. People have been saying that's sick. To the Diamondbacks, Sean Green, 26 of 46. Is that 09? No, oh, six tops. Just kidding. Oh, six. Sean Green. No, these are legitimate buybacks, JT. So like you can see right there, 2006 tops. These are legitimate buybacks. So those are legitimate cards that have just been re-signed and slapped. Just freaking awesome. Nice. 58 of 9 and 9. Burt Bly 11 is a pirate. That is 81 tops. 81 tops. Burt Bly 11 with the Pillbox hat on. Classic Pirates hat. Oh man, that's sick. 13 and 99, 100 pence. Who's one of my favorite players growing up? 09. Is that 09 upper deck? No, 09 tops right there. That's sweet. 100 pence as an Astro, 13 of 99. Great. Philly, a pretty good giant. Ranger for a bit there as well. Nice. For the Reds, Eric Davis, 18 of 20 on 88 tops, I believe. Eric Davis. He was a beast. Yeah, shoot us an email. Shoot us an email, Amuse. LaytonSportsCards at gmail.com. Marty with a spot in the IMAX select, same with Jordan on that one. Yeah, LaytonSportsCards at gmail.com, Amuse. That is a Julio Franco, one of one, hit parade. 2007 tops. Julio Franco is a met. Very nice. Uno de uno. That's sick. Dave Stewart, three of 64. For the Rangers, that's 85 tops. Dave Stewart. Savannah Banana Legend. Yes, sir, Devin, you know it. Savannah Banana Legend, I love it. I'll see a duel come out of here too. That is a Huddy, Tim Hudson, 50 of 60. That is a 2006 Allen and Ginzer. Thanks, Doke. Thanks, Doke. Doke, the man as always. Again, guys, 20 to go on Don Russ Timo. 22 go.  A lot of potential autos, a lot of choice exclusives. Cards number to 25. All that kind of good stuff. Oh, Coco Crisp as a royal, 14 of 17. And it's going to be 2010 tops. Grats on that. Good luck, everybody. Coco Crisp to 17. Yeah, we got to get you that Franco. We got to get you that Franco, Phillip. Yeah, Hunter Pence as an Astro, Chris. He's on your, Hunter Pence on your squad too, which is awesome. Hunter Pence is one of my favorite players growing up. 34 of 81, Mo Vaughn for the Red Sox. That's a 93 tops. 93 tops. Now, Giant Legend. Powers here. The riverbend, it's stink. Nice for the Mariners. J. Buehner, 52 of 73. He's a Mariners legend for a bit. 91 tops. J. Buehner. He's part of that 116-1 team, I believe. Buehner was there. That's another John Crook. 67 to 99. 87 tops. Oh, nice. Steve Garvey. 15 to 22 is a Padre. That's sick. 2011 tops like reprint. Grats on that. Padres, that's pretty sweet actually. Steve Garvey. Wish they were Cardinals affiliates again. Got nice. Another Scott Rowland. 14 of 30 this time on a 09 tops insert. Our 09 tops heritage, excuse me. Yeah, and as an Astro as well, which was really cool. Scott Rowland. Next up, we've got a mini one-of-one, Gary Sheffield on a 2002 tops T06 reprint. Gary Sheffield. Nice one-of-one. It's a brave, brave Sheffield. That's sweet. I love when they do the mini autos. Those are cool. Next up, what a throwback. One-of-one, Jason Worth. Jason Worth as a national 2011 Bowman platinum. That's sick, dude. I know. I noticed that too, Tocque. I have no idea what that means, to be honest. One-of-one, Jason Worth, man. Hell yeah. That's so cool. One-of-one, Jason Worth. Philly legend for a baby who's on that Philly's World Series team. He did hit bombs. 7 of 32. Ted Simmons as a Atlanta brave on a 86 tops or 87 tops. That's cool, man. Empty last box here. What's that worth? Yeah, that's gotta be a yellow. I agree, Chris. That's gotta be a yellow on the forest. All right, last box. Come on. Oh, another one of my guys, dude. Detrain. 53 of 98. Dontrell Willis. Oh, man, as a red. Cincinnati Reds legend. Detrain, Dontrell Willis.  I love this stuff. Empty. Let's go, my guy, dude. Dontrell. Just picked up a Dontrell Willis auto as well. We're influencing the ref. We're gonna go over here, guys. Again, we've got our 20 participants and our 20 hits. Let's go ahead and roll the dice. How many times we're gonna go on each? Six times. One, two, three, four, five, and six. Francis M down to the Landy B. Six times on your hits. One, two, three, four, five, six. J. Buehner down to Jason Worth, one-on-one. I don't get yellows, man. I don't get yellows. Got Francis M with the J. Buehner. Jason R with the Ted Simmons. Let's go, Philip with the Julio Franco. That's the card he wanted. Philip is a Mets fan. Congrats, Philip. That's awesome, dude. Hell yeah, man. Shane with 100 pence, the 99. Ryan R. Detrain to 98. Christopher P with Eric Davis to 20. Edward P with the Coco Crisp to 17. We've got a Swaggy G with the Tim Hudson to 60. Scott F with the Paul Moller to 18. David D. Miguel Taha to 42. Eric E with the Mo von to 61. Eric B with the John Crook to 99. Robert H with the Sean Green to 46. Philip T with the Mini Gary Sheffield one-on-one. Damn, that is last spot mojo. Philip T hit two of the three one-on-ones with the two last spots. Yeah, congrats, Philip. Jason R with the Dave Stewart. Shane C. Shane Garvey or Steve Garvey. Landy with the Scott Rollin to 30. Got Brody C with the Jimmy Rollins. Congrats, Brody. Jimmy Rollins is sweet. Kevin D with the Y 11 to 99 and Landy B with the Jason Worth. Thank you guys again for filling this up. Like I said, this is one of my favorite products to open. Next up is the last case of this tonight.


Welcome back to Think Tech. This is the state of clean energy, our flagship energy show every Wednesday at 4 p.m. I'm Jay Fidel. And we're talking today with Richard Ha, who's what do I call an energy environmentalist on a big island. And for many years he had this wonderful farm. Was it Hamakua farm? Hamakua farms in Hamakua, Richard, wasn't it? It's Hamakua Springs. Right. I remember. Now you're not doing the farm anymore, but you're certainly doing energy. Tell us about it. Yeah, so well, you know, I went back in 10 years ago, I went over to explore and learn about energy because my costs were going up, you know, about 2008 when the oil price spiked. So I went over there to one of five associations for the study of pink oil. And it was in Houston. And the first thing I picked up, there was about maybe 500 of us there. The first thing I noticed was they said the world had been using twice as much oil as it had been finding and had been doing that for the last 20 years. And right there, you know, as a banana farmer, I knew that this, you know, it's a finite research. It's something that I'm going to pay attention to. Oh, yeah. Well, I mean, I, you know, I just got off the phone with Lou Pugirisi, who was a think tank person with E. Prink in Washington, and he was in China and Japan looking at their energy supply. And one of the elements that came through is that the U.S. is under Trump is exporting substantial qualities of gas, including LNG. And for that matter, oil. We were having what he called an energy and that's mostly fossil fuel or renaissance right now.  In fact, we're generating more gas and oil than we were a few years ago, and we're selling more of it to the world. And that's likely to continue. So here in Hawaii, you know, we're focused on clean energy, but, you know, we're probably ahead in terms of percentages of most of the states and the national averages anyway. And so to me, it's very important that we keep on doing that, although I think we have to recognize that the world right now under this administration is not as concerned about it as we are. Yeah, absolutely. You know, and the second thing I learned when I was there was this concept called energy return on investment. And there's this Professor Charlie Hall, who's the father of EROI. And basically what it says is it takes energy to get energy. The net energy that results is what society gets to use. So as you know, several examples, back in the 1930s, the energy in one barrel of oil would get you 100. Today, it's down to around 20 to one. Shell oil and the tar sands is like six to one. So it's getting harder and harder. The net return is less and less. And the more you go, the more energy it takes to go find it. When you get to one to one, there's no point in going anymore because there's no benefit, energy benefit to it. That doesn't mean there's no oil. That means there's a lot of oil. It's just not worth going after. Well, arguably, people in the industry, the engineers and scientists and the like who work in and for the industry are finding new technologies that will improve that ratio and will make it cheaper to get this fuel out of the ground. But certainly, we knew a long time ago that shale requires a lot of energy to squeeze the shale and sort of crack the shale physically into oil. We know that the oil is in the shale, but getting it out is expensive. And the same thing with the tar sands. The tar sands, tar is mixed with sand and the tar is oil, but you have to process it to get it out. And what all that tells me is that there's not as much liquid oil fuel available and that we have to use these new techniques and we have to use this technology and it's going to get more expensive as we go forward. It's very troublesome. We need to focus on other kinds of energy, especially clean energy, given the crisis of climate change.  We all have to be very mindful because our lives and fortunes are going to depend on how we handle climate change the whole world. Absolutely, absolutely. In 2009, that's when shale just started and I was going to these peak oil conferences and we realized it's just started. Nobody really knew what the potential was, but I was sitting in Denver at the next peak oil conference and in this panel discussion, Mark Berman said that they studied 4,000 shale wells in the Barnett shale because the Barnett was the new one at that time. That's 4,000 wells. He said the average well produced 96, I'm sorry, 90% of what he was going to produce all within four years. And I'm sitting in the back of the room thinking holy smokes. Does that mean that every four years we have to drill another well just to stay even? And in 2009, the answer was yes. But since 2009 to now, they found many different shale plays. I'll total of about 10. Most of them are stable and in decline. And the Barnett shale, which was starting its production at that time, is now in permanent decline 10 years later. So I already knew it in 2009. But now it's getting to the point that the Permian Basin, which is the largest supplier of shale oil for the US. But where is that? That's in Texas. Permian. Yeah. So what they found is that the shale wells, what people do is they go and drill the sweet spots wherever they think is the best place so that they can get the most money for their efforts. So they did that in the Permian. But now 50% of the wells in the Permian, all right, wells that they drilled in between. So the mother wells, they set it up and they did pretty well. And now they're going to what they call the shale wells. But the problem with shale wells is it cannibalizes from others. So the productivity of it is less. Now you mentioned technology. And of course, in the last three, four years, technology has increased at the production and lowered the cost, but did not increase the amount of a shale well down there. So what they did was instead of four years, now it comes out in three years. So what we're doing is more rapidly depleting the resource. That's what's going on now. I remember when these issues first flowed into public discussion, one of the big points of discussion was how much oil is left. And that seemed to go away when we were told that oh no, the technology can provide more oil than before. And then of course it also flowed out of discussion when we were told that there's effectively an unlimited supply of natural gas down there, including LNG. But I think the issue still exists, doesn't it? At some point along the way, we're going to run out. We're going to run out of both. Isn't that true, Richard? It is because the shale gas and oil are related to each other. They have the same characteristics. It won't last forever. But the shale gas now, of course, there's a lot they flare it and they give it away practically if they could. But that's not going to last forever. Once they are able to export it and start selling it, it'll decline as well. And the price is going to go up. The trick is to get to alternative fuels for many reasons, but especially climate change and the effect of fossil fuel, but other reasons too. And we have to move that away. Unfortunately, as Greta Thunberg has made all too clear, is that people are not countries and governments are really not addressing this problem. They're rather going the other way in a lot of cases. And for the lack of leadership by this administration in Washington right now, other countries are not as excited perhaps as they were when COP21 and climate change issues were at the top of the agenda. So Hawaii can be a leader in addressing fossil fuels. And when we talked to you and me a day or two ago, you were talking about the great game changer. And I think that's very interesting not only for Hawaii where it would happen, but for any place watching Hawaii, any place affected by what we do here as leaders in renewable energy. Can you talk about the great game changer, Richard? Yeah, but first I'd like to comment about Helco and Hiko and what they've done. They've done a really good job because they've been leading the whole country actually in fossil fuels and renewables and stuff like that. But having said that, we have on the big island geothermal, and not only in the East Rift but around the base of Malacca, a lot of heat. So the way the geothermal company and Helco works together is that they have a lot of the 38 megawatts geothermal provides now. Ash goes to guarantee that they'll use it and they'll pay for it. The other half is standby. So in other words, if something bad happens, then they get to switch through the switch and they expect to get the electricity almost instantaneously. Now the game changer is this. Instead of throwing the switch off when it's not needed and just waiting a standby, throw the switch on and generate hydrogen as a byproduct. So now when you think about it, they're making money off the main product which is the half that they supply and which the utility needs as a base power. But they have all this other capacity that they're not using. They're just waiting to be asked to use it. So instead of just waiting, convert it to hydrogen, it doesn't cost anything. It's a byproduct. So it has really strong implications because the US mainland, there's a difference between us and the US mainland. We decided to stay with fossil fuel, which can hit the oil, which we're using now. The rest of the country went to natural gas and coal and stuff, but primarily natural gas. Natural gas is pretty cheap right now. So everybody is concerned about the carbon that comes out of it and climate change as you're pointing out. So there's two ways to get hydrogen. One is to take apart natural gas. When you do that, you have to add in costs. First, you got to pay for it.  On the other hand, so you build up your costs. On the other hand, you come to Hawaii and hydrogen is a byproduct. It's just an afterthought. It's practically free. And that is the gain changer because Hawaii has never been in front of the energy situation like it could be in this particular instance. Well, what are the other byproducts, Richard? So I know. So the hydrogen is a byproduct of geothermal. What are the other byproducts? From geothermal? Yeah. Well, basically it's just the fact that they're not using the electricity. They're just waiting on standby. I see, I see, I see. And then that's a battery too. The hydrogen serves as a battery. It's dispatchable anytime you want. Absolutely. And you can transport it. And it's a little bit more complicated. But it's been done. Different parts of the world, they do different ways of handling hydrogen. So it has been done already. So it's not something that is brand new? Well, I mean, technically, it's brand new. But you know, let me raise two questions about this model of using geothermal to create hydrogen and then using the hydrogen wherever and whenever you like. I mean, it all rings true to me. I've been following hydrogen for a long time. I've been following geothermal for a long time. And I know that HNEI, White Natural Energy Institute, and Rick Rochelot and Mitch Ewan have been working on this for a long time. And so is Stan Osterman, the hydrogen coordinator for the state. But I get two issues that I'd like to mention to you. The first issue is geothermal ran into, what do we call it, cultural problems, which turned out to be political and legal problems back in the 90s.  And that was offensive to them. And so they fought back. They protested and there were incidents of violence. There were incidents of threats, death threats, through the scientists and the engineers involved. It was all terribly unpleasant. It ultimately settled down in the late 90s. And ORMAT came in and became the operator of the geothermal facility there in Puna, Puna Geothermal Adventure. And it's been generating, you know, at a limited rate, I would say, since then until the eruption a couple of years ago. So the problem with all of that is the resistance hasn't gone away. And then politically, there's been a kind of cap on the amount of geothermal that PGV has been able to take out of it. I want to say it's something around 38 megawatts, maybe 40. But these wells could provide much more than that. And they've been limited as a practical matter because of the, you know, the political limitations, political pushback. And so I don't think that pushback has ended. I think that if you wanted to expand the production, if you wanted to even resume the production, there's an issue there because, you know, as we know, you can make a protest that will stop a project. And right now, I think there's probably people who would like to stop this project on the same basis they wanted to stop it in the 90s. So we have a sort of built in political cultural limitation about PGV, which is the biggest source of geothermal in the state right now. I know there are other possibilities, but the one that's working and has proven out is PGV. So what do you say to that, Richard? How do we get by that problem? Well, you know, I've been involved in that. And I'm trying as much as possible to educate about that because I'm directly involved with the 30 meter telescope. I'm one of the people that are pro 30 meter telescope. So during the discussion and the discussion that I like to have is a courteous one, not us against you kind of a thing, more a discussion about the facts. Yeah. So every chance I get when we're talking about the 30 meter telescope, I bring up geothermal and I talk about the characteristics and why it's safe. And things have changed, you know, I can feel it. It's changed quite a bit. And here's why, because the focus that was there many years ago is might be focused more on monarchy. And then that's one thing. And the second thing is there was a eruption that happened, a couple of eruptions just recently. And the folks that were involved directly, especially Ikeika Marzol knows exactly how the Riftstone works because he was walking the ground. He knew where the lava was coming, where the Riftstone was going all the way out to the ocean. So he knows all that he can he saw all of that. And he knows that the geologists and the science and the people from the Hawaii Volcano Observatory, you know, when they do the reports, because they track all the earthquakes and everything like that.   And if you saw a picture from the air, what you would see is geothermal, except for several two, three wells. It's intact. The whole area around there, several hundred houses disappeared. So why is that? It's because they set up the location of the PGD site, the thermal site on the high ground. So when it erupted, it went all around because it was lower. And PGD is on the higher ground. And the reason it's on the people's side is because they deliberately, they know where the East Riftstone was, and they set it up deliberately to be on the evil side where the transmission lines were. So if you just went to the Hawaii Volcano Observatory, talk to the scientists, they would be able to explain the characteristics and the safety of it. Richard, why don't we take a short break? Then we'll come back in one minute and continue this conversation. Aloha. My name is Wendy Lowe, and I want you to join me as we take our health back. On my show, all we do is talk about things in everyday life in Hawaii or abroad. I have guests on board that will just talk about different aspects of health in every way, whether it's medical health, nutritional health, diabetic health, you name it, we'll talk about it, even financial health. We'll even have some of the Miss Hawaii's on board, and all the different topics that I feel will make your health and your lifestyle a lot better. So come join me. I welcome you to take your health back. Mahalo. Aloha. I'm Stan Osterman, Stan the Energy Manager. Every Friday here on Think Tech Hawaii. If you're really interested in finding out what's going on in energy, especially here in Hawaii, but also all the way around the world, and especially if it has to do with hydrogen, look into Stan the Energy Man every Friday, 12 o'clock Think Tech Hawaii. Be there. Aloha. Okay, we're back with Richard Ha, who joins us on my phone from the Big Island. And we're talking about we're on the Hawaii State of Clean Energy Show every Wednesday at 4pm. And we're talking about the game changer hydrogen on the Big Island and maybe more. So Richard, you were you were telling me how we can get by the political cultural problem that has existed around Poonajee, a thermal venture over the past, however, many 20 years or so. Yeah, and we had conversations with Bill Eyla, the director of the HHL, Department of Hawaiian Homes Lands. And on Department of Hawaiian Homes Lands, there's a geothermal. So, so just imagine part of the problem we have that 30 meter telescope are Hawaiians that are very upset because they're dying on the wait list for the Department of Hawaiian Homes Lands. So if we were able to enable a geothermal operation there, it'll do several things. One is diversify the risk. Yeah. So you put it up there on Hawaii Homes Land. You know, the fact that the Department of Hawaiian Homes Lands owns the resource means they don't have to pay royalties. That royalties get up to about a million dollars a year just in saved royalties. That would be an income stream, not counting the income stream of actually doing a geothermal operation. Yeah, that could really help them. No, that could be a great, great contribution to an improvement in the quality of life. Absolutely. And can you imagine if they can go to hydrogen, which they can, of course, for the same reason that's a byproduct, if they went to hydrogen, and they started exporting it, let's say it's going into Oahu, for example, or exported wherever you because you can be exported. That income stream would be pretty incredible because it would all be focused on Hawaiians. And you know, that's what we're talking about. Yeah. Yeah. Folks that don't get any benefit. Yeah. This is this a direct benefit from geothermal. Yeah. So but one thing though is, okay, let's assume that we have geothermal wells working at Puno or elsewhere. We have to have the infrastructure to convert process, I guess, the geothermal into the hydrogen as a byproduct and so forth. That's going to cost a pretty penny, isn't it? How do you do that? Yes. So if you were to use alcohols electricity to do it, it would be very expensive because it'll be like 30 something cents a kilowatt hour. On the other hand, if you made it on site at the geothermal site, then it's a byproduct. You can charge whatever you want. You could be hardly nothing. It depends on, of course, the deal that you can work. Yeah. But we can, using this as a byproduct, we can undersell any source of hydrogen coming from the US, because they have to either natural gas, then they have to transport it, and they have to give it up the carbon. So for the first time, we'll have an advantage over the whole United States, you take the heat, and you're generating electricity, and Helco takes 20.  So what they do is you just turn it off. What I'm saying is that don't turn it off. Keep it running and be catching it out of it. Because it doesn't matter. Yeah, because we're going to be over the hospital for 500,000 years. I see, I see. So you do go through the process of turning the geothermal into electricity, but that part which would be curtailed or that part which you don't use, you use that to create the hydrogen, and you put that hydrogen in a bottle. Right, exactly. Now all of a sudden, hydrogen from hydrolysis from electricity going to water outcompetes thinking about natural gas. Because it's a byproduct. Yeah, that's the game changer right there is the byproduct. So let me ask you and you alluded to this earlier. And this is the other question I wanted to pose to you is transportation. You know, I've been to HGAT Stan Osserman showed me around. And he showed me these big tanks where you put the hydrogen and you store it and you and you transport it. And it within those tanks, it can it can go really anywhere. And the question question is, how much transportation infrastructure is in place? And how can we expand that to make the export of the hydrogen in the tanks, a working model, you know, a successful export experience to the mainland and to Asia, wherever, you have to have certain equipment, and you have to have certain equipment, the receiving end, take the hydrogen out of the tank, and use it for whatever, you know, whatever activity you want to use it for. So where are we on that continuum? What do we have to do to build that infrastructure, so that it becomes marketable and we can, you know, derive the benefits you were talking about? Yeah, so so Stan Osserman and Mr. and folks, they know people who who actually do it. And as a matter of fact, Japan is moving toward hydrogen economy, if they can get there. And they've had two consciences now, so it's brand new. So there's a lot of things that's going to be developing. But as I understand it, you could you could ship it, it's more involved because the molecule is really small. But but it can be done. And it's pretty much like protein, you just ship it that way is how I understand it. And you can ship more if you put it on the pressure and stuff like that. So that's all chemical stuff. Right. Well, you mentioned that Paul Brubaker was appearing a couple days from now in the Hawaii Economic Association to discuss this subject. So what do you know about that? And when where does that fit in the in the picture of trying to develop hydrogen as an export product? Yeah, so so they'll have it at the Holly Kalani, yeah, and it's all day affair. All day at the Holly Kalani. Okay, okay, it's the annual conference. And Paul has a breakout session. And he'll be the moderator, asking the question, what is the great game changer in the next 10 years? And I'm saying this is the great game changer. Now, you mentioned Greta. Can you imagine now, the geothermal is the greenest of all possible sources right here. Because instead of digging it out of the ground and using diesel to run the machinery to get what you got to get out of the ground, and heating it up and mixing it and making it into panels or whatever else you make it out of. And then with fossil fuels and then shipping it to Hawaii, or to the mainland with fossil fuels. You don't have to do any of that. All you got to do is get the heat from underground. There is no carbon involved at all. So on the front end, geothermal is absolutely clean. On the back end, there's no carbon. The only thing that comes out of your tailpipe is water. Yeah. So now, we can be the leaders of the whole nation and as an example, and somebody should tell Greta that. We should have her here, Richard. You should have her on the show. You, me and Greta. Yeah, absolutely. One thing that occurs to me from what you say, though, is very interesting is that if we made batteries, sort of like that analogy that you made about how many barrels of oil it takes to bring up a barrel of oil. If we make batteries, we have to spend a lot of energy and resources, you know, building the battery. And these big batteries take an awful lot of energy and resources. And over time, they decline too. And you have to get another one. Then you have a disposal problem, depending on what the battery is made of, what are you going to do with it when it's when it's spent. So in the case of geothermal, none of those issues exist. You put it in a tank, it stays in the tank. They don't have any secondary effects to that. So as you said, it's a very good renewable. And it doesn't have any negative implications as other renewables might have or batteries might have. Yeah. Yeah. And then take a look at this. In the industrial scale, geothermal, that Kauai has two, they only have two hours of battery storage, you know, think about that. Yeah. Only four hours. Yeah. You really need about three weeks of battery storage to have a stable, dependable source of electricity. But why don't they do more than four, four hours? Because it costs too much. That's right. So it will never replace oil. So what is our replacement? Because that's what we got to do, right? We have none. So if you had hydrogen, you could have as many of those tanks as you want all stacked up ready to go anytime. And you could use the ones you want to use and leave the other ones for later. So the whole thing would allow for as much storage as you want over as many days as you want, we could have a storm that lasts a long time. And still there would be, there would be power. So imagine what that would look like. So here's a hydrogen fueling station.  It drives up, takes you five minutes, it fills up your tank and you're on your way. And all you got to do is bring the hydrogen to the place, put it in storage, and you have an electrolyzer that puts it in your, so you can put it in your car. So the economics is this. The difference is that above, if the cost of the hydrogen is equal to 10 cents a kilowatt hour at the nozzle, it's equal to gasoline. So now try to think about that. It's 10 cents. Can you get it from the source, which is a byproduct to the gas station at 10 cents?  You know, so either way, you know, it's a finite resource. We're on the winning end of this deal. Well, Richard, we're out of time. That's Richard Har from the Big Island, a fellow who has followed energy, especially renewable energy for many years.  I look forward to the next time. And I'm sure there'll be more to report. Thank you so much. Oh, you're welcome. Aloha.