Live from the MGM Grand Convention Center in Las Vegas, Nevada. It's theCUBE at splunk.conf 2014. Brought to you by headline sponsor, Splunk. Here are your hosts, Jeff Kelly and Jeff Frick. Hey, welcome back. I'm Jeff Frick. You're watching theCUBE. We're at splunk.conf 2014, the fifth annual Splunk user conference here at the MGM Grand in Las Vegas, Nevada. I'm Jeff Frick. I'm here with my co-host, Jeff Kelly. Thanks, Jeff. When we're joined by John Rukowski, who's with Forester and Analyst Covering Infrastructure and Operations Professionals. That's correct. John, thanks for joining us on theCUBE. No problem. It's great to be finally on theCUBE. I've seen the channel online and stuff and it's finally great to be here. Excellent, welcome. That's great to hear. We're glad to have you. So we were just chatting a little bit before we went on air that you've been covering or you introduced to Splunk in any event many years ago, 2007. If you can retell that story a little bit. How you first came upon Splunk. It seems like such a long time ago now. When people say 2007, it's like, wow, that does seem quite a long time ago. So actually, I was first introduced to Splunk. It was actually a Microsoft Management Summit. And they introduced their technology to me as being a bit like a Google search engine for Windows servers. You could aggregate log files, do searches for important bits of information. And I thought, well, that's definitely interesting. And I can see some of the use cases for this. But is it really disruptive? And I looked at it and look at where we are today. Look at the size of this conference.  This is where IBM hold IBM pulse. So the growth of this company has been absolutely phenomenal. Yeah, so let's walk through that a little bit. How did they achieve this kind of growth? I mean, you cover, among other things, application performance management and infrastructure monitoring, those kind of areas. And certainly that's where Splunk plays, but they do more than that as well. And if they've kind of, we've talked a little bit on theCUBE earlier about, they've confounded to Wall Street a little bit because Wall Street doesn't quite understand exactly what they are. Are they a tools company, a platform company? How do you account for this growth? Really, what they're doing very well is that they're able to record, analyze data. And I think in today's kind of modern world, in a kind of digital economy, then data is really the new gold. If you can mine that data, turn that into information very, very rapidly, present it in the right context to an audience, that's when you really generate business insight. So Splunk's development, its platform, has enabled for this rapid recording of data, kind of rapid interrogation of data. It's just had the right platform. Plus, as the solution is developed, it's provided great features in order to interrogate that data. And I think with Splunk Enterprise 6 and moving onwards, it's made it very easy even for non-technical people or technical professionals to interrogate that data, so the use of pivot has been a great addition to the products. So a lot of it, I mean, we have to remember, we live in a world in which technology fuels our lives, it fuels our businesses. And where there's technology, there's data. The more you can understand that data, mind that data, turn it into insight, then the more potential competitive advances you're going to get. Well, contrast that to, or put Splunk into context in terms of the APM market. So what, the things you just described, were the traditional APM vendors not able to do that in a way, were they more focused on reporting versus analytics? I mean, how has Splunk been able to disrupt that particular market? Yeah, well, Geoff, you're kind of hitting the nail on the head, as we'd say in the UK. Basically, you know, a lot of them, kind of monitoring or management approaches when it comes to application performance management, when it comes to infrastructure workload management. I would classify a lot of those approaches as being kind of a rear view mirror look. So monitoring the performance, the availability of applications of infrastructure workloads, reporting, alerting, you know, those are very kind of IT centric terms. It's not very kind of forward looking. Now granted, the APM market, application performance management market, is shifting very, very quickly. If you look at the number of kind of the key players there, they are moving into this area of analytics. I think everyone understands the value of data and so in data and information. But historically, monitoring management has been about rear view analysis and not forward looking analysis, but that's starting to change. So whereas Splunk has been, you know, kind of sitting on top of a pedal still, for a number of years in regards to analyzing machine data, I see that changing very, very quickly. There's going to be a number of more players in here. We're going to see APM players coming in. We're going to see those more established players. So like, sort of, you know, IBM and maybe even CA come in here and try and disrupt that market. Everyone wants a piece of understanding data. And talk about why that's so important, specifically in the application development world of having forward looking, well, both real time and also to some extent predictive looking, predictive analytics, when you're talking about supporting applications in the so-called DevOps mindset. So the reality is, I mean, the world in terms of delivering applications, in terms of delivering business services, we're having to move faster and faster. Customers, employees are demanding this, you know. Internally, business services applications are there to kind of make the workforce more productive. In terms of externally, mobile apps, web-based applications are making sure that you can engage with your customers as and when they want to engage and delight them at the same time. And the reality is, you know, when it comes to predictive analytics, it's really about understanding that customer's behavior. And I think this is where Splunk's kind of evolution is starting to take them. It's not so much about understanding the performance of an application or infrastructure. At the end of the day, you can have a highly performing application, a always available kind of infrastructure workload. It's about understanding that customer, that user, kind of what they want, what delights them. Because if you can understand that, you can serve them in a better way. And that's what kind of understanding data and big data analytics or business intelligence and as a whole host of words you can apply to that. This is what it's really about. It's been about understanding your customer, understanding your customer personas, their journeys, how they engage with your organization in order to better serve them, in order to delight them and in order to stay one step ahead of your competition. Because if you don't do that, if you don't delight them via your applications and business services, then there's one thing which can be guaranteed. They will switch to another provider. So clearly, yeah, I mean, I'd like the way you frame that. They are moving from this idea of keeping your application performing at top speeds and reaction time versus what are exactly, what value is it delivering to customers and how can we improve on that? The value which is delivering to customers today is that as the engagement channel starts to change to being kind of digital engagement channels, so customers are engaging with organizations through digital means, whether that's mobile apps today, maybe tomorrow that's going to be wearable technology, that's just around the corner. All of these consumer devices rely on software, rely on the performance and availability of applications. So for the reality is for kind of IT operations professionals who are here today, and that's kind of Splunk's kind of bread and butter, that's kind of where they come from. They're really helping safeguard and optimize commercial success of their organizations. So the reality is, you know, why this is important is that because it's not just about performance and availability, anymore technical performance and availability, it's about commercial performance and availability. So that's where the use case is starting to shift. And now are you seeing the underlying, so the application developers, do they have to develop a new skill essentially to understand the shift? You know that if they're used to looking at, oh, I want to keep my application running if I do that, I'll check the box versus adapting to what the customer actually wants or how you can serve the customer better. I think ultimately the world of IT, so the world of the IT professional, whether you're talking about the IT operations professional, whether you're talking about the developer, the architect, I think we're going through an evolution, you know, brought about by changes such as kind of DevOps, changes such as cloud, but the reality is organizations are having to move faster. And any IT professional really has to take an outside in approach to the services which they're developing and the kind of services which they're supporting. First and foremost, they need to understand what does my industry do? Why do my customers buy from my organization? Why do they engage with my organization? Because they have to understand that the services which they're developing, applications which they're developing really are linked to the commercial success and more importantly, the brand of their business. What strikes me is this kind of DevOps move into the way things are delivered, not so much from a software development methodology, but really from a perspective. It's not the day where you go into the room with the whiteboard and do an MRD, which is a big thick thing, and then you do the PRD and that's a big thing. That's really not the way anymore, right? It's deliver it, listen, make change. Deliver, listen, make change. And making the changes based on real customers and what they want. It's a very different kind of methodology of going to market and trying to deliver that customer value as opposed to trying to design the perfect solution or what you think people want or a better speed and fee than my competitor. Really that outward focus. Yeah, the reality is, or we really see the shift here is that DevOps or sometimes DevOps is really a marketing term. I hear that DevOps far too often. It's really about modern service deliveries, about the evolution of IT. We're delivering faster, but we still need to maintain quality. So those feedback loops become very, very important. The feedback loops between operations and development. But more importantly, what you need to do is understand that there's a full value chain here, which you need to kind of understand all the way from the external customer, all the way through to the backend kind of server teams and kind of database teams. The reality really is that in order to move faster, you need to have information and data at your fingertips. And this is where a good analytics solution enables you to do that. It enables you to record that data and turn that data into information very, very rapidly. And this information and insight can be used to optimize these feedback loops and provide information to the right audience. And not only rapidly, but democratically. I think that's the other big thing.  Where now it's really switched in this post-Google world where the data's all there. Now your job is to kind of query it and take a little feedback, query it again, take a little feedback and to pump that all the way down into lots of people doing their jobs as opposed to the hallowed ones with all the PhDs in the back office behind Mahogany Row. Exactly, we live in the world at the moment which is focused on what's called the data scientist. They're the new saviours of the organization. To me, if you're in the role of being a data scientist, that's a limited role. It's a tactical response to not having the solutions which we require today in order to kind of mind and understand that data and turn it into information and insight. But you know, a good analytics solution is looking to understand the questions which have been asked of the information in order to remove that need of the expensive data scientist. And I honestly believe that Splunk are kind of moving in that direction with the easy kind of pivot tables, being able to easily interrogate that data, making it so that you don't have to be a data expert in order to query the data. Well, that's where then we start to remove that kind of need for the specialist data scientist skills. Yeah, and the other piece that strikes me is their focus on bringing in lots of different data. I don't care if it's structured or it's unstructured.  I don't care if it's between two machines in the data center or a connection to the mainframe. There's value in that and pulling it together, aggregating it together. Certainly they're benefiting from reduced prices in hardware and lots of faster computers. But to bring all that together and then let people query it in almost a more human way. I just think of when you go into an agile software development shop, right? They've got the big boards all over the place and the burn downs and this power of a feedback loop in a software development environment. And now we just had Aaron on talking about his customer agents also having big boards all over the place, looking for changes in application performance or changes in people hitting a particular place and then responding with that real time feedback loop. Exactly. You emphasized the point about structured and unstructured data. And again, I don't care about that definition. This whole notion of structured, unstructured data in 2014, it really is not kind of, it doesn't hold weight anymore. For me, any analytics company is doing business intelligence because data is so and technology based data is so intrinsic to commercial success to the applications which have been run. Then it doesn't matter whether you're looking at structured and unstructured data. And it's actually, I've just finished writing a report which talks about kind of monitoring and how we need to elevate the role of monitoring to be really being about situational awareness. And when I talk about monitoring, I also refer to analytics as well. And the reality is this notion of situational awareness comes from the US Marines. So when a US Marine is on the battlefield, they need to have the right information at the right moment in order to make the right next decision because it's critical. You know, in the world of IT, in the world of business, well, our battlefields may be different, but it's still the same concept. It's about making sure you're getting the right data and information at the right time in order to make the right decision. Now, more and more, as we kind of see the analytics market moving forward, it's not about the human-making decisions. It's about automation then kicking in to make automated kind of actions. That's the kind of next iteration as well. Now, do you see Splunk playing a role in that? Can they move to, right now, most of the Splunk solutions or people are looking at dashboards and then they're making decisions. Can Splunk move into that automation role? And I agree with you. I think when you can automate real-time intelligent decision-making, that's when you can make a significant impact in your organization. Because if you're dealing with applications or a consumer doing something on a mobile app, you don't have time or something to sit back and say, hmm, here's the offer we should make to that person. It's got to be automated in an intelligent way. Can Splunk play a role in that? I think they will play a role because they're setting themselves up as being this operational intelligence platform. You know, any good enterprise solution today has got open APIs, which means that you can easily integrate the solution with other solutions. The automation market is still very much emerging in that sense. But, you know, before we can automate something, there's a fundamental aspect which needs to be addressed. Any automation means that we must trust the data which is being recorded. And that is not a kind of step in which, you know, Splunk can help people trust the data, but that's really an evolution of we as human beings. We need to be able to trust machines. I know for my history, we've been brought up in the 80s, when I was brought up on Terminator. I don't want that to happen. But, you know, fundamentally, this is the way which we're going. For organizations to move faster, for them to serve services to you as a consumer and applications to you, to delight you, to deliver services in context, then automation plays a big field in this. But the analytics engine behind this, the algorithms which turn data for information is just key for automation to be successful. How are we going to get to that point where we can trust the machine?  Is it a generational thing? Is it going to take the next generation? Or is there something that can happen in the meantime to move us along and people like you and I and help us trust the machines? Some of this is definitely, you know, a generation thing. As we see, you know, the so-called millennials coming into the workplace, I think there's more of an acceptance about, you know, being comfortable with the use of technology, with the use of automation. I mean, my two-year-old already kind of understands how to use, you know, a modern smartphone. So she's already kind of saying the name of a smartphone. I'm not going to give that smartphone any more credit or that company any more credit. But she knows how to use it, how to access videos. She's literally knowing which one it is. Yeah. And so some of this is a generation thing. But, you know, some of this is that if you really want to stay or remain competitive, then, you know, every organization today who wants to remain competitive is trying to understand their data, trying to turn data into information, trying to use analytic solutions. But what's the next stage then? Well, you need to automate. You need to do it faster than your competition. So automation just becomes part of being competitive. So we will see that shift coming along. But we have to become more comfortable and trust the data which is being kind of recorded and which has been analyzed. Yeah, and I think, you know, one of the other things that's going to move it along is when those early adopters are seeing success and their competitors are seeing themselves being passed by. Yeah. You know, fear can be a great motivator for the adoption of technology and new approaches. This is more of an approach, not necessarily any given technology. I mean, there's any number of technologies that might help us get to that point where you can automate some of these real-time decision-makings. But it's more about the approach and, as you say, understanding and trusting that the decisions are going to be accurate and are going to provide the value that you expect. Exactly. I mean, you mentioned this is not a kind of solution, but it's an approach. And what we do in IT today, we're very good at productizing different kind of trends. We talk about cloud. We talk about big data, business intelligence. We talk about DevOps. And we productize and segment each of these areas. And in reality, all these areas are interlinked together. This is about an evolution in the way we deliver technologies, but an evolution in our relationship with technology. Well, how do you think this is going to impact the so-called mega vendors out there? This is kind of getting back to the products, but can they, you know, the vendors that brought us the more traditional data management, data warehouse, in the eye world, a rear-view mirror look, the very structured approach where you've got to know the question ahead of time, wait three months to get the data warehouse modeled. Three months is probably being generous. Can they pivot and, well, what stops an IBM or an Oracle for coming in and doing what Splunk does? Can they make that transition? What are the barriers for them? There's many barriers. Some of this, you have to remember these kind of big organizations, you know, they're public companies and they have to kind of address the stock market. They have to address their shareholders. They have to make profit. And so when you're in a cycle of selling solutions, which is still selling, because, you know, IT's still fragmented to the selling, one monitoring solution to the database team, another one to the server team, they're selling, you know, different solutions everywhere, then that makes it easy to generate kind of profit and you get locks into a very kind of short-term cycle. So all you're doing is trying to satisfy, you know, the stock market, Wall Street in that sense. And so in the last year, we've seen actually organizations transition to being private again. So they can start to become innovative but they don't have to answer to the stock market. And so this could be a challenge for Splunk actually. You know, Splunk is a public organization now. So it'd be interesting to see how quickly or how innovative they still are in the next couple of years because they have to maintain and generate profit. Yeah, well, and, you know, we're seeing they're still losing money. They're close to break even, but I guess last quarter, but generally they're still losing money because they're investing it all in the, back in the company and, you know, trying to lay the foundation for even further growth. But that's a critical question, I think you identified, is when do they take their foot off the pedal a little bit? When do they start trying to turn a profit? And how does that impact their ability to innovate and stay ahead of all the kind of next-generation competitors that are nipping at their heels? Yeah, I suppose that for that question, you can say, well, when is Amazon going to make a profit? Well, yeah, I guess they're going to get a lot of pressure but it's not stopping them from just plowing all the revenue back into the company. So we'll see. Exactly. At the end of the day for Splunk, if they really want to be disruptive and they're moving into the business intelligence kind of market and disrupt that market, which I think they're on track to do, then they've got to maintain that innovation. They've got to maintain, they've got to spread their brand beyond IT and into the business. You know, I want to be at a conference in the next couple of years of Splunk conference and rather than the CIO of G Capital coming on the stage, I want the CMO to come on stage and show how they've used Splunk to generate insight, to generate competitive advantage. And I think that'll be the time when you know that Splunk has been really, really successful. Well, John, thanks for coming on. John Rakowski from Forrester, great insight. I always love getting analysts on. Great point of views and some really good insights as the focus and the locus of competitive forces is changing as everyone's becoming a software company. It's terrific. So again, you're watching theCUBE. Jeff Kelly here with me, Jeff Frick. Our third year here at Splunk.conf. We'll see what's happening in a couple of years, how many CMOs we have on the keynote. We'll be right back with our next guest after this short break.


Welcome to DevOps days Baltimore. This is my first DevOps days Baltimore, but not my first DevOps days. I Always joke that Nathan Harvey and I only we both live in Maryland Right outside of Baltimore. We only ever meet in other countries at DevOps days Which is almost always true. This is like one of the first times that that we've collided in our home states so When when I originally was on the program was on program for an hour long slot So you can Hopefully empathize with the challenges of delivering this in 30 minutes So I'm here to talk to you about reanimating DevOps and I have I have a lot of harsh words just to wake you up in the morning Right, so I'll try to keep my potty mouth in control, but that's not gonna happen, but so So the most interesting thing to me that's happened in DevOps is actually CICD All right, so when we look at that first a survey how many people here were in the industry Computing industry in general before DevOps became popular Okay, that's a great number And how many people came from a software development software engineering software development lifecycle side and How many came from an operations side? Oh Wow, it's like we can have some sort of pit fight later. It's gonna be great so when I have some controversial Stay I was around when DevOps was coined and participated in the DevOps community a little more passively than actively since then And I have a very specific view of how this thing played out and how I wish it had played out instead So when I think of DevOps, it was really putting dev into ops what I saw it as it was a somewhat condescending view from software engineers coming in and saying these poor operations people Man, their life is so tough. They don't understand how to automate anything Everything they do is just just wrote repetitive crap I'm gonna take my software magic and put it into your operations And the two things that I would say that came out of that that were really fantastic were CI and CD So the idea of automating the software engineering Software development software release release engineering release management and even deployment which overlap a lot with ops So it was really really amazing So it was a way for organizations to really move faster and it required a tremendous amount of collaboration between Development and operations to do this so This talk is about reanimating DevOps from a from a corpse and carcass may be a bit harsh But then again, it may not be that harsh So so what is it it look like right so the the reason that Cincd became so important is because has anybody ever launched like deployed software before Okay, keep your hands up if it worked Right, so we keep deploying the software and the software is junk And so the interesting part of all of this is that what really happened was that I want to make sure this is the right slide This is the Great. Okay, so the one on the left. All right So software engineers really had this idea that that operations people struggled so much Trying to deploy software and and the fact that they couldn't automate things They you know, they really needed help and operations people were just like, can you just please not give me shit software? That's all I'm asking for is just don't give me stuff that doesn't work, right? So the The empathizing with the target of this this effort didn't work very well, right? So software engineers identified problems that operations people had but it wasn't their primary problem So what do operations people need? So what I want to leave you with is the part of dev ops that never really came to fruition out of that We had put a lot of software engineering practices into operations What we didn't do is put a lot of the operations blood sweat and tears and operational practices and techniques Into software engineering and that was really what's missing so operations desperately thrives Unobservability Right when you deploy something you have to be able to ask yourself Is it working and if you can't answer that question you have serious serious problems? The first thing that we do is always nothing, right? How many people here have launched software and have no idea if it's working, right? Hopefully less today than ten years ago, but ten years ago It was standard practice where it's like the box is on I can ping it and the process is running My job is done as operations and the reason it was done Was because you had no porthole into that software to understand whether or not it was working or not So then what do more sophisticated software engineers do they give you logs, right? So then you're paying splunk a lot of money But you're looking at logs you're doing some sort of red light green light Monitoring to make sure that the system is working correctly, but it's still systems Just because they're up and on doesn't mean that they're they're actually servicing their their users well, right? So then the third phase of that is really metrics analysis, right? It's it's taking measurements out of the software that are an indicator It's a surrogate indicator to tell you that things are actually working the way they're supposed to be working Turns out that's still not quite enough Because when those metrics go wrong and you know this isn't working, right?  Had it break and no matter what question you wanted to ask of the software. It said here's the answer Like why is this slow? What are you doing? Why did that database? Command why did that database operation fail, you know, why did the service crash? What is the stack trace? Like those are the types of things that the software should tell an operator All right, it shouldn't be some sort of magical thing that they have to divine so a really a quick Kind of detour is that we actually really solved this problem on a single system About 12 years ago, maybe 15 years ago with detrace Does anybody here know what detrace is? couple people How many people here run Linux? Okay, how many people run Windows? Okay, those are the two platforms that don't do detrace Every other platform in the world pretty much does it from QNX to free BSD to Mac OS X to anything It was originally developed out of some microsystems on the Solaris platform and Solaris 10 But what it is it was a frustration of software engineers when they were in production and they wrote this beautiful piece of software Which of course didn't work and they deployed it and it didn't work right and the customer was saying this is slow and Then you get in you get in on the production system And then you say why is it slow and they didn't have the tools to answer that question So I don't know how many software engineers have told you when you're operating something You're like this doesn't work right and they're like well, can you put can you repeat it in development? It's like No No, the problem is here go fix the problem So the idea of being able to interrogate a production system when it isn't working is the holy grail, right? I don't want to have to Repeat the problem in a development environment because one I may not repeat the the same problem I may struggle to repeat the problem and in the end of the day that One production problem that I can observe is the real problem. So detrace actually solve kind of all of that it allows you to Instrument basically anything in the system system calls HPA adapters everything from the hardware performance counters all the way up to Lines of Ruby code and stack traces all in one simple language that looked a little bit like awk Where you say when I reach out over this TCP socket and I write some data that looks like this Give me a stack trace and tell me how much memory. I just recently allocated You know like weird questions that one would say why would you ever ask that question and an operations person says don't judge me? Right you're in you have a problem. You need answers to your questions, right? So recently ebpf came out in Linux and we actually have a bright future now So ebpf is plumbing inside of the Linux kernel that really allows for building tools like detrace So it is pretty exciting times. We are in the beginning of those times So really quick a Story about web monitoring does anybody here run a website? Yeah, use the web, you know Okay, cool. You're not all like an app nation So is anybody used like Google analytics or some other tool like that, right? Okay, so Rewind to 1997 1998 did anybody actually run a website in 1998? My they only like three hands four hands. Okay. All right. We're old So I'm gonna give you this insight.  Right. Well one 95% of people didn't know didn't know they had to care But the other 5% we ended up paying inordinate sums of money to companies like Gomez and keynote to ping your site Every 15 minutes to make sure it was working and When it came back with a green report, right? You can do the math on that. It's four times 24 times per day You were like a hundred percent uptime. I'm perfect. This is great and you're all laughing, right? You should be laughing, right? Checking your site is working every 15 minutes should give you about zero confidence that is this available accessible and performing That is the dumbest thing that we can do with today's technology. So what do we do instead, right? We embed analytics into the web page and every single person that actually visit the Visits the site ends up reporting things to you like performance data and availability data So you actually know, you know, obviously don't know who didn't show up, right? If someone tries to access the site and it completely doesn't work They don't report they report nothing, right? And that's a little hard to track Which is why we still ping our sites every five minutes usually every one minute But most of the data that we have about the performance and the availability of the system comes from this real User monitoring right so real user monitoring for the web is Obvious now I'm going to paint a picture of today Which is how many people here run infrastructure at all like any systems like a Linux box or Windows box or you know If you're happy and have de-traced with a free BSD box How many people like Ping it every 10 seconds or minute to ask it questions like is your disk full or are you on? Right. Oh, yeah, it seems kind of interestingly dumb like the other system that we did 15 years ago 20 years ago But the idea is that you know that computers actually doing stuff. That's its whole point, right? It's actually running software. It's servicing users. It's making to speak connections sending packets all that stuff Every one of those things you could potentially measure to see how it's doing But that is the future of what this is going so when we're talking about data explosion Instead of pinging a box every we used to ping them every five minutes Then we ping them every one minute now some people say well, I ping it every 10 seconds, you know, that's fast enough It's like, okay, do you realize with like 72 cores in a box running at you know 2.6 gigahertz? computer actually does a lot in 10 seconds like a lot a lot a lot right billions and billions of things in 10 seconds and Any one of those can have an aberrant latency that can cause problems for users It can it can go off the rails So so you actually observing the entire stack of that and all of the behavior that happens on the system is going to happen So the problem is is that is an information scale problem, right? When we look at at Moore's law That is that technology tends to double every 18 months So if you do that math out from 2000 to 2017, that's a 2,500 fold increase 2,500 times The the size of data ingest so we're starting to look at things like an Exabyte a day coming in to architectures if you start doing all of that and people are like What like exibited that's insane. No one's gonna do an exabyte a day So I'll rewind 20 years and say we have a terabyte of RAM in boxes And if you told that to somebody 20 years though, they'd be like, you know, that will never happen ever like that's crazy There's like one box in the world that needs that who cares everyone else will scale out Like we have racks and racks of boxes with one terabyte of memory, right? So this this is coming So 15 20 years from now the idea of people tossing around their dollar a month exabyte cloud Storage is totally gonna happen, right and the data that you're gonna put in there Is not necessarily information overload, but you definitely have to have tooling around making sense of it all These are the problems that are coming is what do you do with that? So kind of back to systems that are broken so there are two methods for dealing with systems that are broken One is because in this previous slide We don't have an exabyte of data We cannot write this if I observed every single thing that happened on any of my systems I would explode my information space that never be able to store it. I can't really make sense of it So I can't store it all so there's two things that I can do I can either store it all in aggregate throw away critically Important details about that or I can sample that and Anybody who tells you that one of those is right is completely wrong. They're both right They're two entirely separate approaches. One of them is that you sample data. So every 100,000th transaction or every transaction that has like a known anomaly you end up tracing really really detailed information about it Everything you can think about that piece of that of that transaction and you keep that and that's really useful for understanding the the nature of transactions in the system and debugging problems But then there are a hundred thousand other things that happened in that system that you don't know about and what you want To do is you want to store everything but you can't so what you do Is you end up storing behavioral data about that it's like how long those operations tend to take and you aggregate that data so Where does that go on the tracing side you have technologies like ebpf and t-trace event driven stuff There's a lot of stuff from like honeycomb Is starting to look at that? That the product is really cool. I don't really understand the marketing there Because they're highlighting the top one as if it's the only one that matters the top one is super important and Those sorts of tools are really really interesting and help you debug systems help you ask what is going on But without generalized behavior to information you're really left with only one side of the story So we need that generalized behavior stuff. That's what we do at our company. We track data so I Want to dive into things that you can take back to the software engineering organization that can make their software run better in Production right that's our entire goal is not just to ship shit fast. It's to ship Software fast really don't want those two things to be synonymous, right? So how do you make your software better? And the first and most important thing that we can learn and this is much more important now that we have distributed systems everywhere I studied at Johns Hopkins. I was in the Center for Networking and Distributed Systems there I ended up leaving sans a PhD after spending way too many years there if you get me drunk enough I'll tell you the story I Left and said I will never debug distributed systems again and here we are so that sucked that did not work out How I planned but the one most important thing you can do in building a system in particular a distributed system is Understanding how to fail with grace right you need to fail quickly and safely And I would say that the airline industry is a great example of anecdotal stories around this You have engines fall off of planes and they still land and people people survive right so the idea of like when things go wrong You don't want to just keep trying right the engine falls off the plane. It's like we might make it You know, that's not how you do it. You say this is the procedure. We're gonna shut this down We're gonna turn it off as quickly as possible And then the way that that happens in distributed systems is much more complicated because all of the adjoining systems need to accommodate that Right. So when people say new software is really designed for failure That is a layered cake of truths and pains so any part of your system can fail at any time and and being able to react to those different components in there and their states of Uncooperative nature is really important So there are a lot of techniques in doing this. I'm not gonna dive into those enough time The second rule is that autop autopsy are not just for medicine Okay The days of old IT when the computer froze up and the mouse didn't work and the IT person came in and said well If you just restart it everything will be fine. It's not an answer Okay, these things when you do a hundred thousand transactions through them You're gonna have them again and again and again and you're gonna screw more and more users As your system grows it is not acceptable to have a failure in your architecture Especially one that repeats that you cannot explain you have to get to the bottom of the problems Computers are so much better This is not how I want to say that this is being recorded Computers are so much easier than children Right, so I go to work and I tell a computer what to do and it Pretty much always follows my directions You know aside from a couple of microcode bugs, but even then they're following someone else's directions There tend to be very few scenarios in which a computer is actually Malfunction in an on the malfunctions in a non-deterministic way my children are entirely non-deterministic So There isn't because this the system always follows your instructions It means when it screws up it did what it was told. It's very likely. It's going to do the same thing again so The days of did you turn it off and on again? Really need to end and even more so in large-scale deployed distributed systems Another one which if you've been to any software engineering conference, you've probably heard Is using a technique of circuit breakers and this is about failing gracefully. So there is a very very big difference between being electrocuted and being shocked Right, so When something is about to fail Trying harder is usually not the right answer. You actually want to have the circuit breaker flip So you really need to design tolerances in your systems so that when a component or an interaction in that system It starts to go wrong or goes too slow Then then it doesn't just keep piling on more work So our traffic systems and everybody love the traffic right here anybody come from DC. Yeah, there we go. Enjoy going home So that is a great example of a system unlike out west Do you know on on the on ramps out west they have the red light green light? Indicators to flow control traffic into the into the highways I Have no of like one road around here that does that but I the the Beltway certainly doesn't do that And what happens is you get three and a half hour backups because when the road slows down There is no way for the underlying system to stop putting traffic into the problem Right, it just adds more traffic to the problem and this doesn't work Well, it really really doesn't work. Well, if you look at some research about a self-driving cars They tend to go a little bit slower and they leave more cushion between cars And they tend to not have traffic jams because of that because there's no jerks. They're all driving the same way so by By putting circuit breakers into the system you actually can control the behavior of the system better Right, it's much better to have you know 50% of your traffic turned away and 50% of it served well then a hundred percent of it served poorly right and If you serve all your traffic poorly when I say poorly it means that your your web page doesn't load the images Don't all load you can't check out like it's great to go and shop and then you can't pay Right like what do you do anybody know what you do you go to Amazon? They don't have that problem right so now you've lost business Amazon's got more business There's a reason that that that Amazon designs the systems the way they do And has anybody ever had a service failure in Amazon? Yeah, has ever anybody had a really frustratingly long load experience with Amazon where you thought it might work But you just couldn't get it to work Because I have never had one of those When I go to Amazon, it doesn't work it doesn't work. It's like we're a little we're overloaded. Sorry, that's it Right and the reason is is because they are gating traffic in they only want to let enough customers in Where they know they can deliver that service they let too many in a degrade service for everyone to the point where they abandon So rule four is that you cannot under you cannot understand What you cannot measure all right, so So many times in software You know, there's data structures and algorithms and searches that are running everything that you do has a performance If you launch an API service people are hitting that API it takes a certain number of microseconds to service that call I Don't know how many times I've gone into an organization and said how long does it take for that API to call to run? It's like That's pretty fast like what what does that mean? What that's not a that's not a speed fast It's not a speed. This is not the Ricky Bobby of API services, right? so You really need to measure everything an error on the Even if you don't store it all air on the on the side of exposing Everything you can in a piece of software, right? The software should be Instrumented so that someone can ask a question later So understand everything and you can do that through measuring it and in a lot of cases and this is for your job's sake The best way to not get to do an awesome job and not get a promotion or get a bonus or anything Is to go improve the system and not be able to describe how well you improved it like I made it faster I want to go fast. That's great No, but if you can come in and say I dropped, you know user user service experience from from, you know, 700 milliseconds down to 250 milliseconds And hey, look, there's this Shopify report from like eight years ago that says that's awesome for these reasons Look our revenues went up our shopping carts went through or you know, our users are happier If you are measuring those you can see the impact of your work and your colleagues work And there is nothing worse than being able to there is something worse than being successful and not being able to show That's being horribly unsuccessful and being fired But it's very frustrating nonetheless so you measure to understand and One of the reasons that we measure and it's a predicate to this is that a Lot of times in operations is one of the reasons that I railed against Operational culture for a very long time is that in operations people just expect you to keep the website up Right and with a phrasing like that If it's not up, it's down. That's a hundred percent. That's perfection, right? The idea of aiming at a hundred percent uptime and a hundred percent service quality is the dumbest thing that any Organization can do because you can never exceed expectations Right and and you should never set up your environment that way, right? The the culture in the company needs to measure things and they need to use that to build in what we call failure budgets Right, you need to have a budget for failure You need to say that maybe my site only needs to be up 99.9 percent of the time, right? That's actually a pretty loose constraint, right? My site's down all the time because Amazon's only up 99.9 percent of the time But if my site is up 99.99 percent of the time going into the end of the month, I can go down Why would I not go down? Still meet all my constraints Right. Why don't I take those risks? Why don't I launch those features that were marked slightly unstable and test them out in production where I can get faster feedback more valuably and still be able to deliver on my my quality of service requirements and my service level agreements So by measuring everything and defining what your uptime is and what your performance is It means that when you do better than that and you are more stable than that You can introduce the risk that allows us to all go faster Right, so DevOps in general is around moving organizations faster. That's not actually true It's about de-risking the speed and this failure budget methodology allows you to de-risk speed So and my last one is that While justice should be blind operations should be not blind at all. It's it's a it's a sad thing. I Love this picture. It's my favorite Don't show it to little kids. It's kind of freaky the kermit the frog and the hand. What did they do to him? so When you are in production the only failure that matters is the one that you're experiencing right now So it's absolutely critical that everything that you do in developing software and all the practices that you apply and operating that Software can lead you to diagnosing that failure in reality Like as it is in production repeating failures is both a highly error-prone and very elusive I've seen software Architectures where you have a bug report from a customer and an engineering group of three or four people spend three weeks Trying to reproduce the error in development so that they can fix it Right go into production and just figure it out And if you wreck a train car in there you're allowed to in your Environment if you have an error budget, right? So you can actually go and look at these things live I do say that that GDPR and other things like that Make it a little bit more difficult because as you go into production you as a as an engineer are exposed to information that you may Not be allowed to be exposed to so this complicates things But there are ways to do that using canary systems. So you have production environments that don't take normal production traffic Right, but it is in production. It uses all the production systems and when you have a problem It's a lot easier to Divert a problematic user to that canary system or infrastructure than it is to repeat the entire problem in a development environment So there are methodologies and techniques that we can use when deploying production systems that give us access easy access to isolating Existing production problems so that we can solve them And that is that is it.  Thank you very much


Welcome to Ruth McGregor Hall. When you first enter McGregor, you're greeted with a beautiful common room that has bay windows that overlook upper lake. Ruth McGregor Hall was built in 1967 and was named after Ruth McGregor who was a part of the Class of 1910. She died on her way to serve with the Red Cross in France in 1918 during World War I. At Holyoke also transitioned this old dining area into a study space. Here's a quick look of what a triple could look like on the 5th floor of McGregor Hall. One of the best parts of living on the 5th floor is this beautiful view of upper lake. Each person in the triple has their own closet space, desk space, and drawer space. Not to mention you're in close proximity with Kendall Sports Complex and Upper Lake Walk. I hope you enjoyed checking out McGregor Hall.


Hi everyone, Joe for jazby's casebreaks.com with our last break of the night. It's gonna be 2018-19 Panini Prison Basketball Choice Edition. This is one box pick your team number nine with a chance to win the Mavs in this break So let's grab everybody's names and sort it by column A One spot or two entries gets you one two teams purchase you one entry four gets you two six gets you three So on and so forth Adam gets four so that's two entries Bradley has two teams. That's an entry EA has two teams. That's an entry guy has four teams. That's two entries Josh E has eight teams. That's four entries Kevin with two teams. That's an entry Matt Silverstein with two teams at the entry Rhea with four teams so that's two entries and That is 14 so one spot in 14 It's gonna get the Mavs in this one box break. Let's roll it Let's randomize it three and a two five times name on top after five one two three Four and fifth and final time after five times. It's Adam Kelly. There you go. Congrats Adam After five the Mavs are yours now will the box cooperate? I don't know that's the that's the That's the rub that's on that's what's on everybody's in the mind. It's like all good and well you get the Mavs, but Will the box deliver? That's the big question Yeah bears I run an eight hour show five nights a week the other two nights the other guys are running eight hour shows, too When you love what you do bears Doesn't feel like eight hours. Oh, that was a Nick can he was helping us out with a mixer We can make that a little bit smaller right here. There's the final printout right there just these choice boxes It's just eight cards one pack eight cards They sell these each of these boxes sell for a lot due to all these exclusive parallels Right. Oh, I see a tiger stripe. Nice. I haven't seen those in a minute. All right We got we're gonna start off with Nicola Batum to 88 That'll be for Charlotte that's gonna be for EA We got the blue green and yellows patty mills spurs Darren with that tricolor Trey Lyles Denver Nuggets Adam Kelly with the Denver Nuggets Danilo Gallinari Clippers That's gonna be for Adam Kelly Where can you find one of these? These are like over a thousand dollars. I think You can you can ask us we might have some available for personal boxes again in the future We've got Elia Kobu Rookie Silver Yeah, I got a personal box you can buy from Jason and then you can We can pop it open for you or can get it sent to you sealed Kevin with the Suns Clay Thompson Silver for the Warriors. That's gonna go to Adam Kelly behind him is Quitting Richardson Clippers That'll be for Adam Kelly. It's the Adam Kelly show in this box Now what about the tiger stripe? Let's see who that goes to The tiger stripe is I think I saw a rookie there Tiger stripe is Pacers Aaron holiday Josh E With the Indiana Pacers gets the rookie Aaron holiday Tiger stripe Nice holiday family obviously popular in LA. I think all the brothers went to UCLA and everything so Clay Thompson Silver will go to Adam Kelly and there you go We have not seen Luca Donce. It's your Trey young, but we'll try to dig up some more and the hunt will continue I'm Joe for jazzy case race calm. Thanks for breaking with us. I'll see you next time for the next one. Bye. Bye


Transmitting a message by visual signals is much faster than trying to deliver a physical message like a scroll of paper. And to remember why, recall that every message system has multiple delays within. So if we wanted to transmit a message by, let's say, a smoke signal here, there are a couple of different components that we need to take into account. First, we would have to light a fire. Then the message would have to travel between the two points. And then finally, we have to detect that a message is being transmitted. So let's start from the beginning. First, we have to light the fire and that is our transmission delay. So that happens right there. Next, we have the propagation delay, which is the time it takes to travel between these two points. So that's B right here. And we know that propagation delay is a function of two variables. It is a function of distance, which is how far our message needs to travel. And it is also a function of speed or how fast we can deliver the message. And the difference here is, of course, we know that, for example, a horse can deliver the message in with the speed rather of about 24 meters per second. But when we use the visual signal, we are actually traveling with the speed of light. And the speed of light is approximately 300 million meters per second. So much, much faster. And finally, we have our processing delay, which is C right here, which is the time it will take us to detect and verify that the message is being transmitted. And one important takeaway here is that the propagation delay is almost negligible for visual signals because the speed of light is so fast. So really, we are bound by the speed of the processing and transmission delays, which is to say that the slowest parts are actually our operators on both ends because they need to light a fire and interpret the signal. However, there's another and an even larger problem than the speed of our operators here. And that is using smoke signals doesn't actually give us a lot of flexibility in what kinds of messages we can transfer. In fact, we can only transfer one bit of information because either the fire is on or it's off. And if the other end knows what that means, that's great. But we're definitely not transmitting sentences, words, or even characters. So to solve this problem in around 1700s or mid 1700s, we invented the visual telegraph, which is also commonly known as a semaphore. And the idea for the semaphore was to encode visual signals by using these mechanical arms and different positions would mean different letters. So for example, if we grab the one we saw here, let's call that letter A. But we could come up with a different position, perhaps we could reverse it and that would give us letter B. And of course, we could repeat the same exercise here for all of the other letters in the alphabet and then we could transfer entire messages. And the first practical use of the visual telegraph was in 1794 during a war between France and Austria. And messages were transmitted between Paris and another French city called Lille. And there were actually 15 stations of these visual stations which transmitted the message over a total distance of about 230 kilometers. And the interesting part is that this entire distance was covered in about 9 minutes for each character. Which tells us a couple of really interesting things. So first of all, we know that we have 15 stations and it took us a total of 9 minutes. So if we do some math here and we know that it's 9 minutes divided by 15 stations, this works out to be approximately 36 seconds per station. So that is our processing and transmission delays because propagation delays can be very, very small. Or said differently, this is our A plus C. And this time is the total time at each station for each character. So this is another important distinction for the visual telegraph. Unlike, let's say, a horse where the entire message would arrive at once, we are delivering this message character by character. So if we send an A, that alone will take us 9 minutes. And then we would have to send another character and so on. So one important thing that happened once the visual telegraph or the semaphore came along is that all of a sudden we started seeing code books, which is to say we started encoding messages with fewer characters. Because the limitation for the visual telegraph was not the weight of the message, which is definitely the case for something like a carrier pigeon, but the length of the message became very, very important. And because the visual telegraph delivered messages so much faster because of the speed of light, it actually became very, very popular in Europe in the 1800s. But there are also a couple of problems with it. First of all, it is actually very expensive to operate because you need many different stations along the way. So we have 15 stations here. And each station would need people to operate. So in fact, because of this, it was effectively limited to military and government use. And the second problem is, of course, line of sight. We would need to build these stations where we would need to be able to see them and see them well and hopefully at long distances. And then the last one, which is an obvious one, is if the weather is poor or, for example, at night, we can't actually use the system. And so because of these problems, the visual telegraph actually got replaced as new and better methods came along later. But having said that, believe it or not, we are still using visual telegraph in some instances. So for example, we have flag semaphores, which are used on ships. This guy right here indicates letter A. We have letter B and letter C. And in fact, we have entire alphabet. So you can search for flag semaphore and find out how to communicate using these flag signals.


This is State Tech Hawaii, the immunity matters here.  Yeah. Okay, I'm Jay Feindell. That's Mike Hamnet. We're co-chairs of this Hawaii, the state of clean energy program, which happens every Wednesday at 4 p.m., courtesy of the Hawaii Energy Policy Forum, of which Mike is the co-chair of that, too. Yeah, it was Sharon Morawaki who just retired. Yeah. And we're going to keep you involved. And we're going to keep you traveling far away. Yeah. Also running for office. Yes. Okay, and we're doing today something called the intersection of our energy and the law, courtesy of the William S. Richardson School of Law. And we entitled this on the tagline, will William S. Richardson please speak up? And you guys are going to speak up. We have three of you. I know there are only two of you now, but we have a third one, too, okay? So my immediate right is Natalie Morland, the student at the William S. Richardson School of Law. And at her left, did I say you were at my right? At my left was Natalie, and at her left is Gavin Tom. And at Mike Hamnet's extreme left is Nathaniel Muller, okay? In the wing. He'll come on the table shortly. Okay, they're all students. So welcome to the show, you guys. It's so nice to have you here. Thank you. Nice to be here. Yeah, great.  It's probably a legal experience, you know. I know. I mean, invite me. Okay, so we're going to find out what you guys have been studying at the intersection, the intersection between our energy and the law. Natalie, you're first. Okay, so we are three of nine, I believe, students in Richard Walgrove's clean energy law and policy class. So we've been studying all about the clean energy landscape in Hawaii in particular, and it's been fun. So yeah, we're here to talk about our final projects. So when you say studying, what do you mean? Is there a horn book on the subject? No, there isn't. So we have... They're writing one. They're writing it now, right? So we just have a syllabus, and Professor Walgrove puts all of...he organizes it by topic, and then he'll come up with just the materials that we use for the class. And so he'll give us lists of statutes, he'll give us white papers to read, he'll give us court cases, PUC decisions. Okay, and your topic, you came here today with something in mind, something you've been reading about and thinking about or writing about, what is that? So I'm talking about exit fees, and this is particularly in the context of customers who are exiting the grid or defecting from the grid. And exit fees are becoming a bigger issue because the cost of battery storage is declining, and the cost of electricity keeps rising, and it's becoming more economical for people to install solar plus battery storage and independent from the grid than it is to remain connected. Okay. Yeah, so I'm considering whether the PUC has the authority to impose exit fees and whether it should or should not. Let me add a fact to this porridge and see if Mike agrees with me. Last time I knew about this, if you wanted to merely have a holding pattern in your relationship with the utility and not use any energy from the utility and not sell them any energy, just sort of zero both ways, it would cost you as much as $18 a month. That was if you stayed connected to the grid and didn't use any electricity. So but if your system broke, if your batteries failed, if your solar cracked, I don't know what happens, you could call on the utility to supply you just as much as you need, and the cost of that insurance, so to speak, was roughly $18. That's right. Okay, we're throwing that in the barge. How does this affect your thinking, Natalie? Well, I think it all comes down to whether that $18 a month is something that the customer thinks is worth something. I think that if there were, for example, reconnection fees, then the $18 a month insurance policy might be not a bad idea. But yeah, I mean, it all just comes down to the costs and benefits of doing one or the other. And my thinking is that exit fees aren't going to be totally necessary because of those factors, because of the benefits that there are for staying connected to the grid. And for the benefits the utility will get from those customers. We made a bad decision when we first started with net metering and feed-in tariffs because they didn't really think about who was going to bear the cost of maintaining the wires. And that's really where they got caught. And there's this huge number of people that took the $18.75 a month and the utility said, hey, wait a minute, this is not covering our costs, which is why this whole question is back up in the air. And as you reduce the number of people out there who are in the grid, the utility taking a big loss on this $18, they can't maintain their infrastructure that way. So problem. OK, but the thing about the PUC, I mean, let me just react. Can I react? I'm reacting. Why not? Why doesn't the PUC have authority? I mean, one day the PUC may decide that it's a matter of policy. They want to impose or not impose whatever, you know, address the question. Maybe it would come up on the basis, for example, of an application, a docket application by the utility that they wanted exit fees in order to discourage and de-incentivize people who want to leave the grid. They don't want them leaving it. So the other PUC, is there any question, really, that the PUC could, they could impose a lot of tariffs and charges and all kinds of, look at your electric bill one of these days, there's all kinds of things. Why couldn't the PUC or the utility or combination of do that without legislative, you know, imprimatur? Well, so without legislative authority or without the specific language and the statute, and the PUC does have the authority to set reasonable rates and to impose these tariffs, but that's assuming that you're imposing these tariffs on a customer. And when a customer is exiting, they're basically saying, we are no longer a customer, you don't have the authority to impose this fee. So legally, I think the safest route, if exit fees were a good policy decision would be to go through it through the legislature and put a provision in the PUC statute authorizing that would be a safer way, wouldn't it? Otherwise, we're going to end up in court. Yeah, exactly. Kevin, you've heard Natalie, you know, talk about her thinking and research and her least preliminary conclusions. How much of what she said you agree with? Well, it's interesting. My particular paper is on the city's kind of introduction to renewable, renewable vehicles. So we're talking about clean energy vehicles. And I believe the mayors of all the islands met back, I think in January 2018 and express the resolution, at least the mayors of Maui and Honolulu did to go clean 100 percent energy efficient vehicles by 2035, I think 10 years before the people have come in. But it's interesting in that, sorry, it's interesting because if we look at Natalie's idea of exit fees and the city's idea of introducing all these renewable vehicles, what energy is going to be left if we don't have consumers in HECO maintaining that base load? So the buses, let's say, for introducing 500 new electric buses, that introduces up, I think, if my math is correct then it might not be, I'm in law school for a reason, but it introduces about 180 megawatts. You're just studying derivatives of those. It introduces about 180 megawatts of needed power to just for this new bus system that they may or may not introduce. So if we're having customers leave the grid and just be independent, who essentially will pay for that? So, I mean, other than the city and county, if everyone- Well, Mike and me will pay for it, we'll be after us and we'll have to write a check for it. Yeah. I mean, I'm only half joking. I mean, the taxpayers will have to pay for it. That's what happens. So what is legal about your analysis? It sounds like an economic analysis. True. Professor Walgrove gave us an interesting choice to write our papers on and I took maybe an unorthodox approach. I wanted to learn more about solar energy, so I'm looking at the placement of solar cells on vehicles in comparison to solar farms. Maybe from a procurement perspective under HRS103D and the paper centers around the efficiency of solar cells, essentially, if it's viable, if it's cost-effective. And then you place that under a regulatory framework that Hawaii has where the bidders are essentially low-balling sometimes. Procurement code. Procurement code. You end up with the lowest bidder who has the lowest efficiency, which may or may not be the best way to approach our goals, our ultimate goals of clean energy, 100 percent clean energy. So what would you throw out the procurement code? I threw out the procurement code in a nanosecond myself. I think Mike would agree with me on that. He won't say this after he retired yet. No, but RCH was exempt from the procurement code. Oh, God, that's... Now I understand everything. Yeah. Okay. So maybe there's a legal point there in the procurement code for dealing with government acquisition and all that. But I like to throw something your way also, as I did with Natalie. I don't know if you remember this, Mike, but years ago, when we started with Carl Friedman, and the notion was we're going to make an index of how much oil we're bringing into the state for transportation, for planes, for aircraft, for cars, all that, and for generation of electricity. And we want to develop an index, which we have been developing an index. And one of the participants in this very open-ended meeting was Henry Curtis, and he said, you guys are missing something. You have to calculate the amount of oil that is in furniture, there's plenty of oil in furniture. You know, and at the time, I said, what, what is this, what? But you know, since then, I began to think about it, and we were all down on fossil fuel, and it was a bad thing. But this morning, this morning before I came here, I was watching a video on YouTube, you know, because what happens is one follows the other, you know, it's almost involuntary. And it said, you know, do you realize that there's oil in everything, in petrochemistry? It's everything we do.  It's not just the cars, it's everything. So are you factoring that in, Gavin? I mean, fact is that oil is not necessarily bad. We need oil for civilization. So, you know, and solar, you heard it here, I think that solar is not going to replace, it's not going to make a toothpaste for you, or a toothbrush, none of that. Yeah, I think it's an interesting point, Jay. When, like I was saying before, if we introduce a whole bunch of new battery-powered SA vehicles to the grid, that energy needs to come from somewhere. And we know while we're attempting to build tons of solar farms, maybe wind farms, maybe in the future, even tidal stuff, that energy in the meantime is going to come from petroleum, burning oil, burning coal. And it's the easiest, I mean, we took a visit to the waste energy plant. H-Power, huh? H-Power. And right next to it is the AES coal plant. And that produces, I believe, three times 180 megawatts in comparison. So when we introduce, let's say, 180 megawatts of bus battery, a need for power, either we're going to have to greatly increase our renewable energy source, or, again, we'd still be relying on fossil fuels. Are you going to go into energy law somehow? Well? What year are you guys in? I'm a third year. Third year? Oh, sorry, now it's getting serious. Oh, yeah. If you got this far, you must never go to commitment, eh? I guess. I can't stop now. I can't stop now. Gavin, how about you? Are you a third year also? Third year also. So it's a little too late. You want to get a job here? How do you want to work this angle? Well, I think clean energy is the future. Right now, unfortunately, I am not working in clean energy. My background's more in the courtroom experience litigation. They have energy in litigation, too. Yeah, that's true. Plenty. So I mean, I am always open to any future industries and clean energies. The other thing everybody seems to forget is we have to firm the intermittent power from solar and wind. And everybody's talking about, well, we could go 100% solar. OK, so then when the sun stops shining, then what? Or when the wind stops blowing. And it's still, unless we get a firm source of power, we're going to be stuck with petroleum as the firming power. But that's why I think this issue of exit fees are from customers exiting because they're installing solar plus battery storage is really interesting because the battery can take the solar energy that's produced during the day and export it at the times when the demand is highest. And so it can at least reduce the need to keep all of the oil plants. Right, no, no, I agree. But it's going to be a long time before we confirm the power with batteries entirely. Aspirations is not always materialized. No, and the aspirations cost money, too. Materialization costs more money, right? OK, so what did we learn here today, Mike? So far, we were about to go to our break, but I thought if you would summarize, and then we'll see if they agree with your summary. We've got law students doing interesting stuff on energy. As far as their training, you know? And the exit fee idea is an interesting one. They're going to have, somehow the PUC and the power companies are going to have to deal with this keeping people on the grid somehow. And I think the notion of very high reconnection fees is probably legally the best way to go because we don't need any legislation to do that. PUC can give them the authority. People will see twice, won't they? I told you about that reporter for our civil beef who went off into the woods and separated from the grid, came back and made a speech. If I wrote an article, actually, in civil beef about it, and he said, I'll never do that. So the market, the market and the experience that he had, which will be repeated, which is being repeated, that will affect people's conduct. But what I appreciate about both of you guys is that you're looking into how people conduct themselves, how they think about this because energy is a moving target. And ultimately, it's an expression of the community because that's people either do it or don't do it. And we have to see where they're going without our incentives or disincentives. We also have to see what kind of incentives or disincentives we need to impose on them legally to have them, us, all of us, go the right direction. Well, and to keep social policy. And to keep the energy system working. I mean, that's. That too. Before we go to the break, I just want to tell you a story. Sharon Moriwaki and I are in Linai. And Linai, they didn't want the windmills there. And they ultimately stopped them.  And I said to them, this is all on film. I said to this young woman, I said, if you don't have wind or fossil fuel, how and only have, I said, what do you use? She said, we would like to use solar for everything. I said, but solar goes off at night, right? What are you going to do when the sun says? And she says, no problem. We'll go to sleep early. That'll solve it. This only works on the neighbor islands. Right. OK, Natalie, Gavin, thank you for coming down. Thank you guys, Sharon Moriwaki. We're going to take a short break. And then we're going to talk to Nathaniel. All right, good. This is Think Tech Hawaii, raising public awareness. Hello, everyone. I'm DeSoto Brown, the co-host of Human Humane Architecture, which is seen on Think Tech Hawaii every other Tuesday at 4 PM. And with the show's host, Martin Desbang, we discuss architecture here in the Hawaiian Islands and how it not only affects the way we live, but other aspects of our life, not only here in Hawaii, but internationally as well. So join us for Human Humane Architecture every other Tuesday at 4 PM on Think Tech Hawaii. Hi, I'm Ethan Allen, host of Eligible Science on Think Tech Hawaii. Every Friday afternoon at 2 PM, I hope you'll join me for Eligible Science, where we'll dig into science, dig into the meat of science, dig into the joy and delight of science. We'll discover why science is indeed fun, why science is interesting, why people should care about science, and care about the research that's being done out there. It's all great. It's all entertaining. It's all educational. So I hope to join me for Eligible Science. We're back. I'm Jay Fiedel. That's Mike Hamlin over there. Mike, say hi. Hi, guys. There it is. OK. And then we have Nathaniel Muller. Now we just had Natalie Moorland and Gavin Tom. And now we have Nathaniel Muller. What do you guys have in common again? Oh, we are all lucky enough to be taking Richard Wallsboro's class at the William S. Richardson School of Law. The class course is titled Clean Energy Law and Policy. And he is giving us a myriad of tools in order to properly address the future of our energy economy in the state of Hawaii. Now, he's got a lot of experience in that. You knew. Oh, he is firsthand. He's one of the very sharp little music with every shot. So what did you think of the first half? Both of them are pushing, excuse me, Ms. Moorland and Mr. Tom, are pushing ideas that are. I thought it was Gavin Tom. You're right. Mr. Gavin Tom. Natalie, never mind. Are pushing forward ideas that are. And this is an issue that we are all touching on. They are pieces of a whole because we're trying to rapidly change our entire energy infrastructure, our energy economy over a short period of time. And both of you brought up the issue before of reliability. That is the end goal for this. And so with regards to exit fees, how reliable are those individuals' own energy needs? Are they being met? And with regards to solar panels being placed on vehicles, the next issue is how much energy are they actually going to be taking off the grid? And how much energy could they actually be putting back into the grid? So these are issues that not only compound and aid in the movement towards a new energy future for the state of Hawaii, but also need to be addressed with the potential shortfalls that they can have as well, just like the issue that I'm going to be trying to present as well. Kay, I just want to make a statement and see if Mike agrees with that. Go ahead. You know, the whole thing about energy policy is not so much the policy, but the acceptance of the policy by the stakeholders in the community. And you and I, in our old age, we have seen policy points that could be good and the community rejected. We have seen policy points that could be bad. The community loved it. And so the question is educating the public and the legislature and the courts. And for that matter, to some extent, the PUC as well on whether the policy is good or bad. And you could make so many mistakes, and we will. We have done. We have. So you could find great social and energy technological policy, but at the end of the day, it's like litigation. You have to convince people, all kinds of people. Otherwise, your policy is on the back shelf. Real nice policy right there on the back shelf, yeah. So what's your program about? It's not really a program. It is actually more of a fun fact. And the fun fact starts with the number. Every year it has been estimated by the EIA, the Energy Information Administration, a nonpartisan branch of the federal government, that the state of Hawaii sheds off 6.7% of our annual energy production. Simply be a transmission. That's an electric energy. Yes, literally through the lines. If you guys have ever had a chance to, and I'm sure when you guys are walking along the beach or underneath any of these exposed lines that power our buildings, you can every now and then hear the buzzing. Oh yeah. That is literal heat energy that is being lost. And even though it might only be 6.7% in the grand scheme of things, that actually might be a large percentage of energy that we are actually missing out on. The state of Hawaii produces close to 10 million megawatt hours of energy every year. You calculate that up to 6.7% every year. That ends up to be also an EIA estimate of over 492,000 megawatts. If you are applying the 23 cents per kilowatt hour that a resident of the state of Hawaii pays in their energy, that is almost $117 million, literally lost to the air. The next step that we have to address is who is supposed to pay for these sorts of things? And as we've seen the fuel costs, it passes on over. These are fun facts. They're fun facts, okay. As we've seen the fuel costs, it is completely fine to pass that on to the consumer. But the consumer shouldn't be happy with this. Even though the energy sector in Hawaii as a whole nears almost $2 billion, $117 million is still a good chunk of change. And again, if we're getting to the transition period, that $117 million, every single dollar spent needs to be saved in order to successfully get to the new energy future that we're building up. The point is that by allowing these losses to continue to occur, we are effectively incentivizing utilities and energy providers to shift the costs back to the consumers and therefore succumb to it. So how should they conduct themselves differently? And that's a great question. Actually, recently the PUC just agreed and accepted the, excuse me, I know I have a lot of notes. It's the- Can I take the notes? You're more than welcome to. It is the- This could help me. It won't help you, but it'll help you too. It was an August 2017 grid modernization strategy that was put forward by HIKO. It has just recently been approved by the PUC. And actually within that, they are looking at piggybacking strategies in order to make sure that they are mitigating energy loss. Such things as actual real-time measurement systems of electricity as it is flowing through the lines. And then- Suppose I find out about that 6% of loss, right? What do we do about it? As an individual or as a- As a utility company. I mean, the person who has the power, that's a pun. The power to do something over it. It's very difficult, right? Because we're actually- Give me an example of how I might alleviate the loss. So this comes back to the simple rules of physics behind it. We're talking about ohmic loss. We're talking about corona effects. And again, if I'm coming here as someone who's supposed to be looking at the law, it's very difficult for me to be coming and talking about the science of it. I'm not a physics guy. I'm supposed to be looking at law and policy behind it. This reminds me of the CAFE standards. You know, where the federal authorities say you must make a car that will give you 50 miles a gallon on a gas. You must do that. Or you must make a fleet that gives you an average of 50 bucks a gallon. That's what- Whatever. 50 miles a gallon. However they frame it. Well, you could have something here, too, the sense of the utility. We're gonna watch you guys. And we don't want a 6% loss. Then you have to find a way to give us a 3% loss. Is this your paper? Is this your research? In the end, the outcome, and this is something that I will hope to propose and prove through the fun world of shifting facts and information of saying that it should, those costs should be shifted back into the PUC or the energy provider themselves. But with regards to your question before, how do we- How do we- What do we actually tell the PUC? What do we actually tell these power companies to do? And as of right now, it is most simply putting in new lines. The way that for a comparison, South Dakota and Wyoming have only around 2% energy loss via transmission. But it's because the majority of their energy is actually transported through over 700, it is a 756 kilovolt line, the very big towers. The higher the voltage of the line, the less loss. But of course, this system that we have in place is still fairly old, and so the smaller the voltage, the more distributed these things are. Okay, so the idea would be- The more loss there is. You don't lay this, well, you lay this on the utility. You say, you're the utility, no, we want you to put in new lines and spend the money in order to do that. The utility says, that's fine, but we're gonna build that into our rates because we're a utility. And we come to you every so often with a rate increase request, as they have done recently. And it's gonna cost us $500 million to do Oahu. So can you please give us a very substantial rate increase? Well, actually the grid modernization strategy that was agreed on by the PUC was a six-year plan costing over $205 million. Part of their strategy is to mitigate these losses through line loss and transmission loss. So $200 million. So this is already happening. It is already happening, but of course they are shifting those costs to the consumers. Sure, and you look at the big island and their line losses over there are huge because you're taking relatively small populations and you're taking a very big island to get the electricity around. And now I pay 34 cents a kilowatt hour over on the big island. So you're talking about a change in the utility law because this is a different calculation of rates. If you're saying you gotta eat it, you guys, you know, and instead of giving any dividends to your stockholders, who are gonna sell that stock when the word gets out in this model? Instead of giving dividends, you eat it. In fact, maybe you can make a capital call, get the extra money, make a loan, get the extra money because we're not gonna make the tax payers, rate payers pay for it. Very true. Is that what you're saying? Yeah, it's... But again, this goes back to just how the actual facts, how capital is gained through these systems as it is today. There's really not that much incentive for the utilities or the energy producers to actually eat these costs. They can easily shift it on over to the consumers. But part of this, again, goes back to the actual lines themselves, the actual transmission lines themselves. As one of the... As China is actually going through and doing, I believe it's called the Belt and Road. One Belt, one Road. Yes. It's also known as the New Silk Road. One of the ideas that they're putting forward is a tarot-watt line that is going from one side of China to the other. Something similar that we've been talking about and has come up in many HECO issues repeatedly is the intra-island connection. Now, whether that is, again, politically feasible, socially feasible, whether that will actually happen. Technically or economically feasible, too. It is economically feasible, but it's the same question. It's actually ready to do it. It's going to pay for it. Mike, you got some spare change. Sure. Okay, Mike, we're out of time, okay? And I would like to get your... You can ask some cross-examination questions of Nathaniel. He's ready. I can tell you he's ready. And or maybe you should do that and then summarize. Which way is the wind blowing at the PUC about this? That's a great question. Obviously, the PUC just agreed to $205 million over the next six years for a grid modernization. So obviously, the PUC is all for it. But again, it's how far are we willing to take it, whether these means are edible. Yeah, right, exactly. We got new commissioners who are into policy, who are from UH, not far from your law school. Matter of fact, one from HNEI and then another that has to be confirmed from HNEI. Yeah, just Malcolm from us. Yeah, they're into energy policy and social policy and all that. And they can make these calculations you're talking about. And they're up to this kind of issue. So maybe we'll see some real change here. Who knows? I'm excited for it. But it'd be interesting to see how they're going to deal with the line-loss issue. I mean, modernization of the grid is probably far overdue. And as they move toward distributed generation is another strategy that could be used to reduce the line-loss. Actually, that comes along with a lot of fear as well, because of the varying amount of voltage that can be entered and then lost into the system. Because if it is individual parties that are actually putting more energy into the system, one of the parts of the grid modernization strategy is actually real-time monitoring of that so they can properly calculate it. But then we get into death spirals, whether people just immediately jump ship. And whether, as was spoken before in the previous segment, whether this energy regime, as we know today, is still a feasible and still has enough money to keep it running.  They're going to separate. And in that case, that's a spiral down, because then the utility has less money coming in, and they have to find another way to make it up. So we've really covered some interesting issues here today. Their policy, their energy policy, and their law policy. This is an interesting class. I might sign up. So, you know. You want to sign them up? Your registration opens in two weeks. Yeah, registration opens. Mike, can you summarize this and sort of close the show, could you? Well, I think that the law school is, I think that the class that they're taking is an interesting class. And at about time, we've got some other folks involved in discussions about the whole energy issue. And I think the law side of it is very interesting, especially with what we've gone through with the changes in the feed-in tariff and the net metering and all that. And I'm just waiting for the dust to settle. And I'm not sure. These guys are going to be watching the dust settle. Yeah, they'll be determining what happens as we get to 2040, 2045.  Okay, I heard that. Just speak under Ossil. You guys will be instrumental in the last chapters of this huge initiative that we have seen begin. I mean, Mike was there when it just began, but you're going to see the end of it. Isn't that exciting? And you'll have a lot more to say than he and I then. For sure. I do love a happy ending in movies or otherwise. Okay, well, we're working on that. We're working on that. A happy ending here for the intersection of our energy and the law with my co-host and co-chair of the Hawaii Energy Policy Forum, Mike Hamlet with Gavin Tom with Natalie Moreland and with Nathaniel Muller. Thank you very much. And thank you to Richard Wallscrew for making this all happen. And to Avi Soyd for making the law school what it is today. Very nice. Thank you. And thank you guys too. Thanks, Kevin. Aloha.