Good morning, everybody. Welcome to our panel discussion on Cloud Computing's first economic recession. Let's talk platform efficiency. And let me start by asking a question. How many of you have a mild efficiency panic at your companies, at your organizations? All right. So you are at the right place. We are a group of end users. And we also have an efficiency panic at work. And we've been spending some time doing optimizations and figuring out what's the best way to make the platform efficient. And back in the day, it used to be capital expenditure. Like, you'd have a data center. The spend was predictable. And these days, with the move to cloud, it's a pay-us-you-go model. And it's really becoming an op-ex, operational expense, and impacting companies greatly. So I think that's the reason why we have this, and the increased focus on optimizing and squeezing more. And doing more with less is the mantra these days. So in this talk, we are going to be talking about three aspects, so culture, operations, and design. So these are the three broad categories that we are going to be talking about today. And with that, we'll introduce ourselves. And I'll hand it over to Todd to facilitate the panel. Yeah, thanks, Aparna. Yeah, go ahead. Phil. Hello. My name's Phil Whitrock. And I work at Apple on Kubernetes. I'm Nagu. I'm working at Zalando in the Cloud Infrastructure Department. I'm managing two teams, Cloud, sorry, Continental Platform and Cloud Cost Efficiency Teams. My name is Aparna Subramanian. I'm Director of Production Engineering at Shopify. And I've been involved in platform efficiency efforts for the past year and a half at Shopify. Hi, I'm Todd Ikenstam. I'm Principal Engineer at Intuit, working in our core systems team. That also includes our Kubernetes platform. Well, let's get started talking about culture. Obviously, the goal is ultimately to increase efficiency and reduce costs. But where do you really start? How can you get started? How can you gain organizational commitment to improve efficiency and reduce the costs? Phil, you want to start? Yeah, I can take this one. So one thing I think is helpful to start with is start out measuring where your big wins are. Where do you want to focus? What's going to move the needle a lot, but it's going to take a long time to do? Maybe not going to move it as much, but it's very easy to get done. And then from there, figuring out who the right folks to engage with are. What are the right teams? So you can start moving forward. Yeah, and I can add to that. So this has to be somebody's problem, right? So there has to be some center of excellence or a Phenops practice. And they have to be in charge of making sure that everybody knows that it is a consideration that they have to worry about. So often we run into the situation where it's everybody's problem, but it's nobody's problem. So I think having the central team is really important, but it's also important to understand that it now suddenly doesn't become only the central team's responsibility for making sure the platform is efficient. It has to be a collaboration between engineering, finance, procurement is the team that you have probably that is negotiating contracts with your cloud vendor or other vendors. So I think it really has to be a collaboration between all these different teams and that's how you bring about that awareness and accountability. Those are great points. And I would like to just add on by saying, you would like to drive the cost ownership through financial practices, as Aparna said. You have to have budgets based on accounts, teams and business units. You could also bring awareness in terms of how important this cost efficiency first are, because most of the times we would hear from the business that delivering a feature is more important because that's gonna bring in money to your company. But you have to sort of equate the cost efficiency with your GMV or revenue so that they understand how this impacts the bottom line in the end. And as Phil mentioned, you should also measure everything. At our company, we kind of have a lot of metrics. One of them is kind of looking into the resource level like per VCPU cost or per memory, gigabytes of memory cost. Or we also have unit cost based on applications like card or search or things like that. We can also have very higher level metrics like connecting the infrastructure cost with your GMV or revenue and then see whether it makes sense to invest or spend so much. Yeah. Yeah, great, thanks. Actually, let me get a show of hands. How many people know what their cloud bill is? Like how much are they spending in the cloud? Okay, to me, that's the first step. Like you need to know. You need to know what you're spending. And the old adage holds true. You can't really manage what you don't measure, right? So first measuring that. Let me see a show of hands. Who knows how much a particular service or application costs, not the whole? Okay, a little, okay, good, good. So I think that's the big challenge is taking that cloud cost, that big bill and attributing it to individual teams, individual applications because only then, when you have that visibility, you know where you have the opportunities to improve. So at Intuit, we have a dev portal where we track all of our different software assets, whether it's a service or an application and all those have some asset ID and that asset ID is propagated and tagged to all the resources that are required to support that service or application. And then we aggregate all the billing data and attribute it based on that. And so that gives us a fine grain and be able to put up a number in front of development teams and actually roll that number up to various directors and VPs and so forth. Because it's not enough to give the top level, the CTO or the CEO the bill, right? You need to give that visibility to the people who can actually make decisions and make changes to how the system operates. At that level of visibility is really the first starting point when we started looking into things more closely at Shopify, we were able to see clearly from the cloud bill, you can see what are the different projects, what are the different clusters. But it's not exactly helpful, right? Because if you have a multi-tenanted platform, you want to know how much is app A costing and how much is app B costing. And if you have shared platforms like database platforms, logging and observability, these are all multi-tenanted systems in its own. So you want to be able to attribute that cost per platform to each app. And I think that's where, then you can drive accountability and go to that team or go to that director or VP and hold them responsible for making the changes necessary to improve their efficiency. So how else do you guys give visibility to the cost to your organizations? Yeah, as Todd mentioned, tagging is a great way. That's probably the only way to understand who owns a particular resource. We also try to attribute the cost at the most atomic level, like application, and then you can just work backwards to teams and business interests and stuff. As Todd mentioned, you cannot manage what you don't measure. And I also believe that you won't get any attention to work on these kind of topics if you don't report on them. So you have to create reports that are catered to specific use cases and users. For engineers, I believe they would like to have some real-time data, anomaly detections and alerts and stuff so that they can react really quick on cost incidents. Whereas for managers and management, they could be more of a week or we cost changes or some sort of reports. We also try to sneak this into our ORMs, just make it a topic, and then also we have BRMs with our cloud vendors or even within the internal business units when they are talking about business review, then they talk about the infrastructure costs so it gets more visibility so that we can react on them. Yeah, and I'll add that once you have that visibility, it's important to get down to the level of unit cost, right? Because if AppA is costing $500 and if I'm a developer in that team, well, great, what do I do with that information? So if it gets translated to, well, now it's like for every transaction, it's 50 cents and now it's 75 cents, then I have something to work with and continuously track and improve and optimize. So I think once you have this absolute number per application, it is very helpful to get down to that unit cost and continue to track that continuously.  Can you talk a little bit more, Phil, what about idle resources and how do you manage? Yeah, absolutely. So what optimal efficiency looks like may be surprising, for instance. Preemption is very disruptive, right, or can be. And so you may have a workload that's around a completion workload, not checkpointing, it's 50% done or nearly done after running for a while and then it gets preempted because you don't have any idle resources and you need to roll something out. And so in that case, like having a bit of headroom so that workload didn't get preempted in the middle of what it was doing, may be more efficient, even though your dashboards don't make it look that way. Passive, if you have active passive DR, for instance, what does optimal efficiency look like for the passive instance? Is it 100% utilization? Probably not. Same thing with HA. So the idle resources may be an artifact of what are the capabilities of the platform you're running on, what does it offer, and maybe that slack just needs to be there for your availability needs. Yeah, and I think it's important to know what that slack, good slack looks like, right? So having that common understanding of efficiency and waste for each application and across these variety of stakeholders that we talked about, phenops, finance, engineering teams is really important. So I'm curious to hear your take, Nagu, on this. So Shopify is an e-commerce platform. And sometimes we have to reserve and scale up all the way because there's like a big flash sale coming up and that time you don't want to be scaled all the way down and you don't want your auto-scaler to kick in and your cluster auto-scaler to be kicking in and doing all of these things. So I think there are these times where you want to protect your reputation and it's not about efficiency. Yeah, so as infrastructure teams, it's sometimes hard for us to understand what the application owners actually wanna do and why they have, I don't know, primary, secondary and what are their HIE needs. So at Zalanda, what we do is, especially in case of flash sales, we have, we are utilizing CRDs to sort of pre-scale our clusters and also scale down when it is done. So this is really effective for us because we can ensure the business continuity and availability at the same time. We can also make sure that when things are done, the house is clean, so yeah. Yeah, that's a really good point and Black Friday Cyber Monday is a really big event for Shopify and that's what we do is we leave it scaled up for a certain period of time because there's Christmas and there's peak shopping season but then once we are able to monitor the traffic that's coming to the platform, the FinOps team pays close attention to make sure that things are back to normal. So I think that's why you need that central FinOps team because there's somebody looking at this every day and reaching out to the appropriate teams to take action. Yeah, I think that's a good segue into our next section of operations but before we move on, I wanted to also bring up that sometimes there's costs that are difficult to see or to manage and one area that's in particular difficult is observability, right? Sometimes developers will just emit logs and metrics without really thinking that it actually costs money to store those things and that can actually add quite a bit and it's also difficult to then attribute those costs back to the biggest offenders of the biggest loggers who are sort of wasting log space or emitting too many metrics but at the same time, there's a cost of not having those logs or not having those metrics if you need to debug or troubleshoot the platform. But yeah, with that, let's move on to operations. So we talked about Black Friday also into it. We also have different seasonal type of workloads. We have TurboTax which is really seasonal around the tax season. We have QuickBooks product where nine to five is very busy and then outside of nine to five is not as busy. So we have a mix of different workloads and I think because CPU and memory and compute resources is a big component of cost, you need to really see how can you make your clusters and applications run most efficiently to minimize costs but at the same time, provide the services that you need to. Anyone else has some thoughts on, how do you optimize your compute? So you talked about scaling up.  Yeah, like now we mentioned for Black Friday, Cyber Monday, it's all about protecting our reputation. So we do scale all the way up to the projected traffic and we actually disable auto scaling for that period of time because we don't want the system turning and like Phil mentioned, preemption and all of that is costly to the system. But other times we do leverage auto scaling. So the auto scaling all the way down, right? So we have HPA that we leverage heavily and we don't do auto VPA. We use VPA to, we are the platform team. So we use VPA to recommend what the right memory and CPU that we think should be and we make that suggestion to the respective application team using Slack channel. We are automated PRs and things like that. And then it's up to the application team because they know the specifically the nature of their workload. So it's up to them to review the recommendation and make the appropriate changes. But we leverage HPA, we leverage cluster auto scaler. So for all times other than like Black Friday, Cyber Monday this is the sort of the mode of operation for us. Yeah, so I think auto scaling is kind of a key capability. If you can auto scale your, not only your application but also your cluster up and down. That's the, for cost that's obviously the best but it does come with some disruption. So how can you minimize that disruption? I think a lot of it starts with making sure the apps can be disrupted, right? You can't launch a pod in Kubernetes and expect that pod to live forever, right? That's not the intent of Kubernetes. And there's a, for a 12 factor app there's a concept of disposability which means that any one instance of the app needs to be capable of terminating and gracefully exiting and being disposed of at any time. And so when it gets a SIG term when Kubernetes wants to scale down a replica it's gonna send a SIG term and that application needs to gracefully handle that situation and it shouldn't be a big event but not all apps are perfect and so there are some cases where you wanna try to avoid that but we should always strive to allow that kind of dynamic scaling up and down. What are some misconceptions about auto scaling? It seems like kind of like a panacea where, oh yeah, just do auto scaling but is it really that easy? Yeah, I think vertical auto scaling as you kind of mentioned to Parna as a tricky one like what is, does the vertical auto scaler have the context to know the difference between a failover state and a not failover state or is it gonna see an increase in resource usage due to a failover and say, hey, we need to change the pod spec and start a rollout right now. So the recommendations I think there's a lot to be said for checking those in as part of your regular release process. You can roll it back, it's audited, these sorts of things. The, a lot of the metrics I think get measured at, maybe their averages over intervals, right? Maybe 30 seconds or something like that and so one thing we've seen is if you measure at a one second interval like what's the P95 is actually quite a different number and so that can help with setting your limits in how you think about bursting and these sorts of things. We also kind of don't recommend VPA for our application owners but we do use it for our cluster components for our singleton applications. We could still use them because it doesn't matter if it has a little bit of a downtime but when it comes to HPA we also ask them to be very mindful of the thresholds that they are setting, right? The target utilization that they have to set and also the min replicas that they have to set because sometimes they can be a little bit more generous and then during non-peak hours then this is gonna cost a lot of money so we ask them to be mindful of those things even when it comes to HPA. Yeah, maybe let's pause there. Let me, can I see a show of hands? How many people are using HPA horizontal pod auto scaler or something equivalent? Okay, a lot of people are using that. What about how many people are using VPA or some mechanism to vertically scale your pods? Okay, a few number, a few less. How many people are using cluster auto scaler? How many are running on-prem? So there's some different challenges between prem and on-prem or in the cloud and on-prem but a lot of the takeaways are the same for both. Because even though on-prem you have to buy servers and rack them and take some time, you can't just order more from your cloud provider. It turns out that when you're in the cloud you're probably dealing with reserved instances and you have a certain committed number of servers that or instances you're buying from your cloud provider. So you have a similar kind of capacity planning issue for both cloud and on-prem. I have a question for you, Todd. I know that you've mentioned that in Intuit you have a really nice way to sort of rotate your clusters every once in a while, because the one problem that we've seen at Shopify is with auto scaling all the way down, including cluster auto scaler, your cluster can sometimes become fragmented and you can have, for lack of a better word, empty pockets of available resources that then sort of never gets reaped. So, and I know that at Intuit you have some interesting ways of dealing with it. Yeah, so that's the bin packing, right? So over time, if pods, if you have an application scaling up, scaling down, pods can get scheduled on different nodes. Some nodes could get underutilized and maybe just have only one pod on them and not really taking full advantage of the resources of the node. There's a project called de-scheduler, which would go and actually de-schedule pods and reschedule them onto another node that, and restore that bin packing. At Intuit, we actually, we terminate our nodes every 25 days or so, regardless. And that's kind of a little bit for security and compliance reasons. But it also has a side effect of forcing, all those applications to get rescheduled and kind of trains our developers that, hey, I can't count on these pods running forever. It's okay that they terminate. It's okay that they come back up. And so by doing that, we've kind of helped build this culture and understanding of how Kubernetes works for the developers. Then we also, for other compliance reasons, we need to update the AMI and the security patches of all the nodes. And so we'll also, regardless of the 25 days time period, we'll go through and rotate all the nodes to update. And so all these various things help reinforce the fact that you need to be able to scale. So that's good for operations, but also that also applies to auto-scaling, scaling up, scaling down. Yeah, we also add into it, we're looking into developing a system that takes the recommendations from VPA, taking our historical metrics that we're collecting for each application, and then using that to make some decisions and recommendations for both VPA and HPA. Typically you can't run, or you shouldn't run VPA and HPA concurrently in the same cluster. They're kind of working at cross purposes for each other. So we're trying to build a system that integrates those recommendations and then, as was mentioned, apply those recommendations through the pipeline through using GitOps so that if you change the resources of a pod, that change in resource will start back in your pre-prod environment, get tested, validate that it does work in pre-prod and work its way through the pipeline to your production environment. We don't want to just suddenly change the resources in production without being able to test it first. So that's the approach that we're taking. And we've also open sourced Numelogic and Numaflow, which is an open source project that it's a machine learning and analysis project that runs in the cluster that can take advantage of Prometheus or other types of data available in the cluster to help make those decisions. So we're looking to deliver Zetmore going forward. So, see, I think we'll move on to the next topic. So the next big area is design. We talked a little bit earlier about designing a 12-factor app that allows for disposability, right? So that's almost the first key aspect of design is making sure that your applications are cloud-native and can run on Kubernetes and do tolerate disruptions like that. But what are some other ideas? What are some other things that impact the architecture, design, or coding of applications that might impact cost and efficiency when architecting and designing your system? Yeah, so when we think of efficiency, people often think that it's all at the infrastructure layer and it's the responsibility of the platform team or the infrastructure team to take care of it. But it truly is a partnership with the application teams as well. So something that Shopify has been working on recently is continuous profiling of applications because you don't wanna just tell application developers that profile your application, make sure it's efficient and optimal at all times. So in order to reduce the friction, we have rolled out this continuous profiling feature where every application is getting profiled continuously at a certain sample rate. And using this, it's sort of very easy for them to go look at their profiles and see what their application is doing when. And we've been able to identify opportunities where there's like a static list of objects that gets created every time when the process starts up. And if you look at it, it may not seem like much, but then if you think about how many processes that you have across your clusters, across your fleet, and how many times it gets executed, it's actually quite a lot of unnecessary CPU cycles. So being able to create such tools and enable the application developers to make the right decision is also a key part of efficiency and optimization. So one of the things that we do add into it is whenever we have a new release of our platform, we run it through FMEA testing. But one part of that FMEA testing is running a load test, some known sort of repeatable load test that runs a certain amount of transactions.  And then we measure how many nodes did it take, and the application is auto-scaling through HPA. But then we measure how many nodes did it take to do that workload. And that helps us identify some kind of performance regression and performance regressions are also quite often cost regressions because if you're suddenly needing to use more nodes to do the same workload, it's gonna cost you more. So that's another technique that we've used to identify and to compare different releases. We've also used that to compare different processor types. So you might be thinking about using Graviton, ARM processor, machines, or maybe Intel or AMD. And I think based on my experience, there's no silver bullet to say, oh, this processor type is always gonna be 100% better for all applications. It really depends on each individual application. So it seems like a little bit of a cop-out answer, but you do need to test, you do need to measure, and those are the kinds of ways you can, by running this type of stress workload, you can identify that. Make a small change, a change the processor type of the node, rerun the test and then see if you're using a net, for the same amount of work processed if you're using more nodes or less. Any other thoughts about designing? I think. It goes back to I think about carefully what you're measuring, right? That when you initially measure workload, maybe the utilization looks great and then the app developer spends a month optimizing all their code and now their utilization looks a lot less good and are things, does your bad dashboard say things are better now or worse now? Cause you don't want them to say they're worse. So thinking about the holistic picture is always important. Great. Yeah, I think it's obviously as platform engineers, you don't have visibility into inner workings of the applications that are running on Kubernetes oftentimes. So it's really a partnership if you're working in a platform team along with the development teams that are using that platform. So you have to form good partnerships and see. And I see that we're running close on time. We did want to leave some time for questions, but just wanted to summarize some key takeaways. We have the three pillars of platform efficiency that we talked about. We have culture as different things here you can see on the screen, operations and design. So hope that gives you a good framework to think about cost optimization and how you can do more with less. So with that, we'll open up to any questions. Oh, sorry. One last thing, I have to do a quick pitch on people are still taking pictures. But I'll show that list again. Also would encourage you to scan this QR code, sign up for the end user meeting. Every other week we do this, almost the same kind of conversation that you just saw here. So if you're a CNCF end user member, it's a free opportunity to join and learn and share. So with that, we'll open up the mics for any questions. Oh, no, there. So my question would be have, so these institutions where you come from are quite big. So have you ever worked with spot instances on production? So have any of us worked on smaller companies because we're all relatively large companies? So yeah, I mean, we do have a healthy usage of spot instances at Zalando. We run all our test clusters on spot and we have also introduced a way to opt into using spot instances for the production as well. So yeah, it's a great benefit. You save about 70% of the cost. And for the workloads where you could tolerate downtime, then it's obviously a great choice. Question over there? Yeah. Well, thanks for the talk first of all. Very interesting, thanks for that again. So I tried to convince my colleagues, my developing colleagues, hey, could you optimize this and that? And I heard you had the same problem. They say, hey, we are paid to build features not to optimize stuff. Is there any recommendation besides showing up the cost of what an individual unit app, whatever does, to actually convince people that it's worth investing their time into optimization? Yeah, I think at the end of the day, it goes to your company's bottom line, right? So if you have more, if you're spending too much on infrastructure and cloud, that means you have less money to spend on hiring more developers or doing other useful work for your company. So I think that part should make sense to people that we can do more if we can save on this cost, we can use that same money elsewhere in the organization. And also, I think, but you're right, it's difficult to balance feature development with cost optimization. And so I think when you're doing these things, you need to approach it as something that you put in the backlog and you take up as a normal part of work, not something like coming from the side and changing all the priorities. You need to plan it as part of your roadmap to make these improvements and work on it incrementally. Yeah, and the other pitch is like engineers take pride in building scalable and performance systems, right? So scalability, performance are all like good things and everybody takes pride. And similarly, I think it's important to take pride in building cost efficient systems. And what is the unit cost per transaction or per thing my application is doing? So I think starting that conversation within the company and say like we as a team, we take pride in making and building and operating efficient services is another good way to encourage them. Yeah, that's a good point, especially if you have a dashboard that's showing the cost. Engineers can take pride in, okay, last month it was this, this month it's that. I can see a difference, right? You can really see the work that you do making a difference in the cost. So yeah, good luck. Like when framing might be when, right? Instead of saying do we do this at all, it's like when do we do this? Is it never right? And when probably isn't like when it's way too late, you know? Yeah, great, thanks. Yeah, thank you. Have a question on this side? Yeah, so my question basically is you spoke about the cooperation between the application teams and the platform team, okay? So generally, who realize the responsibility for the real cost? Like should the application team, should the platform team just mirror the cost and saying the different teams has this cost per services? Who is the responsible for deciding that a certain service is costly more than any other services and they should reduce it? Is the application teams themselves? It's difficult. I mean, at the end of the day it comes down to business value and what is your ROI? So if you have a service that is generating a lot of revenue, obviously that's allowed to spend more money probably than something that's not as important but not all critical services generate revenue directly, right? So it's difficult to really put a number on something and to know but it's something that you have to arrive with a consensus between business and engineering. So with that, I think we're out of time for questions but we'll stick around and feel free to come up and talk to us up at the front. Thank you everyone.


And so when I was given the chance to run for a seat in Congress, I looked at a few special lessons, and the one that spoke the most profoundly to me was the energy lines. And probably by yourself, I helped go through my career. Every chapter that I made, along the list, was where can I do more of a go. And I thought going a whole lot of good for this country was coming to Washington and establishing a comprehensive energy plan. No sophisticated nation like ours, no world-leading economy like ours, can continue without a progressive better policy that envelops into a comprehensive energy plan that is routinely updated and always implemented. And so that was my goal, and you are our parcel of the elements of that wonderful comprehensive plan strategy to be able to take a certain degree of diversity in our banks to be run off of this gluttonous dependency on fossil-based fuels. Oftentimes, by sending hundreds of billions of dollars a year to foreign governments that are then fighting us on a battlefield, we're paying for both ends of the war. So this whole chapter here speaks to sound foreign policy. It speaks to sound fiscal policy. Because what we do when we encourage this sort of mix in our energy outcomes is grow jobs, American jobs, that allow us to embrace the intellectual capacity of this nation and put the products on the shelf that will enable us to be more efficient in how we use our energy supplies or enable us to do renewable outputs that utilize the forces of nature in a very design fashion, utilizing the wind, the sun, the water, the soil to make a difference. It's a no-rainer, and minor. And finally, when we look at the fiscal policy sets that all of this meant, as we continue to really set energy consumers and our American energy consumers' dollars into these foreign budgets, we're just growing our debt with these nations. And our whole attention over this dependency on oil is leveraging that whole-to-sensitized complex that might end up fighting more or not in the public. So you're doing great work here in terms of stewardship of sound energy policy, fiscal policy, foreign policy. The final element would be that of environmental policy. I'm a firm believer that we have to pay attention to climate change and global warming. We see it happening around us. The extremes are undeniable. And that's the world we're looking at. Look at what it's practically like in Europe as you speak. So the world has been ripped by this phenomenon. We need to be better stewards of the environment. And how better to do that than remove these elements that buy 19 pounds of carbon pollution per gallon for every day of the oil we're using to transport us or break around this country. We need to make sure that we're getting far better than the 7.2 billion dollars of carbon emission that we use. And you do that by using the great opportunities called renewable energies. And whatever our debt is, and hopefully it grows more and more strong and grows more and more domestically produced, however that debt is cheap, it needs to be done. But we need the utmost public efficiency. Energy that should be seen as our fuel of choice. That should be our number one priority. And then we go to concepts that have us thinking outside of the mirror. Bringing us into our goals. I'm so thrilled about this. I, when I came here in my first month in office in January of 2019, was able to sit down with another of my colleagues who formed the city, the Sustainable Energy Environmental Coalition. Today we have grown that to over 50, Karen? 60. 50. 50 plus members. Second largest caucus. Now for us to adapt. Karen Allen Peter is our executive director. City is proud to be reading out the name of all of us in the house. And we use our personal experience, our professional experience to share the kind of little evidence that it's doable. It's not a political killer. It's actually an enhancer. And the second important that I want to mention to you was when I first came to town, I asked to be assigned to Energy. I was an engineer in my background. I developed energy policy for 15 years in the state so that we implemented it through my assignment at my serve. I thought that was a practical amount to try and break the rules. They weren't letting me. They had to put in some time. So I spent time on a science-based tech committee where Gene was one of the students. After that, I was in that committee. Kate Regina was our legislative director. So we're surrounded by people who truly believe in the Greek agenda. And eventually, in my third term, just before the end of my second term, I was engaged in energy and commerce. And now on the ranker, I'm a department head of economics. So I invented this because I'm here, a very willing partner. Use us. We want to hear your strongest voice, your fullest voice, your partner to make this event happen. This is American Jobs. It's America's wisdom and intellect that will guide you to a construct that allows us to cope with it. We are the best stewards of the environment. And I told the group of youth leaders the other day, countries that we want to be the right individuals. Do not let us do damage to your political generation by ignoring the past out there. The current damage broken out of the sequestration and the lack of vision and the lack of proximity is going to give us a de-set including their trouble. As far as I see it, this agenda for energy transformation and energy efficient individuals is a nonpartisan agenda. Republicans with Democrats with light and benefit Republicans with Democrats with light and cleaner air Republicans with Democrats with light have less dependency. I'm born for a new story that will then keep us honest on these conflicts. So we all stand to gain. It's great to have you here in town ready to have your attention. Please follow on us. All time of New York 20 Capital Region of New York. We're here ready to go to this agenda with lighting speed. Thanks.


Bernie Sanders, 2016. Finland and Sweden have national health care, free college, affordable housing, and a higher standard of living. Why shouldn't that appeal to our disappearing middle class? Like millions of Americans, I am inspired by Bernie Sanders. He's a true progressive, compassionate, intelligent, and full of solutions. He embodies a core principle of progressive thought, the willingness to identify problems and then solve them with government. Unlike so many free market advocates, he isn't dogmatically attached to old economic doctrine. His solutions are simple. If workers are getting paid too little, well let's raise their wages. If people don't have health insurance, then let's get them health insurance. Our society is wealthy enough to solve these problems. We just have to be smart about it. So this article is an extension of Bernie's progressive worldview. The economy is like a gigantic jigsaw puzzle and we can finally solve it with the right plan. Economics is sometimes counterintuitive. For example, natural disasters, though they appear purely destructive, can sometimes boost an economy. When stuff gets destroyed, people have to replace it, which increases spending in the economy and creates new jobs. But there's a problem. Natural disasters are unpredictable. We can't always count on them to strike at the right time. So I've thought of a solution to capture all of the economic benefits of destruction without having to rely on the weather. Let's manufacture more defects into our goods. If we make stuff that breaks sooner, we'll regularly spend money on repairs, which would be a constant stimulant for the economy. So I propose a one-year maximum on the durability of goods. We could do this for everything. Cars, blenders, computer software, etc. Imagine how many plumbing jobs could be created if every household had a self-destructing toilet that needed repairs every single month. You see, consumption is the engine of the economy. The more people spend, the more they stimulate the economy. And this is true in reverse. When people save money, they reduce aggregate demand and slow down the economy. This phenomenon is called the paradox of thrift. What our economy needs desperately is a jolt of increased consumer spending. So what's holding people back? I think it's debt. Specifically, there's student loan and credit card debt. Having a bunch of debt weighs down the economy, and I see a simple solution. Have the government pay off everybody's debt. Isn't it obvious? If all of your debt disappeared overnight, you would have a ton more money to spend. The economic stimulus would be enormous. Millions of people would immediately go out and purchase new cars, phones, homes, TVs, etc., which would again create countless jobs. Think about it this way. Millions of people struggle to pay their bills, but the government never struggles to pay its bills. It can't go bankrupt. Therefore, doesn't it make sense to have the government pay the bills instead? It's like having an infinitely rich uncle dropping $20,000 wouldn't affect him at all, but it could completely change the life of his poor nephew. I'd say there's even a moral principle at play here. The person who can pay the debt should pay the debt, and the person who cannot should not. While debt may burden the economy, it's not the underlying problem that plagues market economies. Unemployment, bankruptcy, poverty are all symptoms that have the same cause. It's all because ultimately of the existence of monetary loss, if you eliminate losses, then you eliminate the problem altogether. It's not fair that mom and pop businesses have to close down just because they're not making enough money. Parents shouldn't have to worry about losing their jobs just because their employer is losing money by employing them. We're a better society than that, which is why I propose a compassionate solution. Eliminate all the losses in the economy. Every month, individuals and businesses should be able to submit their expenses to the government. If they're in the red, the government should reimburse the difference. Think of the enormous economic gain. You would have A, full employment as nobody would have to lose their jobs. B, no bankruptcy as businesses could remain open without having to worry about profits. And C, every individual would be enabled to pursue their dreams regardless of the economic constraints. Mom and pop wouldn't have to close down just because they couldn't turn a profit. No artist would have to crush his dreams just because nobody valued his product. A society not worried about money is a society that can focus on higher goals. This world is within our reach. We simply need the willpower to overcome our inertia. Governments raised generations out of poverty. They saved children from child labor and ended the Great Depression, all because strong leaders had the courage to write down their brilliant ideas and turn them into law. I hope that progressives like Bernie Sanders will read these proposals and get inspired to turn them into a reality. Our future doesn't need to rest in the hands of greedy businessmen. We can find smarter ways to solve our problems. And yes, for anybody that was wondering, this article is satire. It was written in response to many of the outrageous ideas proposed by people on the far left during the campaign season. Some of their proposals sound nice, but are unfortunately divorced from economic reality. And if any of these proposals seemed plausible to you, I suggest starting a new investigation into basic economics with a book called Economics in One Lesson by Henry Haslett. It's a great book that will help explain why some of the ideas I just mentioned would be catastrophic for the entire economy.


The digital extenders are a medical device that is used for intubation. The goal of the project is to compensate for the loss of sense of touch when using a device rather than one's fingers by providing a user with referred force feedback through a vibratactile band worn on the palm. The classification experiment that we ran has users determine the difference between the stiffnesses of three different blocks based on referred force feedback. If you would like to learn more about this project, please stop by our poster.


I design board games and I'm a mathematician. I'm going to explain zeroth dimensional persistent homology, an essential tool in my work, through the popular board game, Carcassonne. Zeroth dimensional homology studies connected components, and a connected component translates exactly to a farming region in Carcassonne. As any Carcassonne player knows, farming can win or lose you the game. Even if you don't think you know what topology is, chances are you're using it whenever you play Carcassonne. So, in Carcassonne, a farming region is a set of points such that you can move your meeple within the set of tiles from any one point to any other point. In mathematics, a path connected component is a set of points such that between any pair of points, there exists a path between them. It's the same thing. One technique in farming is to try and add farmers to already claimed valuable farming land. In order to do this, one way is to add it to a disconnected component and then to connect them via another tile. Mathematically, what has happened when I've added this piece is that I have merged two different connected components. Understanding how these connected components evolve as you increase your space is a fundamental tool in applied algebraic topology.


The Text Block A text block is a customizable block allowing you to add images, text, links or even embedded code or widgets to blocks at the side of your course. You don't need to understand any code to use it. You can add more than one text block as you see here where we have, for instance, a block introducing the course teacher and an image of the Arc de Triomphe in Paris. To add a block we enable edit mode and then from the block draw click add block and choose the block we want, here text block. Let's add an image of the course teacher with a short message. We give it a title, I'm going to call it. Welcome. And then in the content we add the image we want. Teacher Sam has her image on the desktop so simply needs to drag it in. Moodle's editor allows you to drag and drop an image. We then click on it to give it a description. Underneath the image we can add text including making a link and then when we click save the text block displays in the block draw.


Welcome to another edition of RCE. Again, this is Brock Palin. You can find us online with our entire back catalog at RCE-cast.com Also again, I have Jeff Squires from Cisco Systems and one of the authors of OpenMPI. Jeff, thanks again for your time. Hey Brock. Yeah, actually it's been a good week here at OpenMPI. We just... I know this will go out in a couple of days, but we just released OpenMPI 1.7.5 yesterday So feeling pretty good today. Also, I'm going to be hosting at the University of Michigan and Xseed Workshop, which is open to anybody you know Xseed Award U of M student or not for OpenAAC. So those of you guys who are interested in programming GPUs, Xeon5s, or just learning different pragmas, go ahead and check us out. It's April 1st. So it's kind of short notice, but go to xseed.org and it's X-S-D-E to check it out. So that's the open accelerator workshop. Okay, so today our guests are Greg Konersberg and I'm... Greg, you're gonna have to correct me on that. Yeah, and Vic Iglesias and they're here to talk about eucalyptus. So guys, why don't you take a moment to introduce yourself? Vic, you want to start or should I start? My name is Vic Iglesias. I'm the director of quality and release at eucalyptus systems So I battle with myself to make sure the code is as good as it can be and right on time. My name is Greg DeKernigsberg. I'm the community for eucalyptus, so I care about making sure we're building the right product and making sure the open source community is a part of that process. So, you know, our focus tends to be high performance computing, scientific computing. So a lot of our listeners probably haven't heard of eucalyptus. So why don't you guys give us an idea of what is eucalyptus? What is eucalyptus? So eucalyptus is an acronym. A lot of people don't know that and it stands for Elastic Utility Computing Architecture, linking your programs to useful systems, blah blah blah. But it's the Elastic Utility Computing Architecture. That's the key there. We're basically open source infrastructure as a service platform and we do our very best to keep high fidelity with the Amazon Web Services API. So the idea is that if you want your own private cloud and you want it to be highly AWS compatible, we're the guys you come to. Now, how did you guys get started?  So how did this how did this start? Oh, let's see. I guess it was a research project at UCSB and 2000, what? 9, 8 bit, you know? Yep. Yep, that's right. 2008 and it was a bunch of it was a professor, Rich Wolski at University of California, Santa Barbara and his seven Mary graduate students and they wanted to set up their own version of the Amazon Web Services Cloud that they could use for their own research and their fellow researchers. So it sort of came out of needs in academia and they turned around a version 1.0 pretty quickly. That was a fair facsimile of some of the more basic functions of AWS and then they realized, hey, there are a lot of people who might need this and so it sort of turned into a company from there and here we are still plugging. Yep. So around that idea, there was already, as you said, AWS already existed. So there was public offerings what's the real, what was their driver for wanting to build a private cloud and why do you see people building private cloud now? I think I could take this one. So you know that the university has certain amount of hardware allocated for these projects and they receive that hardware and weren't given extra money to run stuff in the public cloud. So they had to utilize that as as efficiently as possible. So I think that was a big driver for them writing the software and we see similar ideas still where a lot of companies and users have hardware laying around and they can utilize those cores and that RAM and that storage that they have there to run a workload whether it or not it's all of their workload is irrelevant. They get to leverage the things that they have purchased in the past in order to run their current workloads and with eucalyptus you can use any tooling that you had existing for running against AWS or if you want to pick up a new tool off the shelf that was written for AWS you can point it at eucalyptus and use it equally. Do you see many customers coming to you guys or people on the on the open source portion of the mailing list coming to you because they're concerned about putting that much of their infrastructure into hands of a third party? I haven't seen that concern usually, but you know, we have a lot of land and expand type of installations where people will start with a cloud in a box which is, you know, one single node and run eucalyptus on it poke around with it and then add additional compute nodes to it. So there is the possibility to scale this thing at pretty small and large scale.  They there were a bunch of possible motivations for private cloud. There's regulatory you know, for whatever reason they can't put their data into a public service. There's cost considerations. I don't think that, you know, I think whenever you basically use any ISP or service provider of any kind you are quote unquote putting your data in someone else's hands, right? So it's just a different kind of hosting model in that way. But I think that what people do want is the same kind of self-service that AWS provides. They want that inside of their own organizations more than anything else, right? There's a story I tell called the Sysadmin's lament, right? The Sysadmin sort of has this fiefdom, has control over these resources. If you want a system you have to fill out a form, it goes through a process. The Sysadmin basically puts up these kinds of walls to make sure that they can manage the complexity of making sure that everyone has compute power. Well, the Sysadmin starts having a problem when the end user rather than having to go to the Sysadmin can just take a credit card out of their wallet and go get a testbed server at AWS. And now that that is reality, Sysadmin in the real world are having to sort of scramble and say, oh, well, how do we compete with that? And so providing the same kind of self-service capability in a private cloud is something that organizations recognize that they need now and there are lots of ways to do it. Okay, now part of the value add here, of course, like we said, multiple times is the compatibility with AWS. And you said a very interesting thing a moment ago that you can take any tool that is created for AWS and just point it at Eucalyptus and be good to go. Tell me a little bit about how that works.  Is there some central Eucalyptus server that I have on premises that I just changed some server name in the package and I'm good to go? Right. In essence, what we do is we try to mirror the API and its semantics. So we use all the SDKs and a lot of tools to verify that functionality. Once you have one of these tools, you would change, in most cases, it's presented as an endpoint. And you would change the endpoint URL to point at your Eucalyptus installation and start plugging away. All right. Now, with the Eucalyptus infrastructure, does that allow me to overflow to AWS if I need to? Like, you know, I have 100 resources and I have 150 requests. And so my 100 resources are full. Can it fill over those extra 50 into AWS? That kind of thing? That is the dream called hybrid cloud. And I think that everyone is sprinting to get to that same place. You can do that today. There are some caveats. There are things that you have to do correctly, right? You need to make sure, for instance, that you've got images that are identical, that are running in both places. And you have to make sure that you're using the same subset of services that Eucalyptus supports, right? AWS has 40 plus services out there. And we support seven of them. We support EC2, S3, IAM, ELB auto scaling. Can you rattle off the others? Yeah, ELB auto scaling and CloudWatch are the last three that we had in. Right. Hey, can you give us the one sentence description of what each of those are? Because some of our listeners might not be familiar with them. Yeah. So EC2 is what you would generally think of as the compute side of the house. So that's spinning up virtual machines, attaching volumes, which are your block device to those virtual machines. S3 is your object storage, where you would put your files, your images, your data that you want to grab as whole files rather than as blocks. IAM is the identity and access management, which goes across all the services. That's how we manage accounts and users and policies and all those kinds of things.  Auto scaling is the ability to use triggers to scale a group of instances that are equivalent, a cluster maybe in your guys' parlance, that will scale up and down based on certain criteria. It could be CPU usage, it could be disk usage, anything like that. And you can scale up and down programmatically. CloudWatch is the service that monitors your instances in order to provide a data point for auto scaling and or it's also possible that you just want to use it to monitor a group of instances to see trends over time. Okay, so you're doing this AWS compatibility. Is this kind of being done in a black box? What is eucalyptus as an organization's relationship with Amazon? Are they kind of promoting this idea of having some local, some public, or do they kind of not care? Like what's kind of their take on this? Yeah, it's completely orthogonal to Amazon. We have a relationship with Amazon at sort of a high level that allows us to explore some business opportunities together. But the code is open source. We've been basically writing a completely reverse engineered application stack based on the API itself since we started. And we don't share any code. We don't share any development resources in that way. So it's completely engineered from the ground up based on the APIs. So now you say it's orthogonal, but how much do they care? Do they see you as a competitor?  So I'm not a lawyer, and you'd have to ask Amazon that question, not us for the Amazon perspective. But from our perspective, we have every right to write code based on their API. Current legal precedent upholds that right. And it's likely that if there is any patent war to be had in this space, there's plenty of opportunities for that kind of shenanigans completely outside of the API itself, right? The API is probably the last place where someone is going to throw up some kind of lawsuit if a lawsuit is to be had. But again, consult your own counsel. Okay, so right now you guys are based around AWS APIs and being able to be compatible with them. Do you plan to support like OpenStack or any of the other kind of VMware service or anything like that to be compatible with tools that use those APIs? We have a very base level compatibility with VMware, but really AWS is our focus and will be for this foreseeable future. It's so obviously the dominant cloud provider. If you've looked at the, for instance, the Gartner Magic Quadrant for public cloud, AWS is farther up and to the right than any entity I've ever seen on any Gartner Magic Quadrant, right? They completely dominate the space. And so there's nothing in the Eucalyptus architecture that precludes pursuing other APIs at some point. But we think the most important thing is to be the very best at one thing first. And that one thing is the AWS API. What are some common scenarios that your customers come at you with? You mentioned the, I love the phrase you used earlier, what was it, land and expand? But what is driving this? Is it the stuff that you cited earlier, such as regulatory, or is it people just experimenting with consolidating IT resources? Or why do you see customers exploring the private cloud space? Well, I think there's myriad reasons why people do this. And the market is still forming itself around different use cases. But one of the ones that we see a lot of is continuous integration. This is kind of an engine that drives software development, right? So you have a fleet of servers that come up and down hundreds or thousands of times each day in order to test a piece of software. And having that process go through a manual process where people are creating VMs slows everything down.  So that's one of the big drivers we've seen for private cloud. This also provides data locality. You have your own repos locally and all that stuff and your own corporate resources that you can access. So let's talk about the technology a little bit, how you actually do this. What is the architecture of eucalyptus if I'm setting this up? What's the minimum number of machines I need to get going? What do I need to be familiar with? And let's assume I just want to set up all of the seven services you guys support. Right. So we also try to be easy to use as well and kind of come up out of the box with a lot of functionality. So the minimum footprint is one machine. That machine, obviously, if you run us in a single machine deployment, it's not going to be the most performant ever, but it does let you kick the tires. So you basically install CentOS. CentOS 6.5 is what we currently support on a machine. You load up the eucalyptus packages, configure the config files, and start the service. And you're up and running with those seven services that I mentioned. So we also have an ISO that you can download from our website that lets you have CentOS and all the packages in there and a workflow for configuration that should get you up and running in half an hour. Now, what kind of software pieces are in there? Are you reusing any notable open source packages inside or is this all from scratch or what are you doing? Yeah, we're leveraging a lot of open source software and utilities for orchestrating the VMs. So we use KVM in order to run the virtual machines Libvert in order to orchestrate the creation of the process in order to have a open source implementation of EBS, the Elastic Block Store. We use TGT and iSCSI-D. And then we run in OpenJDK JVM for our control components. So what kind of control do I have to kind of give over to Eucalyptus? Because if I spend up VMs, they need to be given addresses and host names. Does it need to run its own DHCP and all that kind of stuff too? Right.  You get IPs, your host name can be provided through the metadata service. So these VMs basically configure themselves. And the way we do do that is on the networking side, we're using IP tables for NAT and firewalling and DHCP for the DHCP process of the VMs. And then how do you exactly, I'm curious how you do this, having just gone through an AWS workshop of the ELB, the load balancer, how exactly you guys handle in that? So this is actually a VM that Eucalyptus spins up as kind of the system user, if you will, or the administrative side of the house. It spins up a VM when you create a load balancer. And that VM is a prepackaged VM that we created with HA proxy and some control code around it that basically hits our service API, figures out which instances it should be load balancing for and what configuration, and then begins load balancing for those. Now how do you orchestrate all the networking stuff?  Right, so we have in our older modes, we have these managed modes, which basically take care of all the networking for you with the caveat that you've given us at least some L2 space to, sorry, L3 space to chunk up. And that basically handles in the Amazon model, you have a private and a public address. So your private addresses end up on the L3 subnet that you've handed us and we chopped that up for each security group. And then you also provide Eucalyptus with some public IPs. These public IPs are things that are routable on your current network. So what about availability though? I mean, Amazon has the idea of different regions and availability zones. Can I use Eucalyptus and kind of set up my own like two split data centers and say like run over there run over here and ELB across them or that kind of deal. So currently we only support one region, which is the Eucalyptus region, but you can set up multiple clouds. We don't currently have any credential federation across the clouds. But what we have seen is that people will use trains to come around. People will use a separate cluster in order to provide hardware availability or to just have a separate place maybe for a different hardware profile where they can run their VMs. Now I want to go back in time a little bit a second ago because I forgot to ask this question when you were talking about it. You said use KVM. Are you guys looking into containers at all that are a feature coming up in the recent Linux kernels or is that outside of AWS and therefore not interesting at least at this point? Well, we just haven't heard the customer demand in the use cases around containers. So we haven't invested too much time. We've been tinkering as most people have with containers and docker and such. But yeah, we haven't seen anything to drive the product roadmap that way. Having said that, using Linfort allows us to easily implement losing Linux containers because we already have an abstraction layer above that. Okay, so what's the actual management of this stuff look like? Is it a bunch of Java things or can those servers be federated to provide availability or is this kind of a single service right now that then it orchestrates everything? So we can split out all of the components. We have five main components, the cloud controller, which handles kind of the services, the web services, the endpoints that you hit, the cluster controller, which handles scheduling and networking, the storage controller, which handles our EBS service, exporting a volume to OVM, and the VMware broker, which is the kind of API endpoint that we use to get to vSphere and ESXi, and then the node controller. And the node controller is the machine where instances your VMs will run. So that's the one that's running the KVM hypervisor. All of the components there are able to be set up in a highly available situation where you have a redundant component that is passive until it detects a failure and then comes up. So the node controllers we don't have high availability for, but we do allow for VM migration. So if you didn't have to take one out, you could migrate your VMs off of that node and on to another and then take that node out of service. Now, you mentioned you're referring to KVM and open source, but you also mentioned VMware on there and ESXi and vSphere and stuff. So what exactly is going on there? Why both? So, you know, we've seen a lot of enterprises have, you know, paid out licensing fees to VMware. Now, like we discussed earlier, I want to use my existing stuff. So we have to overlay on what we see customers have around. And ESXi boxes we use as, you know, a kind of node controller where we can basically spin up the same VM we spun up in KVM on a different VMware node. Yeah, the purpose of the VMware integration though is really for customers who are ready to start moving from VMware to either the public cloud or their own more cloud based infrastructure. You know, and the story really there is a waypoint along the migration path, right? That's sort of the, you know, we're not going to be supporting deeply every feature that VMware supports. I would say we have just enough VMware support to help users move cleanly from sort of the locked in VMware world to, you know, either their own open source private cloud or as a waypoint towards AWS. Going back to the storage a little bit, you said, you know, all kinds of, well, no, let me just ask what kind of storage do you support, particularly on the management and then the back end hardware itself. Do you have direct integration with specific RAID controllers or are you just looking from the Linux file system and above, you know, what's your level of integration there? We have a few different options there. So yeah, you can use basic Linux file system. We call that the overlay driver. It's basically we're going to drop files down and then export those files. That was kind of the original design, you know, back in the day, just because, like we said, people just want to drop this stuff, get it running on whatever box they have. And that's the easiest way to do that. We also have the ability to use direct attach storage.  And we also have the SAN integration. So SAN integration for NetApp, EMC VNX Equalogic and NetApp cluster mode are the ones that we currently support. Now what about backups? So does the Eucalyptus infrastructure handle backups automatically or is that something that goes outside of it or what do you do? Currently we don't do any automated backups. Generally the users are using snapshots and EBS volumes to, so EBS volumes to hold their data, that's their block device, and snapshots that are a point in time copy of that. The snapshot actually gets stored in two places on the EBS side and in Walrus. So you have redundancy there. As for the object storage, we use DRBD to replicate the object store data across two nodes. So actually, explain what Walrus is and where that comes from. I just find it funny. So Walrus, sorry, Walrus is our S3 implementation. And so S3, the nomenclature for like a folder or anything like that in S3 is a bucket. And if you've seen any of the memes where Walrus has my bucket, that is where that comes from. And you can look that up on Google, Walrus, my bucket, and you'll see the pictures of a Walrus holding a bucket. And that's where that came from. Outstanding. Sounds like something that needs to go in the podcast on the on the show page, Brock. Well, it's actually kind of funny because aren't all the buckets in like Varlib, B-U-K-K-I-T. Yep, yep. We went for it. You can't go halfway when you're going for a meme. No, you got to own it. You got to own it. Just using your marketing material sometime. Yeah, still still there today, five years later, six years later. So expanding on the storage stuff a little bit.  Can you have, can I set up like many Walrus nodes and kind of just have this, you know, infinitely large S3 equivalent service or can I keep stacking more machines that export EBS volumes? And when I like, you know, snapshot one and make a new one from that snapshot, it'll do all the right things moving it between them. So currently the initial Walrus design, Walrus itself was to be on a single node and then have replicated data that could be brought up on another node. In 4.0, which should be released in a month or so, we have the ability to use a redesigned architecture which uses an object storage gateway. Now this object storage gateway becomes the endpoint for our S3 implementation and then uses a scalable backend to store the data. So the scalable backend, in our case, that we're supporting out of the box is going to be React CS. So React CS has this ability to scale out. So you just set up your React CS cluster and point Eucalyptus at it and we'll start doing all the administrative part of putting in and getting your objects out of there. Do you plan to expand and include something like Ceph and, you know, its whole CRUSH system and then you could also use that for block devices and everything else possibly later? Right. So the design is specifically in order to support these kinds of things where some, you know, backend is, you know, scalable and supports the S3 API. We can interact with it. So we have seen requests for Ceph and Swift and others and this, the design is explicitly in order to support those at some point. So the actual S3 service and you're able to kind of, you're able to request objects by HTTP but you can make them public, private, multi-part uploads, all that stuff. Are you generally supporting all that API and that capability? Right. And in 342 we support versioning, bucket logging, the ACLs and that kind of fine-grained access control and no multi-part upload in 342 but in 4.0 with the object storage gateway and its redesign we are supporting additionally multi-part upload. Oh and getting back to EBS and the block storage capability in terms of adding capacity to, you know, either create more block devices or things like that. What are the options for scaling that and like adding capacity? So currently, you know, we recommend using a SAN for a highly, a system that needs to scale. The commodity block storage, scale out block storage options right now are not quite stable enough for us to be comfortable to put your data into it. So we do have, you know, the NetApp cluster mode driver where that allows you to scale at a pretty easy rate by just adding more disks or, you know, adding another shelf to your cluster. All right. Let's move on away from storage here. Let's go into a couple other things here. Just random question. What's the biggest eucalyptus deployment you've seen? We've seen clusters in the 10,000 core range, clusters and installations and people who run many, many clusters at that scale. So with that size of a cluster, are they mostly just using EC2 or they actually tend to use all the features and is it generally inward facing services or is it actually people running production public services on these platforms? We've seen in that scale, we've seen mostly EC2 usage and even in Amazon, that's 80% of their their bill there is EC2. So we do expect that to be the main driver. People just don't have yet an application or a use case that can leverage ELB. They're working on getting it to be cloudier and, you know, to be able to scale horizontally rather than vertically. So I think we'll see that in the coming months and years, but not quite yet.  I think that we've actually got some of that going on in the academic space, which is a where a lot of that kind of thing tends to happen. I think Indiana and Cornell are both running things along that line. I couldn't tell you any details because they used to retell us, hey, we're running HPC. You know, they don't share a lot of details with us because they don't need to. But it's a platform that's used for those kinds of applications, certainly. So actually, my experience with eucalyptus was actually on Futuregrid. They're running some eucalyptus there. They're also running OpenStack. You know, they're a testbed platform. They run a bunch of different things there, but those nodes have infinite band and the whole deal. Okay, keep, again, a direction change. You keep talking about the open sourceness of eucalyptus. What's the eucalyptus community like?  Or do you just get a random submission of patches here and there? Or, you know, what's your involvement like with both the user community and the developer community? I would say, generally speaking, you know, most of our engineering, most of our product comes from internal engineering, right? So it's a complicated product. There's a lot going on. There's a fair amount you have to know to really get into the code base.  But sort of more largely speaking, our community is essentially the entire AWS community, right? We've got, you know, lots and lots of open source tools out there that work with AWS and also work with eucalyptus, sometimes right out of the box and sometimes with minor tweaks. So, you know, most of our community contributions actually come in the form of patches to various tools in the ecosystem to make sure that they work with eucalyptus as well as they work with AWS. All right. So just out of curiosity, this is something I ask a lot of developers and I just love to hear the different responses and why, what version control system do you guys use and why? Get it and get hope because they're awesome. Okay. Yeah, I think that the move to, you know, a public and easily used GIF repository was a huge benefit for us. And also, I think it benefited our developers to get on what I like to call the new hotness, the thing that everybody's doing right now. So we're really happy with that decision to move from BZR to Git. So what are some of the challenges you see customers run into? Like, you know, they're moving to this private cloud to get the flexibility, self provisioning, all that sort of stuff. But what's some of the problems they run into when they do this? So, you know, what you're still doing here underneath everything is you're essentially right. And so I think a lot of what we see are there's some folks out there who sort of want to build a distributed system at scale. But you still have to do a lot of the basic sort of Linux system and kind of tasks to do that effectively. Right. So, you know, we see, we see a lot of folks who sort of don't necessarily know how to set up and maintain a distributed Linux system underneath. So that's one set of problems. Another set of problems is that once they have the private cloud, they're not always sure what to do with it. Right. I think that everyone is interested in private cloud, but you really have to write applications to be to some degree a cloud aware. Right. There's, you know, if you look at Netflix is a great example of how to do things the cloud way. And if you've heard of their chaos monkey, for instance, that's a perfect example of an organization that is really committed to doing development the cloud way. And, you know, every so often they just turn the chaos monkey out into their infrastructure and start blowing stuff away. Right. They blow away instances randomly to see if instances stand back up and do what they're supposed to do. That's sort of a mindset. And users have to learn that mindset to get the full value out of, you know, the elasticity that cloud offers. And so people who tend to see it as just, you know, more virtualization, they don't necessarily see a lot of more benefit than that. Right. There's, there's things you have to do to get full value out of a cloud and people are still learning that, I think. Here's another infrastructure and developer question. What language do you guys typically write in? Or is it a smattering of different things because you're spanning so many different stacks and there's different tools to use for different areas? So we have a smattering definitely. Most of the code base is written in Java. And that's the control components generally. And then we have the C components that are kind of the ones that orchestrate the node controller and the networking stack. Those are pretty lightweight in written in C. And then we have some Perl scripts, some Python for tooling, as well in there. So I know I want to ask something because we already asked about containers and you're like, you haven't had a lot of requests for it. Have you also not had much requests for just using this for provisioning bare metal? You ever see anybody do that? I want this AMI image to effectively be put directly on that hardware. I think we have seen requests for that. It does kind of break the Cloudy model that we expect for things to kind of share resources. Basically, in AWS, you have an instance type and that instance type fits within a single node and can be possibly run concurrently, many of them in a single node. So we would probably have a hard time kind of defining the type of a bare metal machine if there was a large kind of deviance in the hardware profiles of each node. But presumably it's possible. We just haven't seen people kind of go back into bare metal as much. So what license do you guys distribute Eucalyptus under? Eucalyptus is distributed under the GPL V3. And what is your value prop? Are you the paid support? How do you exist as a company? Yeah, it's essentially the same as many open source software companies. You're paying for the support model. So you can get the bits for free and you can run Eucalyptus perfectly well. On your own, but when you need help, you basically buy a support contract for us and that's a subscription. Generally, customers who pay for it are customers who need both the support and they also need an insurance policy so that they've got one through to choke if something goes wrong, if they have concerns. And the more your infrastructure, the more an open source product sits underneath everything, the more mission critical it is and the more important it is to have someone to be able to pick up the phone and call on your day. Now, what parts do you actually support? Because you're using a ton of third party software inside Eucalyptus. So how do you draw that line? Like, do you actually have to go in and fix bugs inside other software packages? Oh, yeah, the customer doesn't care. The customer is buying support for the whole thing. So we have gone in and found bugs in Limpvert when we have both worked around those kinds of bugs and sent patches for those kinds of bugs to multiple projects. So that's the table stakes for running an open source business that relies on other open source software. You have to be conversing up and down the stack and we are. Okay, guys. Thank you very much for your time. Where can people find more information about Eucalyptus? Join the mailing list and download it. Well, so you can go get Eucalyptus just by going to www.eucalyptus.com and you can see all of our source code at github.com slash eucalyptus. We've got tons of repos out there and the eucalyptus slash eucalyptus repo is the main product. And then if you go to the github wiki, you can see all the mailing lists and we hang out on IRC on FreeNode. So it's PanEucalyptus on FreeHead. Okay, guys. Thank you very much. Thank you, guys.


Hi, Brad here. This video is over abolishing operation. This is a motivating operation that decreases the reinforcing effectiveness of a stimulus or an item or something you do. So an example would be like I crave donuts on Friday. If I eat all the donuts really early Friday morning, by noon I really don't want any more donuts. Another example would be going on to a party, meeting a lot of people, a lot of social interaction, and then the following day you don't feel like going out to another party because you're tired of talking to people.