Hopefully everybody had a good lunch. His name is John Howard. He's a software engineer extraordinaire. I think you guys have already talked with him. My name is Amir Abbas. I'm a product manager at Google. Today we're going to talk about reliability. I did a talk last year in Amsterdam. This is a variant of that. We'll put a little bit more Istio spin and talk about some architecture patterns that we're seeing and some best practices. But I know it's after lunch. Everybody's had food and dessert. But I wanted to start with a thought-provoking question. Can you build four-nine services on three-nine infrastructure? And what's interesting about this question, you can take it as rhetorical or not, is the answer is not important. And I think like many thought-provoking questions, the answer is it depends. But what's interesting is that usually when I ask this question, the room is split into half. Some people think, yes, you can. Some people think, yes, you can. Even a room full of SREs, there's a split between them. And I think it really just kind of depends upon what school of thought you're coming from. So if we look at traditional ways of building software, you have data centers, you have redundant power supplies, you have compute, and then you have hypervisors, containerization, any of your applications sitting on top. It looks like an right-side-up pyramid from a reliability perspective. Each layer inherits reliability from the layer underneath. Couple of things are happening. First of all, it's very easy mental model to understand. If the computer is more reliable, the thing that is running on top of that computer is going to be a little bit less reliable, because it inherits the reliability from the computer. But the other thing is that the entire onus of reliability relies or is put on the IT team. It's put on that. And the application teams just don't have to worry about it. In the cloud, it's reversed. In the cloud, they look like upside-down pyramid, and you probably are wondering, why did we even do that? The answer is scale, first, which is written on the server. The second one is cost. It's very hard to run scalable applications on very reliable infrastructure. But it does two things. First, it flips the mental model. We have to start thinking about architecting applications a little bit differently. But it also puts the onus of reliability also on the development team, on the application team. So this is how all globally scalable consumer apps run on these distributed models. The key is to pick one or the other. If we don't assign valence to this, if one is not better than the other, I'm not here to argue that. Obviously, I think we are here in KubeCon. We probably tend more towards the right side of this diagram. But regardless of where you sit, just know which school of thought you're coming from. The biggest mistake people make when they're doing a lift and shift, let's say, to the cloud, if they're doing it for the purpose of reliability, without architecting your application, you may not get that. A lot of the times, at Google Cloud, for example, our VMs have a 2 and 1 half 9 SLA. So a lot of the times, a customer asks me, can I give you more money to give me a much more reliable VMs? And we said, no, you have to start thinking in this particular realm. So we're going to talk about reliability. And you can't really talk about reliability without talking about failure domains. And before we even talk about failure domains, I really just sort of back up and talk about, what are the needs of a service? And I think every service sort of uncontroversially needs these four things. So every service needs to be resilient, both to known and unknown failures. So if something goes down, it should remain alive. Every service needs to be scalable. You want to run it on many, many computers and clusters and zones and regions. Every service needs to be manageable. You want to be able to get telemetry and SLAs and those types of things. And every service needs to have some way of developing a policy framework that you can enforce. And when you start thinking about the footprint of this service, so let's say you're just running this in a single cluster, you can always get two or three of these things. But it's very hard to get all four. So in this case, let's say I'm just running this service, a front end, as just a single deployment, maybe even a single pod. It's running in a single zone, in a single cluster. Yes, it's very, very manageable, because it's a single pod. I can manage that. It's also probably very easy to secure, right? It's easy to manage. I can think in those terms. But it's not scalable, and it's not resilient. So singletons are not good. So how do we adapt these new patterns of distributed systems without losing the manageability and without losing policy? And that's sort of the problem we're trying to solve. In cloud, at least, we think of failure domains. We simplify it in just these two realms. So failure domains are either regions or zones. Zones are akin to data centers or campuses of data centers that are co-located. And regions are simply collections of zones that are geographically near one another. Yes, there are other things like Kubernetes clusters, or VMs, and databases, and so on and so forth. But everything resides in one of these two locations. And if you think in those terms, and if you think these are the things that I want to go resilient against, then it's a lot easier. Within those regions and zones, you have products, or you have services, I should say, that are running, depending upon where you're running, in which cloud, or data center. And they have their own sort of domain. So VM, for example, is zonal. So it relies on the zone that it lives in. If it goes down, it doesn't affect anything else. But a zonal outage can take down a zonal resource. Kubernetes cluster could maybe span zones. It could be a zonal cluster. It could might span zones. And what that means is that within a region, a Kubernetes cluster can withstand a zonal outage, one or more zonal outages.  When it comes to Kubernetes clusters, the largest footprint of a single Kubernetes cluster today, at least, it still remains a region. So as customers start getting into multi-cluster architecture patterns, this is where sort of things start to get a little bit more challenging. And Kubernetes folks are doing a really good job in making multi-cluster a little bit easier. But I think with Istio, we've got some really good patterns that we're doing. Yeah, so to architect things reliably, we often need multi-cluster. And there's a lot of good features of Istio that can be leveraged to help out here. And we're going to go into some more concrete examples of how you can use these for different architectures in a bit. But some of the main features Istio offers here to help is we've got service discovery. So Istio can connect multiple clusters. We saw a bit about that in some previous talks and even multiple different networks and allow communication between all of them just like as if it was a single cluster. On top of that, it can also do traffic management. That includes all the standard Istio things like canary rules or header manipulation, routing. You name it. But there's also all sorts of aspects that are specifically related to multi-cluster, such as location aware rules so we can keep traffic within a zone or within a region, automatic failover between zones, things of that nature. We've also got security and policies. This is not super traditionally related to reliability, but if your application is compromised by an attacker, for example, that could hurt your reliability. And the policies and security that Istio provides also carries over across multiple clusters, which differs a lot from something like network policy, which is generally only applying to a single cluster. And finally, we have observability. Again, observability is not the thing that's giving reliability, but it's very important to be able to observe your system so you can actually understand if it is reliable and if you're seeing errors and what those errors are and why. And Istio can provide a single pane of glass to observe all your applications, even if they're across multiple clusters. On observability, I get asked a question a lot that I have services running on multiple clusters, even multiple regions. I just want to see just a global metrics view of how everything is running before I can dive into individual clusters. I want to introduce this concept of archetypes. We talked about it in last KubeCon as well. But just to give you a model of how to think about running distributed applications, we start with archetypes. Archetype is nothing. It's just a fancy word for an abstract model. So it is infrastructure agnostic, and it is also product agnostic. So here you start thinking about things like, what is my replication strategy? What does my DR flow looks like? What are my recovery points and times per application? You start looking at an application and you decide what are the things that it needs? What model would it fit under? You haven't even chosen a runtime or a database or a CICD strategy, any of that.  From there, you get into architectures. And architectures just basically takes that archetype model and then start implementing products and services in there. So this is where you say, for runtime, I'm going to use Kubernetes. For service mesh, I'm going to use Istio. And for CICD, I'm going to use blah, blah, blah, for databases. That matched that archetype. Archetypes are universal, so you can switch between runtimes and or between services, going from Istio to a different one or going from Kubernetes to cloud to some sort of a serverless technology, your archetypes remain the same. On top of your architectures is your applications. And applications are constantly evolving. Obviously, the application, you're adding features to that. Also, it's evolving in the matter of footprint. And what that means is where the application is deployed. Again, you might start with a singleton. You're running the application in a single cluster. And then you may decide that it needs to be multi-regional, and then it needs to grow globally, and so on and so forth. And you have to start thinking about it. And then when you start managing that application, we manage that using SLOs. And this is a quantitative way of measuring the health of every service. It's contracted between your consumers or anybody that is using your applications. We've published this paper. This is the only link, I think, in the entire presentation that if you want to write down. It's a bit.ly slash app archetypes. There's tons of archetypes that goes into much more detail on how it is. But from a reliability perspective, it really is a variant of zonal, multi-zonal, regional, multi-regional, global, hybrid, multi-cloud. We're going to not talk about hybrid and multi-cloud just for the sake of time. And we've picked five of these archetypes that we believe majority of the applications will fall into one or another. And we're just going to talk a little bit about those. All right, the first one up, kind of skipping over the most basic archetype of a singleton, is this active passive zones. And what this looks like is we take the full application stack and replicate it into a separate zone. But we don't send any traffic there. And if there's an issue, then we do kind of a manual failover action and start sending traffic to our replicas. So this is kind of a manual operation, generally, when an issue is detected. Traffic is manually operated by a human SRE over to the other zone. And eventually, when the issue is resolved, they can bring it back. So when evaluating this archetype and others, like there's a few areas we want to look at, it's kind of what are the operations that a human has to do when we face errors? It's what kind of failures are we resilient to, like is the zone going down, is the region going down? What kind of availability we can get? What is the cost? What's the complexity? So in this case, the complexity is fairly low. There is in the typical flow, we just have kind of a single zone. There's nothing fancy going on. The main complexity risk here is that the passive zone becomes stale because it's not actively tested at all times. The cost is also fairly high because we need to fully replicate the entire application. I can be improved a bit by having fewer replicas of some of the services. In terms of what failures we're resilient to, we can survive the one active zone going down. But if the entire region goes down, then we'll have issues because the passive set is in the same region. One thing that's kind of unique about this approach, though, is for licensed software because the passive applications aren't serving any traffic. Generally, it's possible to actually reduce cost in some sense because the passive zone doesn't need to have the license activated until the failover occurs. So we have a bit of math here. If you want to go into way detail about all the math, you can check out the talk from Amsterdam, I believe. Or last year's. Yeah, but the general trend here is that by having two different zones that are kind of in parallel, we kind of aggregate the availability of them, and we're able to get more reliability than any single zone. So here and in future ones, we're showing SLA numbers from GKE and Cloud SQL just to give concrete numbers. But you could find similar from any other cloud provider, I assume. In terms of implementing this one in Istio, it's actually generally you're probably not going to use Istio for this. And the load balancer would be something external Istio. But if you were using Istio to do this as a load balancer, Istio has a variety of ways to route traffic like this. So one example would be to use the locality of load balancing to specify which zone the traffic should go to. When there is an incident, that would go be changed by the SRE to split from 100% in zone 1 to 100% in the other zone. Yeah, I was talking to, I was having drinks with an SRE. And I said, what do you call an action that you do when something fails?   That basically just means do you need to do anything or not. This is the second archetype. The first one was active passive. This one is active active. Or in this case, it's active active active, depending upon how many copies you want. A little bit of refactoring is needed. This is where you start to separate state from statelessness. Maybe not license applications, maybe not COTS application, typical consumer web front ends, web applications work. Cost actually goes down a little bit here. If you want to do the math, if it's a three zone environment and you want to design for a single zone failure, so imagine that you're serving 50, 50, 50 in all three of these. So when one zone goes down, you still have 100% of the traffic that can be served. Global load balancer automatically fails over. In GKE, it would look very similar to the last one. Only in this case, you are splitting the traffic from the initial load balancer across all three clusters. The numbers remain the same. And in the Istio implementation, it would look just like a very simple destination rule with locality turned on. You don't put any manual failover in there. And Istio automatically determines the closest zone and then moves to the next closest and then the next and so on. In this one, we did want to mention with Istio, there's a couple of ways to do this. So in the archetype picture, there's a concept of an internal load balancer between every single layer because we're in multiple zones. So you need to send it to some sort of a load balancer that can send traffic between one zone and another. With Istio, you have both models. With the sidecar model, you don't need that internal load balancer. The sidecar does act as that internal load balancer. So if the front end and zone A can talk to all three backends, A, B, and C, and it'll just failover automatically, or if you're using something like an ambient mode, then it would look very similar to with an internal load balancer, which would be your waypoint proxy. Extending the first and second archetype, we slap these together and we have this active passive region. Very, very similar to the other two, except that instead of failing over to a different zone, we're failing over to an entire different region that would be doing the zonal deployment there. So very similar on both of these. Typically, how this would be done in terms of the top layer would be something like DNS doing kind of geo-wear, or not geo-wear DNS that's coming later. Do the failover policy at the DNS layer, or there could be multiple tiers of load balancers. So the benefit of this one is that it can survive an entire region going down because we failover to the other region. So again, the math here is quite nice, very high availability because with two regions, it's quite unlikely that both regions go down at the same time, I hope. By the way, just to be clear, these numbers here are the maximum that really can be achieved here. That's what I was going to say. There's many ways that you can have lower reliability. So in terms of the Easter implementation here, there's generally not actually much Easter involvement here. Generally, each of these regions would be entirely separate meshes. They would not know about each other. So in the Istio multi-cluster terminology, you probably would not link these clusters together at all. All of the routing into which cluster or which region or which mesh should be used is at a higher level outside of Istio. And then once the mesh is chosen, then it can go into the standard Easter routing, which is kind of more like the previous archetype where we would use the locality load balancing to keep things within the zone as much as possible but failover when necessary. Yeah, I believe there was a talk this morning that actually talked a variant of this archetype. This was just interesting. Whereas previous one, there's no sort of ingress and egress control, you can just manually failover. And maybe the manual failover is because the data is not fully replicated and you need to be able to manually decide when to move things over. This one is isolated regions where you just cannot move serving traffic over between regions. So it's very, very similar to the last one. In this case, you can create two separate trust domains for Istio in each of these. This is for regulated industries. When I talk to EMEA customers, this comes up quite a bit. I want traffic to be localized in a specific region. In this case, you have a choice, whether you want to just archive data in a different environment, but the serving is always served from one. And then you have the manual control over DNS. So imagine that you're sending your EMEA traffic to EMEA regions and APAC traffic to APAC regions. So if something goes down, it's only going to affect half of the traffic. And yes, you can use each other as backups if you'd like, but that just totally depends upon, it totally depends upon the regulation. And the Kubernetes slash Istio world, it would look exactly the same as the last one, except the Istio trust domains will be separate. Another thing I want to just point out that, yes, the numbers have gone up, but you can also see the complexity goes up, right? So there's cloud DNS or some sort of a DNS, and then there's regional load balancers, and then there's six clusters to manage those types of things. So your ops cost kind of goes up, but you do get a much higher reliability. In Istio, it would look extremely similar to the last one, you would have two istiological meshes and two different environments that don't know about each other. And if you do need to do any cross cluster sort of a thing, you would just use ingress patterns, otherwise it's identical to the same. All right, the last one up is the global pattern. I think this is the most fun and interesting one. This is like something like YouTube or Google search or any other large scale consumer application that's highly available, highly replicated, we run as. There's a lot going on here. So kind of at the top of the stack, we have a global anycast load balancer that can accept traffic from anywhere in the world and kind of send it to a load balancer instance that's close to the user so they can reduce latency. Next up at kind of the bottom, we would have a global database, something like Spanner or Cockroach DB or something similar that can be accessed again from anywhere in the globe, but generally we'll do some awareness of the region so that it doesn't increase latency cost too much. Then we have multiple regions with multiple different clusters and each service is replicated across all these different regions and zones and clusters. And at each hop from say front end to back end to multiple tiers of applications, the traffic can flow between any of these. And so this is something that ECGO will help facilitate and we'll generally this will be done in a zone or location aware way such that we're not sending traffic from US to Europe to Asia back and forth and have crazy latency costs, right? But we're able to if there's failures or some services are going down in one zone we can easily fail over. So the cost here is higher but it depends on exactly how much higher based on how much you're replicating the data. So in this case we have the front end is replicated across six different zones, three regions, six zones, six clusters. So that replication cost is fairly high but we also get a corresponding high availability out of this. The complexity here is also again fairly high but again it comes with a lot of nice properties. In particular, unlike some of the other modes there's no fail ups required, right? When a particular back end goes down ECO can detect that and automatically fail over. So we don't need to page estuary in the middle of night and say hey go trigger the rule to start sending traffic to the new zone, right? That's all automatic by ECO. So again this looks something like this we'd have six clusters here in parallel. We can choose at each hop along the way which cluster which workload to go to based on the health and latency of each hop. Okay so as you can see the complexity goes up you get more reliability but also the cost goes up so then people ask like well how do I use these archetypes to my service? So the way we think about it is a service belongs to one archetype. A service let's just say for the sake of this conversation is a Kubernetes service. An application can have a collection of services and you can put services in different archetypes and then you're probably saying well why would you wanna do that? And the reason is obviously to save cost but you also wanna do that to architect for what we call graceful degradation. So let's say in this example service D is just some low priority service. Maybe it's an email service that just sends an email out and if it's down it's not gonna affect anybody you're just gonna get delayed emails. The rest of the application can continue to work you can save some cost. So when you build an architecture it would look something like this with multiple services. Here you have service A running as global maybe that's your front end, service B running as regional, maybe that's some catalog that's regionally significant and then service D that is maybe not as important. Yeah so some of the benefits of this as mentioned like because we don't necessarily need every single service to have the highest amount of availability which increases costs and increases complexity. Sometimes it's the right choice to run in some of the kind of lower tiers of the archetypes that are less available but they're simpler to operate and they have less cost. And what that does is free up a lot of capacity for other services to run. So if we had some fancy new AL or AI or ML workload that we wanted to try out we've got room in these green clusters because we didn't make service B and service C more available than we actually required. I just want to say for the record we waited 23 minutes to say AI. All right so let's sum it up. Thank you. Let's sum this up. So basically what we're saying is most people start with architectures. They immediately like oh we need to put stuff on Kubernetes and we need Istio and we need Argo and those types of things. We recommend you starting with archetypes and you can by the way use archetypes at different dimensions. Maybe they're security archetypes and regulatory. We talked about it from the lens of reliability. So look at it from that particular perspective. Read the paper. I'll show the link here again. Then you can go into the architectures. We think Istio and multicluster Kubernetes match made in heaven works really, really well. It's been working well for a number of years. Kubernetes is doing a great job but you can use Istio to accomplish majority of these as we talked about it. Individually, Istio on a single cluster is great but Istio on multicluster is a very, very good experience. Here are the links. You can learn more about multicluster on Istio.io or bit.ly slash app dash archetypes if you're listening to this on YouTube later. I will say that out. Otherwise, thank you so much for coming to our talk. Thanks everyone. One minute. Do we have time for questions? Maybe we have five minutes for questions. Hi, Lynn. I guess we can. All right. We will be here for five minutes if people have questions. I think we've got one over here. Oh, we have questions. Archetype five. No, but I could hear when you shouted. Shout it. I'll repeat the question. I think you're good now. As of legal, right? Bunch of laws make us to make weird decisions. When you disable the service in that region or in one of these, you get an imbalance in the other regions as well. So do you have like an approach to solve this imbalance because it's complicated to solve it now because it doesn't support advanced load balancing weights or any other thing? Yeah, it in many ways depends on the application. Ideally, so there's always kind of this trade-off if I have a small number of local instances or a large number of remote instances, which one should I send it to? I know that from the location standpoint it's better to send to the local ones, but I may overwhelm them and it may be counterproductive, right? So there's a few ways to handle this. Istio has recently changed its default to be kind of the least request load balancing, which to some extent helps solve this by being kind of latency aware. The faster service will finish request faster then we'll have less requests so it will have traffic preference there. Some other tools are kind of outlier detection, for example, so you could say if the latency of this service has exceeded this amount or if it's starting to return errors, that's no longer healthy even if it's passing the traditional Kubernetes probes. And so that will kind of be evicted from the pool and encourage sending traffic to the remote endpoints. Those are kind of the two primary tools for that. So I mostly had a question about, say you have multiple control planes in the same region so you're not as much worried about the regional failures or even zonal failures, but you're more worried about someone doing something to a control plane and you just want to make sure your control plane fail over, well, not necessarily fail over, but you have redundant control planes or redundant clusters even within the same zone. I suppose that's also where you would make your meshes or your networks connect together in a multi-master setup. So even if you're not going to multi-region or even multi-zone, you would still be able to employ the last diagram in the series for that. Yeah, so that's a very good point. So we didn't really talk about running the Istio control plane here, but there's a lot of similar decisions to be made of how many of them you run, how do you replicate them per zone, per region, per cluster. Istio allows you that flexibility and probably could be a bit more opinionated on suggestions on which one you should run in which cases, but in many ways, like running your own application, it's about the trade-off of complexity to availability requirements on which choice is kind of best for you. I did want to add one thing to this. So the reason we talked about two failure domains, regions and zones is most people aren't even doing that. I know they're kind of, the reason is because you can go fairly granular and you can start doing resiliency and redundancy for every single component, infrastructure component, control planes, API servers and stuff like that. There's a concept in SRE called generic mitigations. And what that means is that if there is a problem with the control plane, you just route the traffic somewhere else at a bigger, at a higher failure domain. And zones and regions just fall really well within that. So if you just architect for zonal failures and regional failures, everything just gets subsumed within that. Those are great points, but let's say there is a failure within a sub component within that zone. You just simply assume that it's a zonal failure and then you just move over and then you can fix that and figure out what's going on. But I don't think it's worth your ops effort to just make every single component duplicated. That's question. About stateful applications. Do you have the supplies to stateful applications as well or only to stateless applications? Yes, we didn't hit on the stateful part as much because we wanted to, first of all, we had 20 minutes and we wanted to stay closer to the Istio. If you read the papers, it will talk about exactly which database technologies do what. Certain archetypes are actually architected based on the database technology. For example, the active passive ones because we know the replication time might be too long where you can't just automatically move an application over and you also have to do the replica switch over to make sure that that becomes now the main. So yes, the paper talks a lot more about databases and stateful stuff as well. We just didn't hit on it. Yeah. The paper is pretty good too. It's very approachable. The paper is, yeah. It's actually a part of the Google Cloud Docs so that link just takes you to like an architecture center and yes, I mean, feel free to go to it right now and you should, and it's very approachable in a sense that it's divided into I want to build zone elapses. What are some of the archetypes? I want to build regional elapses. What are some of the archetypes and what type of applications? What's my database strategy? And it doesn't tell you any product so you can use them with any products you like. Yes, I can go back. What did I do with the one slide back? I think we'll be in the back. I think we're out of time but thank you so much. We'll be in the back for more questions. Thanks everyone.


That filler sure is pretty That's a hot mug guy Hey guys, this is my review for John Wick 3 Parabellum I have been a big fan of the series I went into the first film thinking it was going to be garbage and I was very very heavily and Pleasantly surprised and the second film while not exactly having the same gusto in terms of motivation and story is the first one Still had some absolutely fantastic action and that tunnel gunfight is still one of the best gunfights put on film And I would say the last ten years if not more so with the third film We are at the precipice now for John's Mortality as his ex-communicado is put into place and he is literally fighting for his life This film is a great stunt work feature There are so many different variations of stunts not just with humans, but with animals with vehicles with Objects with situational matters. There is so many variations of stunts Fights gunfights and various other things in the film The creativity is constantly throughout the film in terms of giving you guys a different fight scene a different gunfight a different action scene However, and this is a big however It doesn't mean anything for a large point of the film John Wick is trying to reach a person Who can help him with his ex-communicado? He is trying to reverse this along the way He meets a variety of characters including Halle Berry who while isn't in this film as long as I would have thought She has a great role in it and the dogs have some fantastic stunts all the people who are associated with John Wick back in New York are facing the consequences of Essentially aiding him and killing a member of the high table. We see Ian McShane going through this We see Lawrence Fishburne going through this and we see Angela Houston who actually has a bit of a connection to John's past Which is a little cool Like this is the one little tidbit about his character that him and Halle Berry has one a little bit too But really we actually find out a little bit about John's history with this woman So he goes on this journey. He goes through all these perils He goes through all these issues of why we see New York kind of falling to the wayside of its own punishment and Then he has given a means.  That's honestly how the film goes. It essentially is a filler level a great looking filler level and an entertaining filler level But it's still a filler level because when the film ends it ends exactly how I expected this movie to be essentially Is nothing except a really great fight scene and I'll give them credit There's some really great stunt work guys in this movie. It has two guys from the raid It has the dog and the guy that Rama fought at the end of the second movie and They admittedly are moving in slow motion Keanu Reeves is a fantastic actor for what he's doing the guy is plus 15 He can still move far better than I can however when he's fighting these two guys You can tell that he is setting up. He's waiting to get hit. There's shots that should have been edited tighter and Admittedly these guys are really moving in slow motion in comparison for Keanu So that's a little bit like sometimes the shots are much better and they hide that But I still don't understand why they don't have the editor from the first movie She did a fantastic job with the first film the guy who edited the second film edits this one as well And he's an okay editor, but admittedly he's not as good as her. She was much much better It was also really cool to see Mark as a villain as a adversary to John in this film Mainly because I worked with John before I worked with him on a Hallmark movie It was pretty cool because one time we were in a gym and he was like all right Let's get ready and he started doing these calisthenics and these flips and these stretches that was like holy crap this guy is plus 50 and He is a much much better shape than anyone else in this entire area right now It was great to have a villain that while was a threat was also someone you kind of enjoyed their presence for that Was something that was missing really in the second film except for maybe commons character So in the end John Wick 3 is a fun time But in terms of ranking this film with the other two films it is definitely the third best It has great action, but it has little substance It has great choreography and setups and variety But it doesn't bring any of that to its story Because essentially this is just a setup for the fourth movie Which I didn't even know they were going to be making one until I walked into the theater And I read an article saying that they were excited to make a fourth film So that essentially kind of ruins the aspect of this film and it really makes it as a placeholder One of the guys who I went and saw the film with had a perfect summarization It's a fun time, but when they make the fourth film you will not have to have seen the third film To really understand what was going on that is essentially just how little of anything happens in this movie I still enjoyed it. I'm still saying I enjoyed it But I don't think I'll ever care to watch this again unlike the first two movies So in the end I'm gonna give John Wick 3 Parabellum A4 out of 7. Let me know what you guys thought of the film I'd love to see your comments and about not only the fight scene But also the story and just the idea of what you think might happen with the fourth film Otherwise, hope you guys enjoyed this review. I'll see you guys next time Thanks for watching the video It's been a while, but I'm happy to say the click is finally getting back together in an all-new movie Thanks to a successful Kickstarter campaign, but we are still asking for your support You know, you can't get more money unless you offer questionable favors Yeah, unless of course those favors involve the ladies guy by support. I mean getting the word out guys Oh, well, couldn't you find it better means than this guy only seems to talk about a supernatural or hold a coffee mug real Awkward why didn't you ask a Kardashian or something? Yeah guy get in with the ladies guy. Hey, he's trying to help out Like you've been trying with Kimmy Burton. I've seen job of the hut finish a marathon faster. Yeah guy You're a massive slug thing guy To see any and all updates about the upcoming undergrads movie be sure to check out and like the bring back undergrads Facebook page and with any luck. We'll see you guys soon


I'll ask it just the why diamonds question, why the high class assets? Everledger started in 2015 and we believe that there is not a platform provenance in the world. We've seen applications come into the marketplace for procurement systems but there is no platform provenance. And then when you imagine what a provenance platform would entail, we have to go deep within understanding the material science of an object. We've seen a lot of activity in the space with provenance now, from tracking fish to I think avocados and pigs I think is one, but diamonds have some underlying constructs that are really important. Firstly, every diamond is unique in the world and to be able to capture the uniqueness of that diamond in a digital twin was able to be done, not purely from a blockchain perspective but a combination of other technologies. So we take high definition photography, we can enable the identity of that through stress mapping, we can also use resonant ultrasound, there's a whole bunch of other science that has to be combined to take that digital identity into the blockchain. And we're not talking about just putting a label on it, unlike some other things for example a motor vehicle would have an RFID tag or a serial number. So that's one reason why. The second reason is the value construct around diamonds of course has enough yielded margin and enough yielded value to offset the requirements of the development of a platform. So there's a value construct there that actually made a lot more sense. The ability to track a diamond that might start at a rough of $10 million in a wholeness of stone through to a diamond in the marketplace where that's still in the thousands or tens of thousands is a very different economic proposition to tracking a tomato that might be worth less than a cent. Who is still buying diamonds? Because I've heard some media reports, you know, like millennials don't want to buy diamonds for engagement rings anymore, things like that. So who's the diamond market still aimed at really? Well I think it's aimed at the same, you know, the same set of people that it's been aimed at, you know, generations ago. There are certainly life events that want to be celebrated, whether it's a marriage, whether it's the birth of a child, you know, these events are still happening. People are still being getting married and people are still having babies. And so that's certainly one part of the consumer market. Millennials are asking themselves a lot of different questions, you know, they are involved heavily in conversational commerce. So they're making certain aspirational choices. They're really looking now around sustainability and where do items come from? That's the reason why Everledger is providing the service that it provides to the market. Certainly we are seeing synthetic stones, you know, being manufactured. And of course there are other life events that might not be as memorable as a one-time marriage or a birth of a child. But there's still celebrations that are happening in the space of a new job. You know, a career choice, a PhD that's being celebrated. And we see that potentially some of those lesser but just as equivalent celebration topics that people could choose fashion-type jewellery and that's really where synthetic stones sit. So I don't think it's all or nothing. The light switch isn't going to be just turned on or turned off on one day. We still have healthy growth within the industry. You know, whilst natural mining of stones is retracting back, certainly we still see that diamonds and luxury jewellery. I mean, I'm just looking at you. You have probably 15 pieces of jewellery on you today. And I would guess that every woman that walks in the door has some form of luxury piece that's important to them that would have a natural stone in it. That's true, yeah. These are all from my grandparents. There you go. Yeah. Exactly right. Yeah. No, it's true. I guess my next question has to do with your work in part of the Hyperledger community and working with IBM. So how is Everledger part of that? How do you guys all work together? So Hyperledger, in the first instance, open source and we make contributions into that community. It's a very important construct. The relationship that we hold with IBM at an enterprise partnership level serves two roles for us. The first role is clearly from a secured cloud deployment environment with their HSBN, I think they called it in the first instance. I don't know what they call it now, actually, blockchain cloud or something. And so that's one part, you know, the opportunity for our customers, the big names within the industry to choose whether they wanted to host it within an IBM infrastructure or an AWS infrastructure sits well within the core constructs of what we do. And the second part is we work very deeply with the R&D team and I'm appointed as a board advisor for the blockchain advisory group for IBM. So helping to lead and understand from a perspective of fabric where that needs to grow, where the engineering could be placed to suit certain market applications. But from a perspective of service offering, you know, we service the industry directly because we've all come from industry. We know and understand the diamond industry and the jewelry industry intimately well from a technology standpoint. So what industry did you come from before you were working for Everledger? I'm a software engineer. So I've spent 25 years working in and around software. And in fact, in the mid 90s, I worked in RFID, which is radio frequency identification. So at the silicon chip and inlay level. So I understand the constructs of that technology and did a fair bit of work on skater devices, which is remote connections on machines. I've had a couple of companies and been OK. I've never had to ask dad for money. So I've sold them at the right price. But, you know, for me, I invested some money in a jewelry business. And that gave me some intimate learning of of industry, generally, more than just the experience of a consumer to buy a diamond. So it's the combination of all of that together.  So when you have world experience, you get to understand that when technologies come together like this, it's not necessarily just about the technology and how cool it is. It's about addressing a particular type of set of challenges within industry. And that's exactly what we probably did different very early in the stage than maybe a couple of others in the industry. So we knew exactly what problems were to be solved, the challenges and the economic pain points and then the economic benefit to applying it. And then the rest is sort of history. You know, you see where we are now in the market, we have seven locations. We service the larger names within industry and we've extended well beyond diamond. So Colour Gemstones, Enrilds, Ruby, Sapphires, Tans and I. It's important to us. We look deeply at wine and wine authentication. So how can we apply, you know, all of the geniuses that we've built in terms of our engines with machine vision? How can we apply it in wine? And, you know, last week we were awarded 2018 Tech Pioneer for the World Economic Forum, particularly around the work that we're doing in sustainability. So not just about tip the grower for the coffee that we've drunk, but, you know, more deep constructs on how do we enable actual artisanal, small-scale mining communities? How can we enable that so that marketplaces can be built with fair pricing? And that's really the big work that you'll start seeing, start seeing coming out of Everledger in the next sort of 12 months to two years. I was going to ask, are there any new asset classes you wanted to add? I mean, the wine I hadn't heard, that's interesting. How did you guys come up with that? How do you track the wine? So, I mean, we were first tapped on the shoulder with a wine expert. Her name is Maureen Downey. She's the Sherlock Holmes of wine, heavily involved in the US with the FBI, and which, you know, parts of the investigation led to the the conviction of Rudy Cornier wine, which was about a $200 million fraud. You know, there's certain types of knowledge when you're a wine expert like Maureen is that you can gain from just being within industry for 30 years. So most of these experts within industry have sat within industry for 30 or 40 years. But the reality is there's no underlying system or construct to capture, you know, each of those methodologies or systems or or insight. And that's a part of the work that we've been doing from a technical standpoint. So we were just asked to have a look at it and we said, sure, that makes sense. But for us, we're really interested in physical assets. How can we digitise those at a material science level and some objects won't be able to easily or affordably? And then we're interested in rising asset classes. So I guess when you look at our portfolio, diamonds, jewelry, watches, wine, collected automotive cars, maybe even helicopters, I guess it's motivated by me. The things I like to eat, to drink, to fly, to wear, handbags, you know, selfishly, why not be motivated by that? For us, I think it's around a rising asset class. It's items that we also believe that might want to have fractional ownership. We're interested in futures. So diamonds, you know, there is no financial products on diamonds at the moment. There's no ETFs yet, no futures market. But you could imagine that that would be the next sensible step for the industry to walk towards. So but it can't enable that unless there's pricing and transparency. And so, you know, when you think about the positioning that we hold today, we could become just an important one part of an ingredient towards that new economic modeling for diamonds. And speaking of fractional ownership, I read a few weeks ago that an Andy Warhol painting in the UK was sold on the blockchain for fractional ownership, so people actually own parts of the painting now. What do you think about that? Does that make any sense?  If you can think of it like that. I don't think that they would explicitly own the pixel of a painting. So it's like saying I have a fractional ownership of a property. It's unlikely that the positioning of that ownership would unless it's under a strata title environment, that it would own this window or this door within the house. So I don't think that that is the construct that was being built out. We've certainly seen fractional ownership with diamonds. You know, one of our very first customers was the Singapore Diamond Investment Exchange, which has looked at how do we take diamonds, put them in a fungible sort of form? And then how can we enable the ownership of those? So three carats could be made up of three stones or you could have a three carat diamond being owned by three different people. And I think this type of fractional ownership is likely to occur across many different asset classes. The trick is going to be how can you ensure that the economic value of that investment is yielded correctly out over time? And that's that's difficult to know how you price a market. How do you provide for indexing around around those around those types of asset classes because they haven't been seen as a traditional asset class with fractional ownership? So until we get the construct where banks, insurers recognize that these are potential or asset managers can start to recognize these as investable vehicles, then I think it's likely to be potentially too early in the stage, you know, to have far reach. If you could explain simply to our viewers how you track a diamond from the mine to the store on the blockchain. So diamonds, you know, to understand the process is actually a highly complicated supply chain, but it's also relatively consolidated geographically and within each parts of the process. So diamonds are mined within a number of countries in the world. And those countries probably end up on on, you know, two hands and they're sort of 10 to 12 major mining companies globally. Diamonds will be extracted out of the ground. It will go through a certain set of washing, bay processes and sorting. And there is machinery that actually exists inside of mines. And so you can capture, you know, the the diamonds in parcels. Those diamonds already they're a part of the Kimberley process are already packed within tamper proof containers. And so those tamper proof containers are recorded under batch and the stones are recorded within those within those parcels. The diamonds as they cross borders are then attracted with a certificate called a Kimberley certificate, which is also recorded as a part of the chain as well as the invoice. So the commercial construct of the of the pricing and then diamonds will end up in a place called Surat where they're cut and polished. And those parcels are then opened under camera. And so the parcel and the under camera opening is then matched back to source. Again, with the matching of the appropriate trade documents. Then a diamond will actually go through to the next stage, which is where it's cut and polished. And typically they're actually polished with inside of one control mechanism in a factory that also have connected devices. So we're a quiet IoT enablement company where we connect not only the opinion of experts, but actual machines across the network. And these machines scan. They use a whole bunch of different technologies, whether it be high definitions, spectrography, resonant ultrasound, light refraction. There's a number of different methods and each of those methods are captured. So think of it like how I would be able to know you as an identity. I know your name today. I know your height, your weight. With the photograph, I now know facial recognition about you. But I don't know medically how you are made up. So you would need to have a blood test for me to know your DNA. I take a piece of your hair and I test that as well. Maybe your teeth records also give me some indication. So it's all this forensic layer that we get involved with. And the forensic layer is already captured within the industry, within laboratories, particularly if they're certified diamonds. So we have capture points there. We underpin part of the work in digital certification with the GIA, the Gemological Institute of America, to understand where this sits. They grade 80 percent of the entire world's diamonds are graded by them in their laboratories around the world. And so that's another very large toasted capture point. And then diamonds will eventually out into the retail network. And then the retailers will again check the diamonds through machines that are connected and then sold to a consumer. And so the entire story is captured and told then to the consumer. So there's really no room for fraud in everything you just told me. There's always room for fraud. Of course there is.  So not all diamonds are going to be captured on a blockchain. There will be some diamonds that might start on the blockchain and then all of a sudden they do not eventuate into the retail network, so their history will not be persuasive.  So if I go to sell my second hand car and I don't have a, you know, a Kelly Blue Book record in the U.S. or a Carfax or in Australia, there's another name for it. Then ultimately the value of that second hand car is then in question, right? The provenance is not there. So we'll see the same thing happen in other types of assets. So there's always ways to defraud systems and there's always ways to subvert certain constructs. But there's a value reason why people would be deterred away from doing that.  It's up to the government's controls to help to reduce that, not eliminate it. What do you think about Bitcoin? Do you invest in Bitcoin? Do you think that it's something that's also going to be used in the future, like blockchains becoming more prevalent now? I mean, I was fortunate enough in 2010 to buy Bitcoin. And so that was, believe me, it wasn't an educated decision. It was just one that kind of felt like a de-risk at the time. So why not buy some? And so, yes, I mean, I certainly believe that in a cryptocurrency sense that, you know, Bitcoin, of course, is the grandfather of the industry and has been there since day one at its inception. I think that it has survival and longevity beyond just today's, you know, current competition that maybe is seen between Ethereum and Bitcoin. Where the entire future goes for that is not really for me to say. We don't, you know, we do not see ourselves as a crypto company. And to be honest with you, we don't even see ourselves as a blockchain company. You know, we're building a platform of provenance to help with help with transparency in conflict and opaque markets. And we want to build an ethical trade platform. So whether we start to see diamondback crypto currencies come into the space, which is I think we have seen a few, at the end of the day, we still need to have a good anchor between the crypto and the real world. And that's where we sit and we will probably forever sit. Coin Telegraph, like, subscribe and hodl.


Hello, can you hear me? Can everybody hear me? Yes, hi. OK, welcome to Runtime. I want to say real quick what this talk is and what this talk isn't. What this talk isn't is a talk about the architecture of the CF Runtime. If you came here for that, that's not what this is. I'm sorry. What this talk is is a talk about the process of the CF Runtime team and how you can contribute and how you can get the most out of being part of a really excellent open source project that has a really excellent team behind it. So let's get started with a little bit of an overview. So we're going to talk about our development process on the team, how we collaborate with other teams, our continuous integration and deployment, how we interact with the community at large, how you can contribute back, and hopefully at the end we'll have a few minutes to open this up for some questions. So without any further ado, we're going to talk about our process. Some of this is a day in the life of a Runtime team member and some of this is overall pivotal process and why we think it's a good thing and why we're talking about it. So we'll start with stand-up and pairing. Every morning we do a stand-up. The stand-up is our morning meeting. It's where we talk about what happened. What did we do yesterday? What did the organization do yesterday? We have two stand-ups, big stand-up. That's organization-wide. Everybody at Pivotal comes, or in person or digitally, as it were. And we say, do people need help? Do people have interesting things to share with the rest of the organization that might not be shared otherwise? And finally, we talk about events. Little stand-up is a very similar format. We say, what did we do yesterday? Is there anything that people need help with? Is there any interesting things to share? And finally, we set up pairing. Pairing is our process. Every developer has a partner for a day. Sometimes longer. Kind of depends on the team. And the reason we do this is because we've found that the standard idea where I write some code, and I want this code to be included in the project, and so I give it to Dan. And Dan looks over it and says yes or no. That's inefficient. And there's a better way to do this. So we have this idea of a pair. A pair is a cohesive unit of engineering work, or the ability to do engineering work. And the idea is that two minds working together are actually greater than two minds working apart. And we always get two eyes on every line of code that's written. And if we don't, there's some other things that we can do to make sure that the code is worthwhile. So we will submit pull requests if we are soloing on code. But ultimately, the idea is that pairing creates this concept of two eyes on the code, and it makes sure that whoever's writing some code, if they miss something, the other person can catch it. There's lots of little safety nets in the pairing process that help make it a more efficient way of writing code. We think this is a good thing. IPM. IPM is part of our agile process. Agile is just a word that doesn't really mean anything. We like to think of ourselves as being able to iterate very quickly on our code. So we break code down into chunks that we call stories. We put them in a tool called Pivotal Tracker. If you are contributing to Cloud Foundry, you should be aware of Pivotal Tracker. It's this really nifty tool for organizing an agile code base and an agile project. But the idea is that we come up with all the things that we need to do for a project. And we can look at them and see them. And our product manager, Yui, can organize them into a timeline that makes sense from a product standpoint. And every Monday morning, we go into a room and we talk about, what are the next two weeks worth of work that we have to do? How hard is it? What are some edge cases that we might not have thought of initially? And we say, all right, we think that these things need to happen in this order. A great example is in the mornings, we had a new track called Context Paths. And this was a request from a foundation member that we have a new way of routing to different apps. And I won't get into detail on that. But the idea is that we suddenly had a bunch of stories. And we didn't know how much work they're going to take. And the idea is that before somebody actually starts doing work on them, we have to have a conversation. Every story is a conversation about the work that's going to be done. So we have this conversation. We come to a conclusion that adding this functionality to the router is going to be harder. And we mark that. And I say harder. What does that mean? What is harder? It's going to be harder than something that's easier. What does that mean? All right. So the idea is that we're not actually coming up with quantifiable ways of measuring hard. We're coming up with an idea that we can say, we understand that this is going to be a difficult thing to do. And part of that understanding is the idea that you can't quantify difficulty. You can't say, this is 10 units of hard. And that thing is two units of hard. And so this other thing is much harder. You have to say, this is relatively more difficult. And we understand this because we had a meeting and we had a conversation about that unit of work. This conversation is really important. The next thing that we do, and this is part of our overall process, is we test drive everything. And one of the really things that helped me understand test driving was the idea that your tests aren't a way of verifying that your code works, but are rather a way of specifying how you plan on writing the code. And that's one of the reasons why in our spec in Rubyland, tests are called specs. It's a specification. It's you saying, this is how I want my code to eventually work. And you're not necessarily coming up with implementation-specific details about the code when you write a test. But instead, what you're saying is, this is how I expect the code to result in certain functionalities. And I think that's a really good way of talking about test driving because the idea is that we're allowing our tests to drive how we end up developing. And this is something that Ansi talked about this morning and is really excellent talk about Diego. He touched upon this feedback loop, the idea where you write a test and that informs how you're going to write some code. You write some code and that informs more information about what you really need to test. There's lots of places where this comes up. So we came across this really interesting edge case in this routing situation where we came up with a new way of storing routes in the go router. And we had a test for the new way. And then we kind of discovered that we hadn't covered an edge case where if you hadn't put the route in with the leading slash, the go router wouldn't accept it. But we didn't have a test for that. And we saw that the go router was actually accepting things that shouldn't be accepting. So we had to backfill a test for that. And it's one of these ways that as you write code, as you discover more about the code you're writing, it informs how you should be testing it. And as you write tests, as you think more about what is the actual functionality that you should be testing, it informs the code that you're writing. The feedback loop feeds back into product and engineering. As we finish stories, it informs the product team or our product manager Yui about really what are the next stories. And as she writes stories and informs us about how we need to be doing them. So that feedback loop is really important. The last two things, DevOps. DevOps is the idea that we are our operators. If you saw Andrew Clay Schaffer's talk two talks ago in this room, you had this understanding we're not picking up CF release and throwing it over a wall, as he put it, to operations. We're deploying it. We deploy it many times a day.  And we manage that. And we manage deployments of CF release all across the organization. Other people manage CF release all across the organization. And we help them, too. The idea is that not only are we writing the code for CF release, but we are managing deployments of it and having a really strong, deep understanding of how CF release, the code that we're writing, how that works in a deployed production environment. That's super important to our team, because we get direct, immediate, actionable feedback about the code we're writing by the virtue of being the people who are deploying it. Finally, Retro is something at the end of the day that, under the week, I'm sorry, and we talk about the week. And some of it's technical. We talk about things that went wrong technically. And some of it's emotional. Things that we thought, oh, we had this interaction or something upset us about something that happened in the workplace. But it's a way of reflecting on the work and the workspace in the week and talking with your coworkers about how can we not only iterate on our code, but how can we iterate on the process by which we develop Cloud Manager? And that's super important, because not only is our code fluid and not only do we want to iterate quickly and develop quickly Cloud Foundry, we want to make sure that our process always supports that goal. And that means we have to iterate on our process, too. So Dan is going to come up. Dan Levine, he is a contributor from IBM. And he's going to talk a little bit about our development process from the point of view of somebody outside the pivotal organization. So thanks, Zach. So when Zach was talking about this stand-up and the entire group gets together, IBM and a lot of other companies who really aren't following this process kind of take a look at it, and we're like, OK, how can we do that? So I live here in the South Bay. I'd like to not commute two and a half to three hours every day to go up to work. That's kind of annoying. And we are an open source community. And so we leverage a lot of tools that the open source community feels comfortable with. So stand-up and pairing. Every day for our team, we jump on a Google Hangout chat. It's just there. I get context on what happened yesterday. I can bring up my own issues. I can help other people resolve their issues. So that's really important to keep that flow and have everyone communicating. Besides that, when we actually do our TDD and we do our DevOps and we actually develop, we use another tool, Old Screen Hero. It allows us to actually sit down with our pair and go through the same user interface that you would have if you were actually working side by side with your pair. It works pretty well. The only thing that's a little bit of a bummer is when you're not working there with your pair, you can't have the kind of communication that you might have with another pair. Working with your pair is great. It's fine. You can hear them. When you're working with someone else that isn't your pair, then it kind of breaks down a little. It's a little bit harder. Your pair is then relaying information back to you. Hopefully, nothing gets lost in that translation. And so then the last thing when working remotely is our IPM in retro. Now personally, I like to go up, beginning and ending of the week, because that's when those two meetings are. It's a lot nicer to be in person, especially for retro. And you get beer and snacks, so a little bit of incentive to go up there. But being remote, it's not too bad. And so with that, we use kind of the same techniques when collaborating with other teams.  Video, it's way easier to see someone's face and communicate with them. And the other tool that we have for non-face-to-face communication is Slack, which is just an internal messaging tool. But something that's really important about the runtime team is what we call an interrupt pair. So every morning, two people or one pair is randomly selected as the interrupt pair. So anyone else inside a CF that has any issues comes and literally interrupts the interrupt pair. And these people are responsible going to help them, taking a look at their issues, sometimes debugging it, sitting down with them. And kind of understanding if there really is a problem, do we need to add a story? Do we need to get this work done? Or are they just not configuring something right? So they really take care of those kind of issues. Another thing for Slack, it's pretty great for simple yes or no questions. Hey, could you take a look at this for me and just what do you think kind of idea situation? You'll get simple questions. Someone's like, hey, we can't seem to get our app staged. And it seems like the container is being deleted every time it fails. Is that the intent behavior? And we're like, yeah, that's the behavior we would expect. So then maybe they say, well, how could we go and debug this issue? So obviously, the logs, blah, blah, blah. But sometimes it requires more than that. And that's when the interrupt pair is really handy. You could do something insane, like fork the build pack, put a sleep in there, go into your container, and start inspecting the files, which is just one example that had to be done. Another form of communication that we have is we have a pretty sweet bat phone. Now, I can't give you the number to the bat phone. Only a few people can have that number. Yui, who's basically the mayor of Gotham, has the number. So she can reach us whenever she's remote. And that's just another tool that's pretty easy for communication, where you don't want to go through a giant text list for, here's all my issues. Pick up the phone and chat. And then lastly, how do other teams actually contribute code? Well, most teams, obviously, they just directly commit to Cloud Foundry. They're there. Most of them are in San Francisco. It's pretty easy. Once in a great while, there's actually PRs made. It could be someone from the team who is soloing for the day. Like Zach said, we want two eyes on everything. So someone who solos for the day, they'll make a pull request. Ask a pair the next day, hey, could you review my code? Them and the pair can re-review the code that they submitted, something like that. And then all this code that goes through, actually goes through our continuous integration pipeline. And our CI is really important to us. We do TDD for everything. So if we're not running the test, what's the point? So our CI is always running all the time. All of your changes are going to start off by going to a pre-staging environment. So we named all of ours after sauces. So we have Tobasco as our first staging environment. And it's a pretty lightweight deployment of CF. It's good to see that, hey, since our last code changes, we can roll everything and upgrade everything in a clean fashion. There's minimal tests that run against it. We run our tests as our teams to say, OK, this seems to be pretty good. And once it gets through that phase, it goes into staging. And staging is where things are much more interesting. Staging is a semi-small-scale production environment. And we have everything work in there. We have services hooked into our staging environment. We have Diego running on our staging environment. And it's also the place where most of the PMs go for acceptance. So it's the place where, hey, look, we pushed the code. It's working there. Everything says it's green. And then the PM is like, OK, well, let me go take a look at it. Let me play around with it on the staging environment. And it also serves as a great place where we're all coming together. And then we can step back and say, did we break anyone else? So maybe we pushed some code that accidentally broke services.  Well, we care about those. Could you please fix the code that you've pushed accidentally? And then besides that, we have smoke and stress testing. So we do smoke tests to make sure that our environment is always up and running. When we do these staging rules, we have little apps running in there saying, hey, we're rolling everything, but we can still talk to you. CF is still running. Fine, we're happy with that. Occasionally, we do a bit of stress testing against our deployments. It's not all that common. But when we want to check backwards regression, we'll do some stress testing, maybe potentially spin up little environments that are isolated to test a specific feature. But for the most part, we really don't do all that much stress testing. Another thing that's important is our environment differences. So as the runtime team, we are kind of the meld of everything. We manage the giant CF release repo that everyone's contributing to. And so we literally deploy on OpenStack, vSphere, AWS. And we manage all of these different deployments. And they're pretty useful to us. There's small things that people do all the time that this isn't going to have any consequences. And these are where we catch those problems. For example, there was a Ruby gem that we're using called Fog to manage the back end of connecting to AWS or NFS for OpenStack. And someone said, these symbols look a little funky. Can we just change them to strings? Sure, it's Ruby. You can do that. Totally broke OpenStack. Whatever the gem was doing underneath, for some reason, strings and symbols were not handled the same way. And so our OpenStack environment exploded. And that was good. This is why we have trust in the CI. So one of our environments went red. And so when you hear, oh, Cloud Foundry, it's safe to fail, it's because we have these giant CI environments that we feel comfortable that we're safe to fail. Red doesn't mean, oh, something is wrong. Red's a cue to us to be like, hey, let's go investigate why this went red or why something didn't pass. So red isn't a problem. It's just a great way to be sure that you have pushed all your code. Everything is working in there. And when everything is green, then you're golden. And with that, I'm gonna throw it back over to Zach to finish up on contribution. Hello, can you hear me? Yes, good. Hi. So that's our process. That's what we do internally. This is how we work as pivots on the runtime team at 875 Howard Street in San Francisco. I see lots of familiar faces, but I see lots of unfamiliar faces. And first, I wanna say the community is great. It's awesome that we get requests from the community. We get code, we get issues, we get conversations that inform new features. All of this is awesome, especially code, because it's work that we don't have to do. So yeah, pull requests, come on, keep them coming. But how can we help you understand our process in order to help your code and your issues get solved quicker and get pulled in quicker? So that's what we're gonna talk about today. So a little bit of information about how we work through issues and information that comes from the community. We triage stuff in Tracker. So again, all of this stuff is completely open. You can take a look at our Tracker project and see all of the stories. You can see all of the features, all of the bugs, all of the chores that we have in there. You can't edit them, but you can see them. And we really want you to. We want you to take a look and understand Tracker and try to get a better understanding of how we iterate on Cloud Pantry. For the most part, issues come in, bugs come in, conversations and pull requests come in. And Yui, product manager, takes a look at them and says, this is this first line of defense. Is this reasonable? Does this belong to runtime? Sometimes we get a bug in, you know, VCAP dev, that's the runtime mailing list that really belongs in the Bosch mailing list. That's our first line of defense. Yui's gonna say, is this reasonable for runtime to be looking at? She puts it in the Tracker backlog. We have a little section for the community. The community pair, which is kind of like the interrupt pair, only they get interrupted by the community. We'll take a look at that stuff. The sources of these things, like I said, they're gonna be pull requests on GitHub, issues on GitHub, email sent to the mailing list. And we also get foundation requests that sort of go straight to Yui, and I don't really understand, there's like a kind of a secret chip implanted in the product managers and the foundation has a little remote control and then she puts in a story in the Tracker. Some feedback we got when we were beta testing this talk was it needed to be funnier. So could you all laugh a little harder? Great, okay, okay. So, thanks, Jim. Yui puts foundation requests in the Tracker and we take care of them. How can you contribute issues and get the most out of asking us these questions? Send an email to the mailing list, put something on GitHub issues, but what does it really take for us to help you? The best thing that you can give us is your manifest and your logs related to the issue. This is super important because we get stuff all the time and we're so desperate to help. We truly are, and I don't mean this in a sarcastic way. We genuinely want every issue that comes to us to be solved and we want people to come away from their interaction with us saying, man, that was so cool, the runtime team fixed my problem. They're the best. We want that to happen, right? How can we do that? Well, we're gonna ask you for your manifest and we're gonna look through your manifest. We're gonna say, okay, what might be missing here? What were their changes in the last major version of CF release? That's a big one, is there's a new thing added to the manifest, a new property, or a property changed and somebody didn't pick up on it. Their new manifest didn't get that change and now LoggerGator crashes every time you try to deploy. That's huge. Give us your manifest, please. Same thing with logs. Your app is crashing, but it's not crashing in a way that makes any sense. Probably on the DEA, and you say that to us. You say, it's on the DEA. Well, we're gonna say, can we have your DEA logs? So send us that stuff. And a last little bit of information. Be careful, please, when you send us your manifest because it's got private keys in it and it's got your passwords in it and if you mail vcapdev, manifest with your private key, it's not just us who gets that. It's every other single person in the world who knows how to go to code.google.com or whatever the Google Groups mailing list is, right? And it's moved now, but that's not important. So don't mail us your private keys. That's a thing you shouldn't do. But in the meantime, yeah, send us your manifest, send us your logs. Give us as much information as you can. Sometimes there's this balance between sifting through a lot of information to try to find a problem and feeling like you don't have enough. And we'd rather you err on the side of more information because that just helps us solve your problem quicker. Moving on, code. How can you get your PR merch? Well, so we talked about test driving. This is really important and we talked about trust in our CI. Well, we have this trust because every single feature we write comes with a really comprehensive suite of tests that we've written before we wrote the feature. This means that we can be really relying on our continuous integration to tell us very accurately whether or not our code is working. And that means that if we get code that doesn't have comprehensive test coverage, that's a problem. Because it means that we can't really pull that in and feel that the CI is still as reliable as the standard that we hold it up to be. So I skipped a bullet point. I apologize about that. But is it tested? This is super important. Please write tests. And if you feel like your tests don't make a lot of sense, if you're not sure how to test that, ask us, boy, do we love talking about test driving at Pivotal? Man, that's like, if you say test driving next to somebody from Pivotal, an hour, you're at least in that conversation. It's great, right? We love testing. So please ask us because we can't wait to help you. And the more people who understand test driving, the more people who are able to really confidently contribute to CF, only good things can come from that. Did you run acceptance tests? We don't expect everybody to write an acceptance test. That's an end-to-end integration test. That's pushing an app and expecting that app to work given the feature you wrote. We do not expect you to be writing these regularly. But did you run them? Did you make sure that you didn't break something that was already working already? That's a big one. So please run the test suites before you contribute a pull request. Because sometimes it's just a little tiny thing. And it's so much easier on you, we don't want to waste your time. We don't want to send you back to fix something that you could have just caught. And it's not like we're upset about, oh, you could have just caught it. That's not what I mean.  So we don't want to waste that. We want to know why you made the change. That's important. And the other thing that's important is, is it generally applicable? Does this apply to every possible consumer of Cloud Foundry? Or did you write a solution that's very specific to your installation? If you did, can you generalize it? Can you take a solution that applies to your specific multi-tenant deployment of Cloud Foundry on OpenStack and apply that to people on vSphere and AWS? And in the future, CenturyLink Cloud, can you do that? And that's really important, because we're a foundation and we are platform agnostic. We can't pull in OpenStack specific contributions. But we can pull in generic contributions. And we want to, especially if you wrote it. This code, we didn't have to write. The last thing is just thinking about, what does this break? Start a conversation with your pull request. Pull requests are great when they're a conversation. What does it break? What might it break? And have that conversation. And sometimes the conversation is, it won't break anything. So we'll pull it in. Sometimes the conversation is, this might break stuff. Let's go back and fix some problems. That's really important. And like I said, every story in our backlog is a conversation that we promise each other that we're going to have. And as a community contributor, we'd like to make that same promise to you. So I want to show you the anatomy of a good-looking pull request. This was from Mike Youngstrom. Raise your hand. Yeah. So this came in. This is fixing an issue that Mike also filed. And it's got a really great comprehensive test suite. There's tons of other code in here, by the way. But it's got this nice comprehensive test suite. It's got a great little PR information about what actually does this pull request do. It gives us some hints about where to look for possible problem areas. This makes our job and our life so much easier. And I've got a little link up here that I'm going to click on. Oh, that didn't work. How does this, can I? Well, we'll skip it for now. But the idea is that that link was going to be the full issue. The issue, I'll describe it, had a long discussion about the problem that needed to be solved. And it went back and forth. UEE was involved from the product standpoint. We had engineers involved from an engineering standpoint. The idea is that every issue, every tracker story, every pull request is a conversation that we have about how to make Cloud Foundry better. That's super important. So without further ado, do people have questions? You. It depends. So we have this picture of our board. This board shows four pairs.  But this is the board. So the idea is that at any given time, we might have anywhere between two and four pairs. But right now, I think we're at three pairs? We're odd. We're at three and a half, actually. So being odd, it can be a problem because it means somebody's always soloing. But that's just something we deal with as people rotate on and off the team. And then we do have the morale boosting pair in case somebody needs some stuffed animal love. Right there. Yeah, so IBM kind of faced this problem. We had people working on the runtime team from London, the East Coast, and here in San Francisco at one point. So there's one person in London. So he's going to solo for part of the day. Eventually, then the people on the East Coast woke up. And he would solo with one of them for half a day. His second half of the day. Once he left, the two people on the East Coast would then pair up and continue on for the next half of the day. And then right at lunchtime, if they stayed together, their day is over. But in the morning, they could split off from pairing with each other and pair with someone from San Francisco for half the day. Just keep it rotating, keep all the new faces.  And the two pairs that were pairing here in San Francisco then combined and became one pair again. So we obviously do the best to try and accommodate, like everyone is always pairing. But there are definitely cases where someone is going to have to solo and submit pull requests for a review. Again, does anybody else? Yeah, I'll get back to you. I'm sorry, I just wanted to give somebody else a chance. So performance and quality indicators, a lot of that comes from our continuous integration environment, where we have stress tests.  And those graphs are showing us go-router round trip, for instance. That's one of the things that we're measuring. How long does the request spend in the go-router? How long does it spend in the DEA? How long does it spend coming back? And what's the networking overhead just due to networking, where it's not really in anything? Does that make sense? Oh, yes, I see. Yeah, we do. So like I said, we give points to a story, which is how much work a story is going to be. And I put scare quotes in there because it's not a quantifiable thing. But the idea is that at the end of the week, we can see, oh, you got through 17 points. And again, that doesn't measure something specific. But if the next week, we see you got through five points, then that's showing us, ideally, we didn't change how we were estimating how hard a story is. But that's showing us, oh, suddenly something slowed down. Or we didn't spend enough time working on real features. We were too busy fixing bugs. So those are not so much indicators of performance of us as individuals, but as a team, how much work were we actually able to get done on forward progress on Cloud Foundry? And a little bit of a note here, we only point stories that we think are features. So bugs and chores, cleaning up the code base, don't get pointed. Which means that if you do a week where you were only fixing bugs, you're going to see that number of points at the end of the week drop. And that's a good feedback to say, wow, we just spent all this time fixing bugs. Do we have a problem where we're introducing a lot of bugs? It's a huge, really great indicator of what are some problems in your process if you see points drop precipitously. And likewise, if you see points skyrocket, that can also be an indicator that you might be overestimating things. Does that make sense? In the back. So it kind of depends. Everybody is able, allowed to write a story. And like I said, a story is really just a promise to have a conversation. Anybody can start a conversation. So we write features as developers. Most features come from product. That's just because product is picking a direction. And we are as developers part of that. But that's product. Lots of bugs come from developers. Chores are another type of story. Those come often from developers, but also from product. If, you know, Yui says, we want to create this new environment, that might be a chore. It might be a feature if we think that that's really going to move forward our progress on Cloud Foundry. But it's a collaboration. Like I said, anybody can start a conversation. Right. So there's a couple of ways to move forward on a situation like that. So something goes red. Take a look. All right, what was the commit? The commit was just a few lines. All right, well, let's go in there and let's fix it. Easy enough to do. First is, hey, there is this really large commit. We're not really sure what broke it. Someone made a pull request that is 10 commits long. Let's just revert the commit and get the pipeline green again.  There's some issues here. There's all the logs that we grabbed. We try to be helpful for their debugging as well. That really doesn't happen too often though. And then you sometimes get the really odd commits where we were rolling the UAAs in these new bumps and a migration didn't complete in time. So at that point, your whole system's just kind of like borked and really requires someone going in there fixing the code up and then putting a commit on top of that to make sure it doesn't happen again. So there's lots of stuff that can happen where we have a huge pile of commits. The build goes red. We're trying to fix them.  So this means that services, loggergator, all these different teams that are really important have commits piling up as they do work because we're not telling them to just go home. And now, OK, so now they've got these commits piled up. They've got 10 commits that they're ready to go. And our build's been red for a few days because we had some wacky problem with flakiness. And this is an actual thing that happened where it was four days of a red build because we just had so much flakiness in our testing. And we just had every other team had 20 commits ready to go. We actually ended up doing a staggered commit into CF release where we kept the build red and we said, OK, services, commit one at a time. OK, lamb, do your commits. OK, Diego, do your commits. And we watched as each of those teams made it through their commits, made it through every step of the build into staging. And we're green. Then we said, OK, now we're back to normal again. But it's a dangerous thing where if you don't decide, roll back now, get it green ASAP, you're going to be dealing with not just your backed up work but other teams backed up work. How do we handle them? It's part of the story. The router should be able to handle this much in the constraint. And that's part of that conversation we have. And we did this came up recently where we were doing performance testing against the router. And it was a chore in this case. But it was mostly just saying, well, actually, how much does the router handle? And then we discovered that a new commit to the router made it so that the router couldn't handle, couldn't route as quickly as we wanted it to. And so we had a new feature which was, OK, implement this new data structure for storing routes that makes the router much faster. But it's really just part of that conversation we have. We want the router to be able to route a request from outside to an app.  And as part of the conversation we have, one of the bullet points in the story is the router must be able to handle and such requests in a second. And that might just be a constraint. And as part of the acceptance for that story, product might actually test that, use Apache Bench and see how much can the router actually handle. Thanks all for coming. If you have any more questions, come on up.