Hey folks, if you've watched any of my episodes in the past, you would be forgiven for not realizing that R is first and foremost a statistical programming language. I mainly use R to make pretty visuals and to summarize data and do all sorts of data manipulations. But it's also very powerful for doing data analysis with statistics. And don't get me wrong, I definitely use it for its statistics functions. I just haven't kind of displayed them here in these episodes. Well, in today's episode, thanks to a loyal viewer of the show, I am going to build a regression model relating the amount of snowfall to the amount of precipitation, right? And so my understanding is that they take snow and they then allow it to melt and then they record how much precipitation there was for the day. And that's how we're able to get precipitation data that's on kind of an equal footing all over the year, okay? And so there's a rule of thumb out there that I've seen on various websites that it's maybe 10 inches to one inch, right? So if you have an inch of rain that's basically 10 inches of snow and vice versa, I think the viewer said it was maybe 12 to 1 or something like that to where he lives. Doing a little bit of digging, you see it's also dependent on temperature that depending on the temperature, the snow will either pack more or less, right? So that's what I want to look at today. What is the relationship between the amount of snow and the amount of precipitation? And can we model that also taking into account the temperature? As always, I'm going to do that in our studio. I've got my RStudio session fired up and ready to go. We are going to build off of the code that is in code local weather.r. This goes out to the NOAA website and pulls down data for a NOAA weather station that's close to where I live that has 130 years worth of data. If you want to get that script so that you can plug in your longitude and latitude for where you live and you can kind of work along with me, I strongly encourage you to do that. So go down below in the description. There's a link to a blog post that gets you everything you need to get up and running. I'm going to go ahead and run this script to get the data and also load tidyverse meta package, as well as lubricate and the glue packages. So the data frame it brings in is called local weather. So that brings in our table that's got the date, the maximum temperature for each day, the amount of precipitation and the amount of snow. I'm going to go ahead and remove all of the NA values. So we'll do drop NA. And if you don't give it any arguments, it looks across all the rows, all the columns in each row and removes the row that has an NA in any column, we then see that we've gotten rid of those NAs. Also, I'm only interested in those days where there was snow, right? I'm not going to calculate much snow in the middle of July when even in Michigan, it doesn't snow. So we'll go ahead and do a filter to do snow greater than zero. And so now we get those 4,830 days over the last 130 years where there has been snow in and around Ann Arbor, Michigan. So I'm ultimately might look at the day level. But for now, what I'm going to go ahead and do is look at the annual aggregate to see if there appears to be any kind of association at all. We talked a lot about correlations and ways to view those a few episodes back. So to aggregate by the year, I need to go ahead and extract the year. So we'll go ahead and do mutate year equals the year function on the date column, giving us this column year, right? And then I can do a group by year, summarize, and I will then do prcp equals sum on prcp, that'll be the total precipitation for the year. And then snow will be the sum on snow, right? And so that's a little bit weird because the snow starts in the preceding year, right? So snow might start October, November, and go into March, right? So you kind of span the change in year, but whatever, we won't worry about it so much. We're kind of looking at a pretty broad view here. This gives us the year, the precipitation, and the snow. What we've seen in the past, we can look at this a variety of different ways. So I'll call this prcp underscore snow. And then I'll take prcp snow, and let's make a facet out of it. And so to do that, we need to get prcp and snow in the same column. So we'll do pivot longer, minus year, that gets us the three column data frame, ggplot, aes, x equals year, a y equals a value. And then we'll do geom, a line. And facet wrap, tilde name, that's the default name for the column names, the column that holds the column names, right? And then we'll do n call equals one. And of course, we need to do scale equals free y to free up that top y axis. And in here, we've got that 1891 data, that's the very first time point, which is basically down at zero for the amount of snow. So I'll come back up here. And we'll have to do this after we created the year. So I'll do filter year not equal to 1891. And I probably also don't want 2022, a year not equal to 2022. All right, so that looks better. And so here, I can see something going on, right? So we can kind of see, you know, it's a little bit flat to descending up to 1960 for both snow and precipitation. And then after 1960, they both go upward. So another thing we've seen, we could do is go ahead and plot it as a scatter plot, ggplot, aes, x equals, let's do prcp, y equals snow, geome point. And there's a pretty obvious correlation, right? Let's go ahead and add in the year as a color. Yeah, and so certainly, as we get these lighter shades of blue, we're moving into the present period. And we can do core dot test, prcp, snow dollar sign, prcp, comma, prcp, snow dollar sign, snow, my fingers feel fat, I'm not typing very well today, I don't know why. And then we'll go ahead and do the Pearson correlation. And so if I pop this open, we see that the correlation is about 0.83. The p value is significant. And so precipitation and the snow is obviously correlated. Now I want to go about modeling the data, right? And so I'd rather do this at the daily level. And so maybe what we can do is I'll go ahead and call this annual. And we'll go ahead and replace that in here just so that we can have all of the code, all the data available to come back to in the future. And so I'm going to go ahead and bring all this stuff down to do it on a daily level, right? And so to change it to the daily level, I'm going to go ahead and remove the group by and summarize. And now what we can see is we've got our date, the tmax, prcp, snow and the year. I don't really need the year going forward. But that's okay, or the date really, we'll go ahead and leave it in there for now. Maybe what we can do is go ahead and repeat this scatter plot view, right? And so we could do a gg plot as x equals prcp, y equals snow, and then color equals year. Although Geo point there does appear to be a relationship. There's also some places where we have a lot of snow, but not much precipitation. And other places where there's a lot of precipitation, but not much snow. And so again, one of the things I remembered hearing reading as I was doing some research on this is that temperature matters. So let's go ahead and change instead of color by year, let's do color by tmax. We get this kind of funky gradient that goes from like black to light blue going through zero, which isn't super helpful. One thing I'm wondering though is that zero is zero Celsius, right? So above zero, things start to melt. And so I think I'm just going to guess that that's going to be kind of funky. So I'm going to make the executive decision of removing any days where the maximum temperature was above zero. Because it's not uncommon to get days like in March, I know where we'll get some snow, it's like flurries. But it like melts on contact, right? Because the temperature is still, you know, relatively high. So what we'll do is let's go ahead and do snow greater than zero, and tmax less than or equal to zero. So that really cleans things up. I'm not sure how many points are overlapped here. I guess we can look at that here, right? We can go ahead and run everything through the filter. And so you've got about 2,600 different observations where we had a maximum temperature at or below zero, where there was snow. Okay, good. So I'm confident that there's something going on. So let's again call this PRCP snow daily. And I will break up the pipeline to be that. And then we'll go ahead and have PRCP daily to make the scatter plot. Good, so I'm going to go ahead and remove the legend because I'll know that the darker points are more negative. So let's do show dot legend equals false gets us a bit more room to play with. All right, so I think there's evidence that there's a correlation. Again, we can repeat the correlation analysis that we did up here with core test. I'll bring that down. And again, instead of PRCP snow annual, we want daily. And again, I think our correlation before was like 0.83, it's gone up to about 0.87. So there is a strong association between snow and the equivalent precipitation as as obviously, hopefully you would expect. So how can we start to model the data and visualize this interaction or the association between these variables? Well, one thing we can certainly do is come in and we can then do geome smooth to fit a smooth line or curve through the data will do se equals false. So that gives us kind of a saturation based curve. I'd rather have a line a straight line. So one way to get a straight line through the data with geome smooth is to change the formula. We see here that it basically fits a spline or polynomial through the data by default. And so I can come up back to geome smooth and do formula equals and then in quotes, y tilde x, and then I'll do method equals lm for a linear model. And so that then gives me a straight line through the data. One thing that I'm noticing if I look really closely, though, is that the tip of that blue line doesn't go to 00, right? So if we have zero snow, we should have zero precipitation, right? So to fix that, we can add plus zero to our formula. So now we can see the ever so slight change, right? If I toggle back and forth between these, you can see that the line in this case goes right down to 00, because we added that plus zero, whereas this allows the intercept to to vary, right? Cool. Okay, so that is the smoothed based on the linear model with explaining the amount of snow based on the amount of precipitation, right? Something else we could add to this would be like geome AB line. And so that is a line where you can define the intercept and the slope. So I'll say intercept equals zero and slope equals 10. And so we see that line going through the data. Maybe what we could do would be to go ahead and make that maybe a little bit thicker, we could do size equals one. So that rule of thumb of 10 to one actually does a pretty decent job. It's not so far off from the linear model of predicting snow based on precipitation. What I'd like to do though is incorporate temperature. And so the challenge with geome smooth at least is that I can't add or say like do y till the x plus temp or T max, right? What I need to do is build a model outside of this gg plot pipeline and then bring that model in and render the data through geome smooth based on the model that I predict or generate outside of gg plot. So how do we do that? Well, we're going to do some statistics. What do you know, right? So we're going to do LM. So that's the linear model function, the same method that we use down here for GM smooth, right? And so we'll take snow and we'll do tilde. So this is a formula notation where we take snow and explain it by T max times PRCP, right? And so that T max times PRCP tells LM to look at the main effects of T max and PRCP accounting for each other, as well as their interaction. And then we'll do data equals PRCP snow daily. This then gives us some summary output. It's not super clear what's going on, but we see the coefficient. So the intercept of 7.7. So I need to get rid of that. So I can go ahead and change my model to do plus zero. So what we find is the rule of thumb actually isn't that far off, right? So 8.6 units of snow gives you one unit of precipitation, right, which is pretty close to 10 to one. And, and that's going to be at zeroes degrees Celsius. As you decrease the temperature, you're going to increase the amount of snow that you would get for a unit of precipitation, right? So it's kind of the opposite effect, right? Cool. Another thing we could do with this would be to assign this to a variable, I'll call this a snow model. And I could then do summary on snow model. And again, looking at the table, we find that the main effects of T max and PRCP are significant, they have very small p values. But the interaction is not significant. So we don't have to worry about the interaction, interactions make things just horribly complicated to deal with. And again, as we saw before, we have a pretty strong correlation coefficient. Good. So now we want to bring the model into our data. And so how do we do that? Well, what we can do is we can add the fitted data into my data frame. So these lines in the plot that we've already seen are basically predictions based on the amount of precipitation, basically how much snow would there be for that much precipitation, right? And so to do the same thing based on snow model, what I could do is the predict function on snow model applied to PRCP snow daily, and then this will output the predicted values for each row of the data frame, right? And so ideally, this would be a column in the data frame. So we'll go ahead and do that. And we'll do mutate. And we'll do predicted equals that predict function, right? And I think I've got the right parentheses. And now what I can do would be to add to all this a geome smooth, and I'll do AES y equals predicted, right? And so that's the column that I'm going to be mapping into the aesthetic for geome smooth. And again, I'll do se equals false. And let's do color equals red. And so that line looks a little bit different than the simple line that we did, just based on precipitation and snow. Let's see what would happen if we had done our snow model without T max. And so we see that the estimate for PRCP is about nine, whereas when we added temperature, it was like 8.6, right? So I'm going to go ahead and put T max back in there. So this is pretty cool, right? We can fit data using linear model. There's also a general linear model, there's also nonlinear models, right? We could make this model a lot more complicated than we already have. It looks like it might be a little bit nonlinear, there might be some quadratic function that might be able to fit the data a little bit better, especially for precipitation values that are less than 10. So anyway, but again, we can fit the model with linear model, we can look at the coefficients with summary, and then we can then plug all that in to our PRCP snow daily, and then plot that with geome smooth. So let's go ahead and clean this figure up a little bit to make it a bit more presentable. So I'll go ahead and remove the color by T max. Let's also add some labels to our x and y axis. So we'll do labs, x equals total daily precipitation. And again, that's in millimeters, and then y equals total daily snowfall in millimeters, right? Let's also go ahead and use theme classic. So those points are now black, which is the same color as the line. Let's go ahead and change those to be a light gray. So let's go ahead and remove the legend equals false because there's no legend anyway, we can do color equals light gray, and that looks pretty nice. One thing I don't like about the geome AB line is that extends out beyond the boundaries of the plot. That's kind of annoying. So what we could do instead would be to do geome segment. And instead of all the intercept in the slope, so we'll do x equals zero, y equals zero, and then x and equals, let's say 40, and then y and equals 400. So that's better. It would be good to have kind of the maximum value for total daily precipitation. So again, we have that data frame. And so we could get the max of that, right? So that and this and you're probably having a hard time understanding what I'm saying. So what we'll do instead of 40 and 400 would be to do PRCP snow daily dollar sign PRCP. But I want the max of that, right? So we'll do max on that. And then y end is going to be 10 times that, right? So let's go ahead and do that. So we'll do 10 times that max. And so we now see our lines starting at zero and going out to the maximum value of whatever that point is on the x axis. So I would like to have a legend for these three different lines. And something I could do would be to add an aesthetic for the color, right? And so what I'll do is AES color equals and then in quotes, here, I'll put simple, and then for GM segment, I'll go ahead and add AES color equals rule of thumb. So I'll get rid of this color equals red. And then in here in this aesthetic for GM smooth, where I'm using the predicted, I'll then do color equals advanced. And now I have a legend that I've created, based on forcing the names or the groups to the color, right? And so now what I could do would be to go ahead and do scale color manual. And I'll then do name equals null, because I don't want to name for the legend. I'll then do breaks equals. Let's do rule of thumb. And let's do simple, and then advanced, right? And then labels of 10 to one rule of thumb, simple model. And then we'll do advanced model. And then our values will be the three colors, right? And so I'll do for the rule of thumb, let's make that black, the simple, let's make that blue. And then the advanced, let's do red. And we get a legend and our three colors. Of course, we can save this out to figure. So we'll do g save figures, model snow ratio dot PNG. And let's do width equals six, height equals four, very good. We now have our legend going with our plot. We could maybe move the legend around, but I think it's okay for our current purposes. Again, what I want to drive home in this episode of Code Club is that R is a statistical programming language. And we shouldn't forget that. There's all sorts of complicated things you can do with statistics within our really encourage you to explore that further. Hopefully if you're taking a statistics class at a university, by now they're using our because that's probably what the professors are using, right? Anyway, encourage you to play around with this type of approach of modeling the data, really appreciate the viewer who suggested this idea that they went and looked with their own data from I believe Indiana of what the ratio was between snowfall and precipitation. I think they found a value of about 12. So again, maybe it varies by geography in addition to temperature. But yeah, let me know what you find works well for your data. I'd love to know what kind of ratio or what kind of coefficients you're getting for both the precipitation, as well as for the T max values. So play around with this, try doing this type of work with the data you're using for your job. And we'll see you next time for another episode of Code Club.


TfNN, The Tiger, Financial News Network, TfNN, Headline News Update. Good morning, everyone, Basel Chapman here, 10 o'clock, Tiger Financial News Network, Market Update, looking at the Dow made another all-time high today at $37,801. We will go in my show, coming up, the Tiger Technicians Owl, we'll look at this left side high of the 2nd of January at $37,790 and the fractional new high today and we'll look at all the technicals to see if this is sustainable or if we're making a short-term top, we did make a leg D in that 120-minute chart we're waiting for today, we've got it, I'll be back with that a momentarily but let's just look at the S&P, made a new high, not an all-time high, we went to $47.98, went above the previous high of the 28th of December to $47.98.50, let's pull back a little bit now, it's down 5 of $47.78, technicals here and like the Dow, the 9 is over the 14, all the other technicals are starting to weaken, QQQ, and the X100 didn't make a new recovery high, a new all-time high, it's up $0.03 at $408.54, we're struggling a little bit, we'll look at the IWM, the Russell 2000, oh down over two at $190.90, it's been very weak lately, let's go to Gold, Gold is up six at $2033, tried to rally right all the way to $2056, whoops, came back over 20 points from that, struggling, Dollar, did the Dollar rally at all? Yeah, did seven ticks at $102.45, it's really just stuck in this tiny range, let's go to Bonds, Bonds are down almost a half a point at $121.25, 30 seconds, whoo-hoo, those yields are staying up there, let's look at High Grade Copper, High Grade Copper trying to rally now, let's pull back, it's still up a fraction, and let's look at Crude Oil, a lot going on, seizing tankers etc, we're looking at the Crude Oil up $1.66 at 73.03, so we'll be watching that because if the Dollar in the next week starts to trade about 78, that's a whole new ball game altogether, I suspect we've made some kind of a short-term top, we can be watching this very closely, we are in short positions, we'll see if that's going to work, you're back in a moment, oh now the Dollar's down 109, I'll start the tanker, finish this all coming up, check out my link below there, and see you later.


Hey, testing, okay. How's everyone doing? Awesome. Is everyone excited to be here? Yeah. You guys been enjoying the conference so far? All right. Awesome. What's your favorite talk so far? This one. Correct answer. Yes. There was only one right answer to that question. Anyone else from Boston here? Oh, awesome. Even better. That's fantastic. Greatest city in this country. Yeah.  Yeah, it's supposed to get warmer. Yeah. Well, you know, come to Boston, you can have cold, wet, and snow all on the same day. Yeah. Yeah, sometimes. Yeah. Yeah, it's. Yeah. Okay. I guess we're ready to get going. So hello everyone. My name is Raffy Schloeming. I'm the CTO and chief architect of a company called DataWire. I'm Phil Lombardi and I'm a platform engineer at DataWire. I've been working with Raffy since late 2015. Yes, had to. So at DataWire, we build tooling for doing microservices on top of Kubernetes and Envoy. And what we're going to talk about today is how to go about using Kubernetes and Envoy to get a whole lot of developers working together and building awesome stuff. We call this bottom-up approach to microservices, service-oriented development. And I'm going to talk about why we think this is one of the most effective perspectives to benefit from microservices. And Phil's going to talk about the platform we ended up building for ourselves to actually do service-oriented development for the services we were building. And of course, my keyboard is frozen. So first, I want to start with a quick poll, though. Show of hands. How many of you have asked these sorts of questions about microservices? Stuff like, how do I break up my model? How do I take my app? What kind of tooling and platform do I need to actually benefit? OK. All right, that's a good number. So these were the same exact questions we had when we started out, and we kind of learned the hard way that these weren't really the right questions to be asking when you start out. So what is the right question? Well, I'm going to talk about that. But first, I'm going to cover a little bit of our early history so that you can understand the painful lessons that go along with this question. So we were founded in about 2014. My co-founder and I were both sort of tool builders. I had a big distributed systems background. So we wanted to build a company that helps people build stuff. Microservices were the way people were building things. And so it was pretty natural that we wanted to help people build stuff with microservices. Of course, we had a lot to learn. So we started out. We dove right in. We spoke to all the big guys who were doing microservices at the time, still our Netflix, Twitter, Yelp, Google, Facebook, and more. And we pretty deeply studied their architecture and tech stacks that they were using. And pretty quickly, this picture of an emergent architecture formed. It looked something like this, a control plane and a data plane, a control plane doing a bunch of stuff like discovery, logging, tracing, and metrics. The data plane basically consisting of a bunch of smart endpoints with resilience for L7 network stability, things like timeouts, rate limiting, circuit breakers, all that good stuff. And so we looked at this and said, well, that's a lot of technology to build just to benefit from microservices. So we can help people by building some of this for them. So that's what we set out to build. Sort of a control plane in the cloud. We used microservices because it was already decomposed into a bunch of separate services. So it seemed like a pretty natural fit. And things went pretty quickly to start with. We actually built a prototype and released it pretty quickly. But right after we launched, things started slowing down. And so we kind of took a step back, scratched your heads, and tried to figure out why things were going slow for us. And as is the case with this sort of thing, when you're in the middle of things, it's not necessarily obvious, even though it becomes clear later with hindsight. So we went through a couple of releases.   We went through a bunch of different deployment systems. We took a look at our architecture and tried a bunch of different, we thought maybe we decomposed the problem wrong. We tried a bunch of different options there. None of that really made a difference for us. And after going through a couple of releases and looking back pretty carefully, asking a bunch of questions, we finally figured out what was going on. Every feature actually required pretty carefully coordinating the efforts of multiple different people. Some of this was for bad reasons, and we fixed those. Some of this was for very good reasons. We didn't want to break our users. And so in the end, we decided it wasn't really our technology or our architecture that was slowing us down. It was actually our process. Now at the same time, our prospects were actually moving slow as well. We had a lot of them. We were talking to 30 or 40 different companies. They really liked us. They kept on coming back to talk to us because we had a lot of valuable information for them. That's not supposed to happen. So yeah, they really liked us. And they kept on coming back to talk to us so we really could track their progress over a pretty long span. And they were all at different stages of migrating to microservices. And we started to notice a pattern with them. The slow movers kind of fell into two camps. Companies that were doing extensive tech bake-offs, very carefully choosing their technology. And other companies that were very, very carefully re-architecting their monolith sort of in a refactor process to get to microservices. And then there were the fast movers. And we really wanted to understand what was different about the fast movers.  So what really stuck out was the story of their first service. All of their first services were different in some ways. There was a file upload service that was created because of the occasional multi-gigabyte file that would take down a Rails monolith. Another case was wanting to isolate the impact of auto requirements for PPI. And another company just wanted to adopt modern CI CD tools and didn't have time to retrofit their monolith. So they started building new features in separate services with more modern tooling. But the thing that all of these stories had in common was that they all had an urgent need that could not be addressed quickly enough in the context of their existing process. So we put two and two together with our own experiences and said velocity actually comes from process, not architecture. And this makes a lot of sense if you think about it.  You get there by enabling a different way of working that as a byproduct makes it way faster and easier to churn out hundreds of services. And this is when we ditched service- oriented architecture, fine-grained or otherwise, and started thinking about service-oriented development. And to understand this, it helps to recognize two things that dramatically impact the way we work as developers. First of all, in software, stability versus velocity is a pretty fundamental trade-off. The faster you go, the more things break. And this is why things slowed down for us after we launched. When we were prototyping, we could break pretty much whatever we wanted whenever we wanted. Second, we had users. We adopted a whole bunch of practices which slowed us down for good reason. And if you have a lot of users that rely on you, or even just a few that rely on you for something really, the more careful you need to be. And the slower you can move. And what this means is that if you're trying to quickly add features while maintaining stability, there really is no Goldilocks point on this curve for you. And that's because a single process is inefficient. It forces a single stability versus velocity trade-off. And there's another important factor to recognize. The development process involves a bunch of distinct activities that I'm sure you're all familiar with. And something that can be harder to notice when you're small, is that you can really only do one of the activities in this process at a time. Yet when organizations try to scale, they seem to do it by building specialized teams associated with some or all of these activities. And as an organization like this grows, this turns into some combination of dramatic underutilization and departments fighting each other. If velocity is a priority, then operations will get frustrated with developers for breaking things all the time. If stability is a priority, then operations will ultimately end up putting processes in place that will slow down development. And when development slows down, product management gets frustrated, and if your leadership doesn't understand this, they can end up setting up a combination of goals and organizational structures that literally pits an organization against itself. What this means is a single process doesn't scale. This is why instead of asking how to break up a monolith, we like to ask how do I break up my process. This is the question to ask if you want to go faster. And this is the perspective to approach microservices from. Because microservices lets you have as many different processes as you like. You can think of microservices as a distributed development workflow. And this lets you customize the process for each of your teams to that ideal stability versus velocity tradeoff for whatever service they are working on. And when you do this, that's how you can move fast and keep things stable and get benefits almost immediately. You can have multiple simultaneous processes, including your existing monolithic process, all tuned for that ideal stability velocity tradeoff with all your faster moving services leveraging the value you've already built in your more stable services without actually disrupting them. So this sounds great, right? So how do we get started with this? Starting from the right principles makes this thing a whole lot easier. But it is still a big shift in how people operate. This requires both organizational and technical changes. Organizational changes really impact the technology and how you use it. So I'm going to cover some of this in some ways you can make this easier on both fronts. So first of all, the organizational factors you need to consider. You really have to give a lot in order to get. You need a big emphasis on education, communication, and delegation. And if you look at the picture on the right, you can see why. On each of our small teams, our former specialists are exposed to every aspect of the development cycle. And so there's now a big learning curve. And also, because they're all specialists, nobody speaks the same language, so communication can be a challenge. And with this model, you end up delegating bigger parts of your business to much smaller teams. And that's sort of the point of this. But it can also be quite scary. But when you do this right, you get a whole lot. And you get better holistic systems. With learning comes personal growth and much higher job satisfaction. With better communication, that conflict that existed in your organization before turns into collaboration. And with delegation, of course, you get massive organizational scale. And the benefits when this is done well, they're pretty hard to overstate. So what's the best way to actually go about doing this? Well, you need to create self-sufficient autonomous software teams. Why self-sufficiency and autonomy? Well, if you have to work directly with other teams to get stuff done, then their process is interjected into yours. If you have autonomy, you can choose the best process to actually meet your goals. So to get there, there's two things to be aware of. First, you need to be aware of centralized specialist functions and try to eliminate them. Try to avoid centralized architecture teams and centralized operations teams. And don't get confused between a platform team that provides a platform and tooling as a service and a centralized operations team that's responsible for keeping all of your services running. These are two very, very different things. The second thing is to think of your teams like their own little spin-off companies. You probably already consume external services like Stripe and Twilio. Well, you can think of your microservice teams the same way. Pick a real urgent business problem that you wish you could buy instead of build, and then form an internal spin-off to build it. This sets up the right mindset for a lot of things. It helps you define your service because you're thinking mission statement. Who's the user? What are you trying to help them do? This is a way more effective way of defining services than big upfront design. It also helps with communication. You set up a customer relationship versus a co-worker relationship, and it helps you form the right team because you put people with all the different specialties you need on one small team. You can even do this with no fancy tooling or technology at all, but tooling can help a whole lot. That brings us to the technical implementation. Now, if you recall from earlier, this is sort of the rough picture of what most microservices tech stacks look like. It turns out this picture is actually missing something that's really, really important. And that's the people. People are the top-most layer of the control plane. When a service breaks, a person has to find out about it and fix it. And for a service to improve, a person needs to change it hopefully for the better, and the people factors involved have a big impact on the tooling as well. In fact, this gives us the goals for our tooling. We need to build tools that make our small autonomous teams productive. And right away, this tells us a few things. We need our tools to be good for the generalists we're trying to create. And a generalist UX is very, very different from a specialist UX. Think about all the specialist tooling out there for operations, things like monitoring, logging, analytics, deployment, all the L7 proxies and routers you can choose from. All of these tools have enormous surface area and flexibility, which is exactly what you want as a specialist. Specialists prioritize something general enough for any possible application a developer might conceivably write. And they don't care about the learning curve. Generalists, on the other hand, want something that's an easy starting point that works out of the box for their particular application. And that then lets them discover the more advanced features if and when they run into the need for them. Now, the other thing this tells us is that our teams are learning, so we need our tools to actually help with that. This means building on familiar concepts and providing training wheels. Safety faults so we don't get into trouble and great feedback. Every error message should really have a pointer to whatever it is you need to learn to solve the problem. And finally, because our teams are autonomous and each one can choose their own process, our tools need to be flexible enough to fit many different processes. So this is sort of the platonic ideal platform for doing service oriented development. And we've even managed to actually accomplish some of this. And Phil is actually going to talk now about what it is that we've built, what's worked well for us, and what hasn't. Yes. Thanks. So it may be nice to believe that Rayfee has just come up with all these things on his own by sitting in an armchair and drinking scotch and going, man, I can tell you all these people in this audience about this next week but it's actually a bunch of hard-learned lessons over the course of two years and some change from when we started building the backend at DataWire. So a little context, I joined DataWire at the end of 2015 as pretty much the first backend engineer to join the company. And at the time Rayfee was working in Boston. I was working in Boston. We had two other engineers, one down in New York and one in Europe at the time. They were kind of working on some other stuff and I came in to start building the backend. And we started building the backend. We knew we were going to need to build pretty fast for what we were trying to do and also we were going to have to build more than just a handful of services that we were going to need to put all the engineers on eventually. Now, when I came in there was no really expertise in the company for a platform engineer or an operations engineer, but if you're going to maintain an actual production service or you're going to actually try and build something that needs to be running in the cloud, you find quickly you do need someone to do that. So, we're out of luck. Anyways, good news for DataWire was that I had done some of that kind of stuff in my previous job. I was basically working as a platform engineer slash developer tooling and experience engineer for the previous gig. But once again, problem. I wasn't hired to actually work as the platform engineer or developer experience engineer at DataWire. I was hired to build the product in the backend. So, I had to come up with something that would be relatively easy for people we were going to bring into the company to work on and also the existing engineers to just pick up and go so that they didn't have to bother me all the time with their problems. So, I kind of go like, all right, well, the only way to really do this is build self-service automation around all this stuff. And so, I was like, all right, we're going to build out a self-service platform that anyone at the company can use. It'll be documented well. There's a lot of instructions to get a service started. And here's the tools they're going to be using and there's going to be these various URLs they go to. They can put things in here and they'll build out the CI pipeline, they'll build out the deployment pipeline, all that jazz. And so, there were some design considerations along with that that I had to keep in mind. So, DataWire has always been a polyglot programming shop. You know, Rayfee and the other engineers at the time were working mostly in Python. I came on board as a backend engineer and JVM experience, so I was writing Java stuff. We've always kind of had a, you know, openness about picking whatever tool works for you. So, you know, JavaScript tends to just creep into anyone's code base despite everyone's best attempts to avoid it. Go has become really popular and stuff like that. With any programming language set, you end up having a lot of tool chains, so not everyone is going to be using the same tool chain. If you're a Java developer, you're obviously familiar with, you know, Maven or Gradle, but Python folks don't use those tools at all and laugh at you if you suggest any kind of idea of using that stuff. The engineering org started out as a distributed by default and while we're now all co-located together, we still kind of follow a distributed team pattern, so some engineers come in early, some engineers work late, some engineers work morning, take the afternoon off and start working again late in the evening, so we have what would be a co-located team with distributed engineering personality, basically. When I came in, I was really the only person with a strong services development background and microservices development background, so we had to have a way that anyone who was going to start working on this could pick it up easily enough despite not having that background. And because we were building the actual backend for the product, there's obviously the concern that this is all mission critical software, so it has to work and it can't fail and needs to be able to be debugged and logging has to be available, all that jazz. So what did I do? The other way that I went about doing this was I wanted to do it right, because that's what everyone always says when they start out a technical project, I'm going to do it right, and so I started architecting the shit out of it with tools and I threw things like Jenkins and Spinnaker and Terraform and Ansible and EC2 and everything else that you can think of that's popular under the sun for doing this kind of stuff and, you know, I'm not saying these are bad tools, they're not bad tools, in fact, they're all really good tools in the right context and that's what I'm saying, I got the context wrong. I put all these tools together, glued them together, wrote up some documentation around it and thought I was going to get the uh oh, my GIFT didn't load. Oh no. Oh, there we go. So I thought I was going to get that and turns out, no, I got something more like this who is actually so this is not my actual co-worker but he looks sort of like this with long hair and he was looking at all these tools and going oh, what is this? This is insanity. You've got me looking at three different documentation sets jumping across different programming language for different configuration sets and like, what is this? Insanity, insanity, insanity. So all through 2015, 2016 we kind of talked about how Rafe said like prototyping started out initially quickly and then things got slow and they got really slow and change became really slow and painful as people had to start bringing in more of this tool chain into their own tool chain and trying to adapt and use it to actually deploy software and I, you know, put my fingers in my ears and kind of said no, no, no, it's not my fault it's your fault, you're not reading the documentation right or you're not doing it correctly which is completely the wrong response but when I finally sat down and started to look at the metrics for like how slow we were going versus like what we should be actually be able to do it was like ooh, this is really bad we got an issue here. So we kind of had to basically have a reset and fortunately we had some time at the end of 2016 and I set aside, you know, take a look at everything we were able to step back go hmm, maybe this is not correct so we took a step back, looked at it and went what do we want to do but in the meantime it's not what we really learned was none of this is about tools you can throw as many tools as you want at these problems and you're never going to fix people in process because people in process is actually the most important thing and they're not really, they're not trying to fix them when you say fixing you tend to imply they're broken, they're not broken they're just doing things in a way they want to do and they make them the most productive and at the end of the day people are trying to be productive they want to get their job done so they keep everyone else happy, the business is happy and it keeps going but you can't throw tools and the expectation I had started with was let's throw tools at the thing engineers are really finicky they don't like actually picking up tools that they don't believe in they don't believe the tool is going to work for them and makes their life harder they will not like that tool and so I started putting tools in front of their way that were slowing them down even though they knew what they were trying to do and they got angry and they got grumpy and they complained and it was a bad experience so all the tooling in the world it's not going to make your problems go away you got to think about the people in the process finally there's another really important point with tools every tool has basically a cost to it so every time you add a new piece of a new tool into your deployment pipeline or into your dev workflow you're actually asking for n number of permutations to come out of that how people are going to use it the docs say you're going to use it this particular way and you're going to find no one uses that particular way and three developers are going to come up with three or six or 20 different ways across all your projects that they're going to use it differently and you're going to have this explosion of combinations of tools and configurations and usages that don't actually match what the documentation says and more importantly readme's they're just bad so couple problems with readme's I know I do not write readme's particularly well so I have an expert blind spot when it comes to putting this stuff together I will jump from step one to step two and not realize that I have missed you know A, B and C of one that led to not having complete information the next person who reads it then also reads it and goes well I have no idea what's going on but it leads to arguments in the office where people are like well I followed your readme and you didn't actually document what you want me to do here because well I don't have that information that's one problem, another problem readme's are often treated as opinion rather than as the facts of the world so think of it like baking versus cooking in cooking you can take a lot of liberties you can look at a recipe and tweak it and do whatever you want to it developers tend to look at readme's and go step two already so I'll do it my own way because it will end up with the same result I think whereas the person who wrote the readme actually was treating it like a recipe for a cake and we all know when you start messing with recipes for cake like adding baking soda to yield better flavor you're not actually going to get what you expected it's going to taste terrible and you're going to have a lot of problems there so we went back to the drawing board at the end of 2016 we were like okay what do we actually want we really want a way for everyone to kind of have a common tool chain that can be exist on their laptop or on your co-worker's laptop or on your friend's laptop or your grandmother's laptop or in the CI system wherever you put it for whatever language for whatever environment it's going to be the same and we said we're going to do this initially by starting with Docker as our primitive so we will package everything into Docker and we will then figure out how to run Docker stuff so once you got into Docker the big question of time became like how do we run this and I'm sure everyone in this room is familiar obviously with things like miso and swarm but I had been following Kubernetes pretty closely since 2014 when I first heard about it and by the time of 2016 rolled around I was like okay this looks like it's ready for prime time this looks like where everyone's aligning and so I said alright we're going to use Kubernetes this seems like the place it's configuration driven it's pretty simple in terms of what it's actually trying to do has concepts that everyone will find pretty familiar we'll deploy it with Kubernetes and so our Kubernetes journey it began at this point in time and we started to take a look at the existing tools we had we scrapped most of them and said we're going to build tools based on two things either there's active pain in the process that you're trying to accomplish or there is some common pattern we've noticed across all of our repositories and all of our projects that are worth encoding in a tool and so like an example of active pain that everyone feels so a Docker build process where you're going to build say a Python application is really quick but it becomes a painful painful process for Java developers who have to recompile code because every Docker build effectively starts to be treated as if you were doing a clean build from scratch without the ability to have a Java you know build cache for your compiler so that's an active point of pain that we built tooling around another you know example for common pattern is well we noticed everyone has Docker files in the repository and everyone started having the Kubernetes manifest in a particular directory like alright we'll make that standard we'll make it so that whenever you put certain things in these directories you know the tool will just know what to do as it goes off and does the thing so for anyone who actually really wants to kind of go back to their own you know development workflow and start building like this your first step that you actually have to follow is provide fast repeatable builds for everyone so you know figure out what you want to do for your workflow for us it was we were going to have everyone works off of kind of a working at a git code repository and they create branches for the thing they package not only the old resulting artifact for what they're working on but the entire build tool chain is actually built into the Docker image which actually got rid of the problem of having to deal with readme's and having specific instructions for particular tool chains so like when they do an actual Docker build now and you're a Java developer the whole Docker build process will go through and it will end up with you know doing the compilation and producing the image with the jar built into it same step for a Python project or a JavaScript project or a Go project they don't need to know the individual steps going on the person who wrote the project and started it knows that but people who are working on it don't need to know those individual steps they're codified into the Docker build process so it gets rid of that whole problem of well this was a this was actually a baking recipe and you treated it like a cooking recipe and when I said I had to work for everyone I meant everyone so we made sure we weren't forgetting about people who were working with you know different tool chains it's imperative that you have a tool that works across your entire organization for any type of situation because you know everyone eventually becomes a polyglot shop whether they actually believe it or not you're either through acquisition or through some other mechanism are going to end up with services written languages that you didn't expect so you need this process to be resilient to change for in your organization as people come and go so that one was to go anyways so step two you have to provide fast self-service deploy as well so a data wire when we're actually building code we have the ability to either deploy into our shared cluster as pre-commit or post-commit and so that means when you're working on a branch you can have an isolated environment for your code that another person working on another branch is completely isolated from them so like Rayfee will often be working on a particular feature of one of the project I'll be working on another particular feature of that project and I can be running the same code I'm sorry I can be running the same project in the same cluster and none of us will ever touch each other but it will still be in the same cluster so we only have to manage a single cluster in total so parallel isolated deployments throughout your organization so to recap for step one and two codify and put your entire build process into a Docker image make it so that it actually does the entire build to get what you want to do and that's going to be your runnable artifact and then you use Kubernetes to actually do the deployments we found as you know we did these things we started to codify them into tools so like we've wrapped Docker build now in our own tool called forge which does the Docker build so that it can maintain compiler caches and stuff like that it can also do things like shortcut building an image if it knows that the image doesn't bother doing that it just looks at the commit history and goes this shop the entire repository is the same we're not going to bother building a new image for you there's nothing magical at any point in time developers when they need to deviate from the logic we've prescribed it's just a code change to the tool itself and if there's worst case they can just pull the tool out we're not doing anything that fundamentally alters their development process it's just a glue code that everyone ends up building no matter what but there is really a third and the most important green here it's step three you have to make it easy to reach the change to code and we do this with Envoy so we use Envoy and it's our basically entry point to our cluster and we put a thing called ambassador which is this thing that listen to changes on annotations that we put on the service manifests that are applied into Kubernetes so basically a typical setup in a branch at data wire for a particular service they will annotate the service so that they get host based routing that has the branch name in it goes through ambassador which is an API gateway which handles things like off which then goes to Envoy Envoy routes it to the correct backend service so it makes it super easy for someone to actually have like dev awesomefeature.datawire.io or plimbardi.datawire.io where they put whatever they want spin up projects in general because you only have to think about this little block of annotation code that you put into your service manifest and off you are to the races is it perfect? No, it's really not perfect we still got a long way to go but it's so much better than we were we really started putting this into effect earlier this year and like the speed at which we're actually able to build our services now is monumentally faster it's hard to actually quantify how fast it is now compared to what it is but we still have a long way to go so there's a bunch of things we are working on actively going into 2018 that we're going to improve these things really for us self service bootstrapping I shouldn't have put self service there, anyone can start but we'd like to have some easy way for developers who come onto the company to easily start services that doesn't involve usually taking an existing service and ripping out the code from it so we haven't really come out with a good cookie cutter mechanism that we like for me I'm still continually hounded for self service stateful infrastructure so I'm usually the only person who knows how to get that working and so I'd really like to figure out how people are going to deploy their own databases and their own queue servers and all that kind of stuff without having to bother me and really monitoring and logging we have all the infrastructure in place for collecting logs we have all the infrastructure in place for doing monitoring it's mostly we haven't come up with a good pattern and good UX yet for everyone to be able to consume that data in a meaningful way anyways, Rafe here is going to take it away for the rest here and talk to you about like a summary and can I conclude this thank you yes so looking back after kind of reliving some of the past a couple years thanks to Phil if I could give myself some advice from the future I would really tell myself that Kubernetes and Envoy really do have the bulk of the capabilities you need to get started with microservices but capabilities really aren't enough UX matters a lot and I mean a lot and when it comes to UX the people and process factors they fundamentally drive the UX and building this ideal UX it's actually an enormous undertaking and so it's really helpful to understand some of these principles and take an incremental approach with them in mind so that's our talk thanks for coming we're happy to answer any questions now if you want to talk to us more later you can visit us at our booth S58 if you're interested in using some of the tools we've built for your own platform they are free and open source we've published some hands-on guides at the link above send us a tweet email us and we're happy to chat