All righty, how long do we have? How long do we have? Two hours? Two hours, okay. Right, we're gonna go faster than usual. Howdy everybody, thank you very much for joining us today. I'm excited to be here. Before we get started, before we get underway, we have something of the utmost importance that I need to address. When I say spring, you say open source. One, two, spring, open source. Ah, that's so good, that's so money, thank you. Very good, right on. Now, we don't have a lot of time today. In fact, we have almost no time, so I'm gonna go very, very, very fast today. The goal here, my friends, isn't so much that you remember everything that we're about to do. It's more that you appreciate what's possible and you'll be able to find no doubt resources on the inner tubes for your own reference later on. Now, how many of you have used spring? Oh, well, that solves a lot. What about spring boot? Okay, okay, good, good. Yeah, do any of you work here at PayPal? Okay. Apparently there's a framework in here called Raptor. Have you heard of it? And some of it's now being on spring boot or something like that? Yeah, that was a, we had people from PayPal that spoke at spring one last week, or two weeks ago, about their use of spring boots. I thought that was kind of interesting. Anyway, we're gonna do an ever-so-quick review of spring boot just to make sure we're on the same page. And then we're gonna talk about spring cloud, okay? So, hold on tight. Now, while that slides up there, I'd encourage you to grab the bits, grab my coordinates up there for your own reference and identification for later on. So, that's my Twitter. How many of you are on Twitter? It's 2016, Twitter. Okay, get on Twitter. It's the new IRC, I love Twitter, and you should too. What about email? How many of you have email? Email. Anybody. Okay, we'll grab those as well. I'm happy to answer questions, talk to you, feedback, whatever. Should we turn off the lights? Can you all see that screen? If I were to start going really fast with a lot of code, would your eyes be able to follow that code for the next two hours without blinking? No blinking, right? It's okay. Everybody got that? You got it? You're trying to get it? Okay. A little bit about me, as it was just well explained. My name is Josh Long. I'm the spring developer advocate on the spring team at Pivotal. I'm an open source contributor and engineer. I'm the number one top ranked, seven years running contributor of bugs, two projects like Spring Boot and Spring Cloud, and Spring Innovation and Spring Batch and Bodin and Timelief and Activity, okay? Number one, very proud of that. I've got to have something. I'm also a Java champion. This is a rare honor that's bestowed upon folks in the community who do their level headed best, like I've thus far tried to do, to engage and help the community better themselves with Java. And a big part of that, of course, is my time before audiences like yourselves talking about software, talking about how to build next level software as best as we can. And a part of that, of course, is writing books and blogs and magazine articles and doing videos. So the latest and greatest training video, of course, is building microservice with Spring Boot Live Lessons with my friend, the one, the only, the amazing, the inevitable, Phil Webb, who's the co-founder of Spring Boot. And then, of course, there's my latest and greatest as-yet-unpublished book. It's called Cloud Made of Java. For those of you who are wondering, and I know you are, I can see it in your eyes, that bird is the blue-eared kingfisher from the Indonesian Java island. So it's a bird, you see, that lives in the clouds. Birds fly in clouds. They're birds, you see, from Java. It's a native bird from the Java islands. Never mind, it's fine. It'll come, it'll come. Okay, it's fine, don't worry about it. And I'm also fond of this t-shirt. There's that. And I work at Pivotal. So how many of you know Pivotal? It's not nearly enough. So Pivotal, we're a small company with big dreams. We have big hopes and aspirations. We're the home. The originator is a lot of great open-source stuff. So how many of you know Tomcat? Apache Tomcat. Tomcat. Anybody. So Tomcat, of course, is not just the animal from Indonesia. It's also an application server. And you're welcome. You're welcome. What about Cloud Foundry? How many of you know Cloud Foundry? Cloud Foundry. So Cloud Foundry is an open-source platform. It's optimized for the continuous and safe delivery of software into production. We care about the evolution of software and the deployment and management of that software. And we care about open-source, but let's be very, very clear here. Let's just really, really be clear. Open-source is not our raison d'être at Pivotal. It's not the reason we're here. It's not the reason we wake up inspired to go to work every morning. You see, we care instead about helping customers, community members, and organizations move quickly and safely from concept to production. And we've seen that a lot of organizations struggle with this movement. They struggle with the progression of software from concept through the value chain off, hopefully, onto production. And they struggle with how to go quickly through that circuit from product management, user experience, developers, testers, administrators, and then off into production. A lot of these organizations are lucky enough to have been around for more than ten years. They've got software that predates the era of cloud computing and the economics that sort of inspired it. They have software that is made of lots and lots of code, lots of code and a large, large code base that requires a large team to evolve. This monolithic code base is, you know, a bit of a problem today. Sure, it was well designed for the time in which it was conceived, but today it presents a bit of an obstacle because time is of the essence. Code bases require a large amount of people to be able to change, to evolve. If you have a lot of people involved, it takes a long time to do darn near anything. So the trick is how can we take this large code base and this large team and then decompose it into smaller bits? How can we make it smaller? How can we break it apart in such a way that each bit can be more nimbly and more agilely evolved? We can turn to this idea of microservices, right? Microservices are a fairly obvious idea. The idea here is to break apart our large applications into smaller bits so that a small group of people can work on smaller batches of work so that they can focus on a small batch of work and push it more quickly through the value chain. This is a hack on Mel Conway's law. How many of you have heard of Conway's law? It's not entirely new. I'm sure we've talked about it before. The idea, Mel Conway talked about software as a reflection of the organizational structure that it serves. So if you have two different teams that don't do a good job of communicating with each other and they're responsible for different modules in the system, they're going to do a crap job of integrating that software together. I've heard it alternatively stated that if you have four different teams working on a compiler, intellectually, you'll have a four-pass compiler. They've done tests that confirm this. They've done studies. They did comparisons of open source and proprietary software. The dynamics of the teams involved creating the software they change the way that the software is created itself. So modular, sorry, open source software rather, tends to be more modular, better defined. And this is because the people involved in that open source software tend to be contributing at different paces, different times, different opportunities, different days of the week, different time zones. So the communication boundary between the different teams has to be the API. It's not easy for them to all just jump on a bridge and talk about stuff on a phone call. Some people will be working on the weekend, some at night, some every once in a month, versus proprietary software where everybody's sat in the same room and they can just turn their chair and say, hey, what do you think of this? Tends to be less formally defined. The APIs, the boundaries between components and proprietary software tend to be less well-defined. So the question is, of course, is how do we define these modules? How do we define small? How do we identify a part of the domain? We can turn to Dr. Eric Evans. Dr. Eric Evans has a great book called Domain Driven Design. Anybody know that book? Preach it right on. So Domain Driven Design espouses the idea of a clear domain model. In the book, he talks about this idea of a bounded context, a part of the domain that, when extracted from the larger hole, stands unto itself internally consistent and reusable. A bounded context is a crispening of the domain of an application. And that's pretty vague. And I definitely recommend you look at the book for more. But suppose you have this context of sales versus customer service. Nominally, you have an idea of a customer in both cases, both domains, in both contexts. But the customer that is trying to get a refund is very different from the one that you're trying to incentivize to purchase something. They have different life cycles. You have different states, different data. These are not the same customer. And to treat them both as the same thing, the same entity, muddies the definition and makes them less valuable for each context. Tease them apart. Treat them separate. Keep them separate. And you'll have an easier time of it. And you now have an ideal candidate for this small batch of work that we're looking for. Small batch of work on which a small group of people can iterate and deploy independently. Another question, of course, is how small is a small group of people? In the West, that answer I think is best delivered by a guy like Amazon's Jeff Bezos. Jeff Bezos talks about two pizza box teams. A two pizza box team naturally has a very different meaning than it does here, right? In the West, where I come from, two boxes of pizza feeds five, six, seven people. Here it's like a school bus, right? So I get the confusion. I get why you're not understanding what I'm saying here. But it's very different, right? Small is the point. The cost of communication between the team members has to be very minimal. That's the ability to go fast. That's what gives you that agility. If you can unite these people, focus all these people on delivering a feature or a bounded context, you co-locate all the people in the value chain involved in moving that feature through production, through development, product management, user experience, developer testers, administrators, et cetera. Get them all co-located in one small feature team or two pizza box team. Then you have what Nifflix calls a feature team. So now you've got a small team working on a small batch of functionality that can be moved and deployed independent of the rest of the software. This is very, very useful. Once you've done this, now you can have different teams that can control their own destiny. You can go faster and you don't have to have constant meetings about what you're going to do. You can just do, right? This is very, very, very powerful. But there are two things that you're going to run headlong into when you move to this approach. Two things that you need to be able to address up front and quickly. The first thing is how quickly can you stand up a new production worthy service and all that that implies. And I mean, how quickly can you address things like observability and security? How quickly can you stand up the infrastructure and the middleware required to support that service? How quickly can you do things like health checks and load balancing? For one service, you have to do all of that, right? Most organizations that I've been to have a nightmarish, terrifying wiki page, the wiki page with 500 easy steps to production. That wiki page is the enemy of velocity. It's what frustrates our ability to stand up services quickly. And if that list is overwhelming, if that list of things that we need to address quickly is too discouraging, then we won't do it. We are all products of the systems in which we exist. I don't believe that there are good people or bad people. There are good systems and bad systems. And if the system in which you find yourself is the wrong thing, the easy thing, then you will, very understandably and reasonably, do the wrong thing. It's human nature. Think about databases, right? Databases are a common example. You see, and so is the provisioning of application servers, right? Infrastructure. If it takes forever to get a new application server, a new environment set up for a new service, then it's so easy just to say, oh, okay, I'll just add another rest in point to this giant big ball of mud API that we've already got. Nobody's gonna notice. It's much better than doing the right thing and factoring it out into a separate domain. And even if I did, I'd still have to get another database, and that'll take forever. And God help you if you're trying to get another kind of database, right? Doing the right thing means being, you know, means exercising discretion about the technology used for your data. If you're trying to do a search engine, then by all means, use a full-text search engine like Elasticsearch or Solar. This is far better than trying to shoehorn it into Oracle's, you know, full-text indexes. If you're trying to do, if you're trying to store binary data, keep bits. The right thing to do is to use a bit bucket or maybe Amazon S3 or MongoDB's GridFS, not shoehorning that into a blob and Oracle. If you're trying to handle geospatial queries, geographic queries, the right thing to do is to use something like Couchbase's GIS queries, you know, geoindexes, right? Not to plunk down another 90,000 for Oracle Spatial, right? If you're trying to randomly lose data at random times for no apparent reason, the right thing to do is to use MongoDB, but for Pete's sake, for Pete's sake, use the right tool for the job. And if it takes too much effort, it takes too many tickets and too many discussions to get that to be done, then you won't. So, you're trying to do something like shoehorn it into the existing bits, which is a shame because doing the right things, it feels better eventually. It's better for morale. So, how quickly can you get past all that? How quickly can you stand up a new production-worthy service? That's the first question you need to answer. And we'll talk ever so briefly about some of that today with Spring Boot and Cloud Foundry. Then the second question that you're going to run into is once you've done that, once you've stood up all these small interconnected but independently deployed small batches of work, communicating over the network. Once you've done that, you've now got... you've got a distributed system. And if there's anything, if there's anything at all that upon which we can all agree, no matter where we come from, east, west, et cetera, it's that building a distributed system is hard. And so for that, today we're going to look at something called Spring Cloud, which makes it easy to build distributed systems. Now, as most of you have already seen Spring Boot or worked at Spring Boot, I don't spend too long in that particular technology because I really want to talk about Spring Cloud because if I can do one thing for you today... I think it's going to be Spring Cloud. Now, any questions on that before we get going? Right now, if you have questions at all, I really want you to feel free. I want you to feel free because I know that it's late and people feel... they don't feel the enthusiasm. So I want you to feel free. If you have any questions at all, go to the end and then ask, okay? Because we're probably going to get you an answer anyway. Those are my slides. I worked hard on those. I hope you appreciate that. Moving on, let's see. I didn't know that worked. Command shift backspace on the trash on the OS 10. That empties your trash. I was trying to clear my history. I didn't know anything about the trash. Cool. All right, so we're going to go to my second favorite place on the internet. My second favorite place on the internet is start.spring.io. My first favorite place, of course, is production. I love production, and you should, too.  It's great. The weather's nice. The people are friendly. It's the happiest place on Earth. It's better than Disneyland. I love production, and you should, too. But if you haven't gotten there, then start your journey here at start.spring.io. If... If you want for inspiration in the early morning before a cup of tea or coffee, start.spring.io. If your children are restless and can't sleep, start.spring.io. And if you suffer from indigestion, perhaps after a long, long night with very, very spicy laksa, start.spring.io. Bookmark it. Keep it close to your heart. Keep it with you at all times. And what we're going to do, my friends, is we're going to build a very simple service. I'm going to call the reservation service, and I'm not going to spend too much time explaining all the options here, but I'm going to take advantage of different technologies that spring supports. I'll bring in the web support. I'll bring in the config client. I'll bring in Eureka for service registration and discovery, Rabbit and Q for stream processing, Zipkin for distributed tracing, Restor polyester support, and JPA, which is the Java persistence API because I make poor life decisions, so JPA. I'm going to use the H2 in-memory embedded SQL database. H2 is an in-memory embedded SQL database that loses all of its state on every restart, and so in this way, it's very similar to MongoDB. Now, what I'm going to do is I'm going to switch to the full version to show you some of the extra options here. When I switch to the full version, you can see there's a variable ocean of checkboxes, different options that you can use to better your workload, better your application. Let's see what you can do here, my friends. I've got this little goofy table. There we are. Now we're cooking. Who does that? There we are. That looks, that's a wee bit better, isn't it? So, I've got, now, so now, I've got a variable ocean of different checkboxes, options that I can use in my application to support my use case, and I could check those, but I'm fine now. A lot of people get very confused here. You see, here we have three very interesting dropdowns. I encourage you to take a look and peruse these options here. You should absolutely exercise some discretion as to which language on the JVM you'd like to use. Any language on the JVM that supports annotations and objects will work just fine. So Java, Scala, Groovy, Kotlin, they're all great choices. Have at them, please. But here, my friends, here we have two non-choices. These are choices that you could make, but that you shouldn't. They're choices, they're choices in the same way that stripping naked and running in traffic is a choice. You could, but don't. Don't. So, for example, which version of the JVM would you like to use? As both 1.6 and 1.7 are more than a year past end of life, gone, expired, dead, no longer supported, not available, past their prime, not fresh. As both of them are end of life, to continue using either one would be irresponsible and an active source of technical debt for the organization. To start a new project on either one is insane, certifiably. So never ever choose either one of these choices and WebSphere is no excuse. And then here, we have the choice of packaging and again, a lot of people get very confused about this. They don't know when and where to choose which, so I'm going to do my level headed best here now to explain if, by some crazy, crazy, freak fluke of physics, some terrible accident of physics, you find yourself stuck in the very, very, very, very distant past. Far, far, far beyond modern help. Then choose dot war. But if you're here with me in 2016, Mayan City, then choose dot jar. This is a big part of my overarching, guiding, personal philosophy of make jar not war. And again, you have options, you have choices. You should do what works for you, which in this case coincides with what I've told you to do. So that works out well for both of us. Now I'm going to go ahead and hit generate and that'll give me a new zip file. And I'm not going to spend too long building this simple application. I just want to have a domain. I want to have an application with which we can work as we build our example up. So I'm going to close this. And again, it doesn't matter what IDE you use. How many of you use IntelliJ? Show of hands. Good stuff. Hot sauce. Well done. What about eclipse, my friends? How many of you use eclipse? Right on. Good stuff as well. And what about net beans? What about net beans? How many of you are using net beans? Cool. Right on. No. Hey, they all work great. You don't even need the application service support in any of those tools. You just need basic Java H support and Maven. That's all. What about Emacs? Are you here, sir? Are you here? Are you here, buddy? Where is he? Oh, there you are. It's this guy. Every country I go to, and I visit probably in excess of 30 countries, 40 countries a year, millions of cities, every city I go to, it's him. Same human. I don't know how you can get the budget to follow me to every place, but it's so, so not okay. We have to talk to my lawyer about that. Now, sometimes I ask who uses Emacs, and it's just, I do, and then he leaves.  So we're going to build a very simple domain model, and I don't care all that much about the domain. I just want to have something with which we can play. Object type reservation. It's going to be a JPA entity, and I'll signal as much by using at ID and at generic value, and then I'll use at entity here, like so, and I'll create some getters. There we are. I'll create a constructor. There we go. Another constructor as well, so good. This is for JPA alone. Okay. We'll create a two-string method. There's that. Now, what I want to do is I want to store some records in the database. So I'm going to use spring data declarative repositories. The idea here is that I want to make short work of the tedious, soul-annihilatingly boring work of reading, writing, and updating records of type reservation, because remember, this is not business differentiating functionality, right? It's not going to further my lot in life, make me a better person. It's not going to advance my business. It's undifferentiated heavy lifting, as Adrian Cockroft would say. And I want this API to be a simple REST API. So I'll say REST repository resource, repository REST resource, rather, and spring data REST will turn it into a REST API for me. And then, finally, I'm going to create some sample data like this. And the sample data, you know, it's a callback interface. This is a command line winner in Spring Boot, and it gets invoked after the application has started up, but before most of the components have been put into service. So it's an enviable place to do any kind of batch or ETL or integration or messaging that you want to do for application initialization. Now, my name is Josh. It's nice to meet you. I'm joined today by my buddy and pal, Sergio and Michael or Michel Michel. And who else? What's your name, buddy? Claren. Sorry? Claren. How do you spell it, friend? H-U-I-R-E-N. H-U-I-R-E-N. Nice to meet you. Very cool. What about you, boss? Thank you. A-D-R-I-N. It's cool. Nice to meet you as well. What about you, buddy? C-J. Just C-N-J? Yeah. Thank you. Nice to meet you. What about you, buddy? Yeah? I mean, yeah. A-M-E-Y-A. Wait. A? A-M-E-Y-A. I'm sorry. I farted. My brain farted. What? A-Y-A. Yeah. Very cool. Nice to meet you. What about you, buddy? C-J. Yep. How do you spell it? S-H-A. S-H-A. S-H-A. S-H-A. Again. Oh, cool. N-K. N-K? Like that? Very good. Thank you. That's eight. I think we can do four more. Is there a female at all? Oh, hi. Good. Thank you for coming. What's your name? Sonia. S-O-N-Y-A? Yeah. Lovely to meet you as well. Thanks for coming. Any other females? F-L? No. Okay. Very good. Thank you for coming. Nice to meet you. We need two more. What about you in the back? Put it up with a red shirt. Yep. Topo. Sorry? Topo. Cool. P-E-N-T? Z-G. Z-G. Z-G. Like that? Yes. Very good. Nice to meet you. Nice to meet you, too. Thanks. And I don't know. What about you in the very back with a blue shirt? Hi. Yep. Yep. Excellent. A-X-E-L. Oh. A-X-E-L. German? By accident, yeah. But the name is actually Swedish. Very, very interesting. Well, all right.  Is it also Axel Fontaine or something like that? Interesting. Usually it gets Axel Fuli or Axel Rose or something. Yeah. Oh, those. I don't know those at all. Yeah. Who's that? Is he as cool as the guy from LiquidBase? No. Ah, whatever. You kids and your rap music. I don't care. Anyway. Shhh. Anyway. For each name, I'm going to write a record to the database. I'll say a new reservation, pass in the name, and then I'll say reservation repository.findall. And I'm going to print out every record that comes back. Now, here I'm using a very nice feature in Java 8. You've used Java 8 so far. Oh, that's terrible. The rest of you, get on it. Come on, be doing. Wow. So Java 8 is awesome. It's great. It even rhymes with great. That's how great it is. It has a lot of nice features that help keep Java at the very vanguard of modern technologies, modern languages. Here, for example, I'm using a feature that is unique to very modern, very new, very, you know, progressive languages like C. And Kobal. And Lisp, Smalltalk, Perl, Ruby, Python, Lua, Erlang, Haskell, Go, Forth, PHP, Basic, Visual Basic, Visual Basic.net, ActionScript, ECMAScript, TypeScript, C Sharp, C++, Objective C, Visual Basic.net, F Sharp, JScript.net. Languages like PHP, no. Languages like, oh. Languages like, no, I can't. PHP. Even PHP has this feature. So it's very nice that in Java 8 there's a new feature that only these other modern languages from the last 60 or so years have. A feature that we introduced in Java two years ago. So what this lets us do is write a lambda expression. I think that's very cool. Okay, if we start that up, we should see results very, very quickly. It's going to write the records in the database, and then we can confirm that they're all there by calling the find all method. And we'll see them printed on the console here, okay? Now, localhost, 8080 ford slash reservations. Oh, something's a foul. Did I spell my entity incorrectly? Oh, resource.  Okay. So we should see very quickly here that everything's come to life. We've gotten the results we expected. We should see on the console. We should see reflected here on the console the names of the records that we just inserted into the database. And we should be able to confirm that each of one of them has been given an auto incrementing primary key here, right? So there we are. There we are. We happy few. Josh and Sergio and Michael and we ran, and Adrian and CJ and so on, right? All of us are here on the console, and it worked. Of course it worked. It was a demo. What were you expecting? It was always going to work. There was no doubt about that. Instead, what I really, really wanted to talk to you about was this. This is the ASCII artwork. Now, I don't know how much you know about ASCII artwork, but I'm a big fan. Now, this ASCII artwork took a long time to get right. You see, we on the spring team have people who are doctors, PhDs, people who in their previous lives worked in nuclear physics, star stuff, the very celestial bodies, and the heavens above us were their daily bread and butter. And so it makes me very happy to imagine that some day, somewhere, somehow, there was a GitHub issue that said, damn it, we need good ASCII artwork, and look at that. I think they did a great job. Absolutely amazing. Now, it's at this point that I'd like to take a brief, ever so brief digression to talk about what I consider to be very, very serious deficiency in the JetBrains Intelligent Product 4. Now, I'm a fan. I think this is particularly poorly conceived.  Why is that there?  And so I did what all people do when confronted with adversity, challenge, and despair. I went on the internet and I cried. And I was greeted by a message of hope from my buddy, Jan Sabrone, who's a software developer by passion in Intelligent JetBrains. And he responded with this message of hope, which I share with you now, today. Don't worry, my friends. They can tell JetBrains great again. Now, sometimes people ask me to my sadness, despair, and borderline rage, how do I change the ASCII artwork? As if wanting to get rid of it was bad enough. It wasn't bad enough. How do I change the ASCII artwork? And that's a dumb question. Even now, I resist the urge to start flipping tables and storming out the room. But I can see that there are some people whose talents are weird enough that they may want to do this. So I'll show you how. You can just put in place your own ASCII artwork. Downloads, reservation service, source, main, resources, banner.txt. And when I restart now, having done all that hard work, we should be able to confirm that it's much better. Now... There's a few things I want to share with you about this. This is a... This is not mine. This is a colleague's of mine. Stefan Nicole Brian Klosell. There's a few things I wanted to share about this. First of all, meow. And then second of all, and this is my favorite part, and it's basically why we're here, and if you get one thing out of this talk, let it be this. Anyway, now we've got an application, it's got hypermedia, it's got a REST API. We can confirm as much by visiting it here. 8080, forward slash reservations. And voila. Just a simple REST API, and we can visit each record. Reservations one, two, three, etc. We can do a search, and I'll go like this. Reservations search. Hello, computer. Where's my search endpoint? Oh, I didn't have any finder methods, so there's no search endpoint. But the point is I have an API. This is Spring Boot, and Spring Boot provides a framework called the actuator. And the actuator is designed to expose information about the individual state of the individual host in the node on which the application is running. So this is meant to surface information for centralized monitoring infrastructure. We need to care about observability in a cloud-native system. The actuator is a part of that. There are four tenets, by the way. We should keep this in our heads as we move forward in time for the balance of our talk. There are four tenets to a cloud-native system. The system should lend itself to agile and easy evolution and be easy to evolve. It should be dynamic and do the right thing in the face of elasticity in a cloud environment. It should do the right thing and be observable, let us to say I should be able to monitor it from its outputs. A part of that is served by the actuator. The actuator provides, for example, endpoints like metrics, which gives me an enumeration of environment variables and qualities about the application itself. I can say that I can go to the health endpoint, for example, and I can see here that there are different components of an application, each of which has its own status. So this is a very simple Spring Boot application. And if you know about Spring Boot, then you know that I can change the different parts of the application by going to source main resources application.properties and plugging in any number of different properties to override the default behavior. These properties inform the behavior, but they have useful defaults often, so you don't have to worry about it. I can change them in the code, in the property file, in the jar, or I can override them in the shell here. So CD, downloads, reservation, service, may even minus D, skip test, and I can install. Okay, I'll take some water. Sweet nectar. Now, if I go to the target directory here, you'll see I have a so-called fat jar. This jar has everything I need to run this code, everything I need to run this code in a self-contained manner. I don't need to deploy it into any other thing except for an operating system and with a JDK. This jar is really easy to operationalize. I can attach it as an attachment and send it to my dear, and my grandmother, who is very intelligent and very, very interesting and very smart, she's not particularly conversant with computers, but she can run this because she has applets. So if your WebSphere adult operations teams have trouble running this, have them call my dear, she'll help them get to production faster. Now, that said, we still have questions of configuration. How do I promote this from one environment to another? I showed you that we can change the properties here, the properties like server.pollard equals 8 and 10, but how do I change those properties as I move the application from one environment to another? And here, 12 factor style configuration is very useful. The idea behind 12 factor style configuration originates from the 12 factor manifesto, which is just a set of 12 common sense guidelines, good clean cloud hygiene, if you will, building applications that live and breathe in the cloud. 12 factor style configuration stipulates that environment-specific configuration should live, guess where, in the environment, not in the code. I shouldn't have to recompile. And Spring Boot supports this very well. I can say server.port equals 8 and 10 minus jar, reservation.service.jar and we'll see the application now spin up on port 8010 instead of the default, which was 8080 now. Here's this. I can also use environment variables. I can say export server.port equals 8040 Java minus jar, reservation.service.jar and we'll see that here as well. So I'm using environment variables to inform the default behavior without having to recompile the code. And this is pretty good. This isn't bad for horseshoes, right? But we need to do a little bit better. For you see, this is going to fall short of four key scenarios. Suppose I wanted to centralize my configuration at the moment and I have to T copy and paste my configuration across start-up scripts and all those sorts of things. What if I wanted to keep sensitive information, locators, credentials, that kind of stuff? How do I store that? Certainly not at rest unencrypted on the file system, right? What if I wanted to change the configuration while the service is running? At the moment I'd have to restart the service each time to observe those changes. And how do I support auditing and journaling? How do I see who changed what and if necessary to roll that configuration back? For all these use cases and more, while what I've got here thus far is a good start. It's not nearly enough. One approach to solve some of these problems might be to use a central directory, a directory full of configuration. That would certainly address the centralization problem. I could also make that directory based on Git that would solve the auditing and journaling requirement. But what about the security requirement? To do effective security, we need something in the middle, something that can act as a little bit of an indirection between the client, the client of the configuration and the source of the configuration. And so that's what we're going to do. We're going to stand up a config server. And I'm going to call it, guess what, the config service. Here we go. And the config server is just that. It's a server that's going to babysit or manage a directory full of configuration property files for us. I need to use the right annotation at enable config server or abracadabra config server. And then we need to give this some information so that it can find the right configuration directory and then broker that out or mediate that out for other clients that are going to use it.  I'm going to run this application on port 8888. My configuration is here. github.com, josh, long, bootable, microservices, config or not. Oh, github. Git, clone, desktop, config. There we are. And that should be everything I need to get this up and running. Now suppose I'm a client microservice identified as the reservation hyphen service. What configuration would I see where I connect to the config service? I can answer that question by visiting the config service here at reservation hyphen service, forward slash default. Or not. Did I fail? Oh, son of a gun. Guys, there we are. So now here I can see that there are two property sources of which the config service is aware. The first is the properties from the file called reservation hyphen service. The second from the property file called application app properties. All microservices, no matter what their name will see the configuration from the property file called reservation service dot properties, but only the microservice sorry, it will see the properties from the property file called application app properties, but only the microservice called reservation service will see the configuration from the property file called reservation hyphen service dot properties. These Two sets of properties are flattened, they're merged together at one time, to form one set of properties that the configuration client will see. So if we connect our micro service, our reservation service to this config server, we'll see it's been up on board 8,000, we'll see that it has access to this message called, interesting enough, Hello World. So let's do that. We need, of course, to have the Spring Cloud start a config client library on the class path to be able to use this. And we need to then tell the config client where the configuration is and how to use it. So spring.application.name equals reservation-client, spring.cloud.config.uri equals localhost8888. And this information is used earlier on in the initialization of the application. This is used before the application started applying the properties. So necessarily it needs to be used in the bootstrap phase, which is why by convention this property file is called bootstrap.properties. Now if everything goes to plan, this application should spin up on board 8,000, but let's take advantage of that message that we just created here. So we'll say private final string value, create a constructor and we'll inject the value there. And then we're going to create an endpoint, public string read, turn this dot value, and then I'm going to map this endpoint to an htb-git that responds to forward slash message. And I may later, at my own discretion, want to change this value while the service is running. So I'm going to make this beam refresh scoped. And then I'm going to restart. And if everything goes as we expect, we should see that, as I say, we should see it spin up on 48,000. And we should see it have access to the new message endpoint, which is injecting the message value, or rather it's not, and so it's going to fail. I forgot about this. OK. I knew I forgot something. So what that's doing is it's injecting the key called message from the config service. Yeah. Local host 8,000, forward slash, reservations, other computer, what the, does anybody here see something I'm not seeing? Sorry? Unless we're doing it on a field. What did I do wrong? pkjava. Start that up. Application bootstrap reservation service. That was unfortunate. That could have been avoided if I had just been a little bit more awake. Bummer. OK. Well, we'll sit here anxiously waiting for the results, waiting on you. What? It's got an extra space. Son of a motherless goat. Look at that.  The config server, what? I restarted it. I felt bad for it. So we're friends again. OK. That's really hard, guys. Let's not do that again. So port 8,000 message. Voila. The message is working, but we can do better. You see this message, while true, isn't really great. And we are nothing, if not great today, right? Let's go to the config directory here. I'm going to open up the reservation service type properties. And I'm going to open this up in my IDE. And I'll effect a change. Hello. The change will be hello singa-sug. There we go. Extra exclamation marks, so as to reinforce my credentials, authority, and authenticity on Reddit. OK. So there's this. And I'll save it. And then I'll visit this.  Now, when I visit the config service, localhost 8080, sorry, 8888 forward slash reservation hyphen service, we should see that the value is immediately visible in the config service. So that's worked. But our downstream service doesn't seem to have any idea of what's just happened. And this is by design. We don't want each configuration client constantly pulling the downstream config server for fresh updates. This would create an unnecessary single point of failure. The values are read in on startup and they're cached by default. We need to force all the configuration clients to revisit the configuration, to redraw or refresh their configuration. And we can do this one of two ways. We can connect all of our microservices to an event bus. And that event bus can be used to federate changes so that all so connected microservices see the change as soon as a message is published into the bus. Fair enough. Another thing that we can do is to specifically, on this one node, one instance, call an actuator endpoint called, guess what, refresh. And that, I think, is going to be our best route for now. So what I'm doing is I'm sending an empty H2B post to forward slash refresh on my microservice node. Now what I'm going to do before I hit go, before I hit enter, I'm going to hit enter. And then as fast as my little fingers are going to let me, I'm going to hit command tab and then command R. There. I did it three times. Because I was focused on the wrong thing. Anyway, what that did was it recreated that one bean in situ. So there's one message rest controller. And it refreshed the configuration by drawing in the value from the config server. If I had injected this, I wouldn't have injected this into anything else. It's a rest controller. But any bean on which this annotation is set can be injected into any other thing, naturally. If I had done that, the bean reference itself would have still been valid. It wouldn't have suddenly been a null or something like that. Instead, this is a proxy and the internal representation of the proxy would have been discarded and recreated in situ. So this is not a great annotation to use for beans that have state. If you've got a thousand in-flight transactions, the last thing you want to do is have them savagely destroyed right in the middle of processing. This is great under the hand for stateless things like a rest controller or feature flag kind of objects. Now this is just a very sampling, a very small sampling of what we can do with the config server. The config service and this facility gives us the ability to support feature flags. I can now decouple the release of software from the deployment of that software. I can make inert certain parts of the functionality in the application, in production, and then activate them based on a flag that I can toggle at runtime. I have a server now in between my configuration source and my configuration clients. So that server can do things for me on my behalf. It can, for example, be locked down. I can lock down the config service and the config client and expect that both sides should do X519 mutual authentication using certificates or HTTP basic authentication. Or I can tell the config service that there are ciphered encrypted values on property files that once the service and the client are connected and authenticated should be symmetrically decrypted and given to the client. So I can do a lot of cool stuff by virtue of the fact that I've added this little bit of indirection. And also I've solved the centralization problem the auditing and journaling problem and the live refresh problem. So this is one very, very useful feature in a distributed system once you have more than a few services hanging around. The next thing that we care about in a distributed system is how do these services talk to each other? How do we make it easy for them to find each other? In a dynamic cloud environment, things will come and go as they need as capacity and demand dictate. So services tend to be fairly ephemeral. We need to make sure that if a service goes down that the other services that are looking for that service are able to find alternatives and that they're not trying to contact a dead node. At first blush, at first blush this might seem to be a use case for DNS. But DNS is a poor fit for a dynamic cloud environment. DNS has several limitations. Naturally it's pretty useful outside of a dynamic cloud environment, but in a cloud environment it has several limitations. The first and most obvious of which is that it requires DNS. If you want to use DNS, you need a separate server, a DNS server. Fair enough. You're going to have something like that somewhere, not to be sure, but a DNS server is typically something that's managed by other teams, operations in particular. So that reduces your ability to affect change. Another problem with DNS is that it requires resolution. Again, fair enough, but that resolution can add latency. And if you don't have that resolution, the easiest way to get rid of that resolution is by doing caching, which creates its own problems. Because now you run the risk of having stale nodes in the cluster. Again, you're going to have to solve that problem with a lot of solutions, but it's something you should care about. There's a dynamic, there's a tension there. Another problem with DNS is that it's a pretty dumb protocol. It doesn't have the ability to answer questions about the state of the service itself. If I make a call as a client to a service that isn't available, it's no longer there, I'm going to block. Hopefully, hopefully you've all read, internalized, applied, and kept under your pillow the copy of Michael Nigard's amazing book, release it. And if you've done that, then you've remembered to apply aggressive client-side time-outs and all of the connections in your code to anything, ever. Haven't you? Have you? Have you? You sure? You sure? Pop quiz, what's the default time-out for Java Net URL connection? You don't know? Yeah, see, this is why you've got to be scared. I'm not saying you should be terrified, but yeah, you should be. The Java Net URL connection has a default time-out of, like, unlimited. So if you've got 100 threads and you make 100, you've got 101 requests and they're all calling a downstream service that isn't responding, it's probably not going to be great. It's much better instead to ask a question, is that service there? And if it isn't, then I'm going to do something else, right, instead of blocking. So that's a problem with DNS itself. Another problem with DNS isn't so much a problem with DNS as load balancers with DNS. DNS load balancers and load balancers suffer from problems. DNS load balancers in particular, well, they dole out DNS entries for a resolved address. Multiple different technologies, including Java, cache that resolved DNS by default. So if I use the JDK, and I make a call to another service and I use the DNS entry for that entry, the JDK by default caches that resolved IP and it uses it for subsequent calls. You've now defeated the DNS load balancing, right? It's going to pin that request to that same node, which may go away. You can get around some of that with virtual load balancers, but whatever flavor of load balancing you're using, you still have another problem still, which is that the load balancer doesn't know about the nature of your work, of the nature of the requests. As far as the load balancer is concerned, all requests are equated equal. It sees 10 requests. It sees 10 nodes, and Robin distributes them among the 10 nodes in the cluster. But we all know that not all requests are equated equal. Some might take two minutes, some might take two milliseconds, right? You're the only one that knows the nature of these kinds of requests. So how do you build the programming logic? How do you, in your own code, address algorithmically as a heuristic? How do you address these kinds of concerns? For all of these reasons and more, while we want something like DNS, a logical mapping from a service ID to hosts and ports for that service, DNS isn't really a great choice. So to solve some of these problems, we're going to look at something called a service registry. A service registry is a phone book for the cloud. It's a logical mapping between a service ID and a collection of hosts and ports. And Spring Cloud supports many, good many different service registration implementations, or service registry implementations, not the least of which are Apache Zookeeper. How many of you have used Apache Zookeeper? Okay, all right, good stuff. What about HashiCorp console? Console, all right, that's awkward. A little bit, a little bit. What about HCD? HCD, okay. What about Netflix Eureka? Netflix Eureka, all right. So Spring Cloud has discovered client abstraction implementations for all of these. Asterisk, the implementation for HCD is not production. It's not GA yet, right, not general availability, and I don't know what it will be, so don't use that. Okay, that said, these are all great choices, and in fact there are others, right, so client foundry can be used as a service registry. There are others, but I'm a big fan of Netflix Eureka for two big reasons. The first, and perhaps most important, although it's a tight, it's a close one, it's a close call. The second important reason I'm a big fan is because it's been used at scale by one of the largest websites on the planet. So there's that. The second reason, and again, I'm not really, I'm ambivalent as to which is more important. The second is that it's really, really easy to set up and I'm very lazy. So there's that. So we're going to go to start.spring.io and I'm going to build a Eureka service registry and I'll say Eureka server, and I'm going to use the config client, the normal sort of pro forma stuff. I need to point it to the config server and start it up in my IDE and so on. So, come on. application.properties spring.cloud.config.uri equals htp localhost 8888 spring.application name equals Eureka-service and we'll rename this property file to bootstrap.properties and we're going to open up the Eureka service application and we'll say attenable-discovery-client and abracadabra-euro-eureka-server Okay? Goodie. Now if I start that up it'll spin up in port h761 if everything goes to plan, which, you know, honestly, it may not. Okay? Voila. Now, a few things worth noticing. Very well done animated GIF. We have people for that. And and no services yet registered in the registry. Nothing available for consumption. And we need to change that. The one big drawback and by that, by big, I mean very small drawback, is that service registration discovery is something that is invasive. It's minimally invasive in the case of spring cloud but your code has to be aware of it. It's not something that's buried in the bowels of the operating systems, networking stack or in the case of the JDK So, we need to activate support for service registration discovery in our code here. In our reservation service. And we can do that because we have the spring cloud discovery client abstraction here. We've got spring cloud starter eureka on the client class path and then we can just add able discovery client there and it's going to register itself for the registry. And then we can consume that service with a client and that's what we're going to do here. We're going to build a client that will talk to our downstream service by way of the registry. I'm going to use, of course, the eureka service registration discovery support. I'll use the config client. I'll use Zipkin for distributed tracing, Ravencube for stream processing, Histrix for circuit breakers, dual for micro-proxies and rest repository support and web support. And then we're going to hit generate. That'll do. And meanwhile this has come to life and we can see it registered there. We can see that there's one instance on this IP this service ID and this port now available for discovery. Had I more than one I would see up n where n could be 1 or 1,000 or 0 or whatever. So, here's my client. Here it is on this. And we might, I don't know how long we've got for time but if there's an overtime round we'll do security. Security. How did I do that? Dependency. Spring. Cloud. Starter. All off too. The memory. The normal things first, right? Spring.application.name Come on. It's just hanging. Reservation-client. Spring. Cloud.config.uri equals http localhost8888 and the normal sort of stuff. We've got to rename this property file to bootstrap.properties. And then we need to think about what we're trying to do here. Naturally we want to take advantage of service registration and discovery but we're building an edge service. This is no meager regular standard client. This is a service that is meant to sit at the edge of the architecture. It is meant to be the first port of call for services coming in from the outside. And there's a good reason for this. You see in the 24-7 internet of things in the in the world where we have basically almost everything has an IP. In the world where basically almost everything has an IP address. We have lots of different clients out there. Even the most conservative and boring of organizations today have it minimal. There are 5 iOS and Android and an HTML5 user experience at minimal. A lot of times they have more and the internet of things makes this very simple. There are smart TVs and playstations and Roku's and Xboxes and cars now and watches and your tablets and the streets in Singapore have IP addresses because of the ERP. Even those have sensors. The sensors are everywhere. You'd be astonished by the things that have IP addresses today. There are organs. There are people with human beings with organs in them that have IP addresses now. Seriously. Just an amazing array of things that are out there that have different security and payload and protocol restrictions. We need to accommodate these different clients but if doing so meant that we had to change every single microservice we'd be very, very quickly in a bad position where we have to retrofit every single microservice for every new client. Which is a non-starter. We'd lose some of the autonomy we're trying to gain in the first place. So instead we need to adapt those requests. Adapt those connections to those downstream services at the edge. The first example of this the first kind of edge service that we're going to look at here is called a Mycoproxy. A Mycoproxy blindly forwards requests back and forth from the outside to the downstream services and then back again. Think about HTML5. How many of you have seen JS Linux?  As a demonstration I submit to you this. This is a project called JS Linux which was created by a guy named Fabrice Bellaud in 2011 who upon the introduction of typed integers and the ability to do bit twiddling in 2011 in JavaScript built an x86 machine code emulator. So right now the latency you're seeing is not anything I have any control over. It's the Wi-Fi here. It's downloading a Linux kernel which to be fair can take a few seconds. So we'll just wait awkwardly. There we go. It downloaded a Linux kernel and then entirely in the client side JavaScript it boot booted the Linux kernel. This is an HTML div though. Right? The TTYS device is a Chrome HTML DOM. Right? So if I now go here LSLA vihello.c Let me escape this. Escape I because I'm in vi. Hello SINGLE SUG SUG ESCAPE ESCAPE WQ TCC minus O hello hello.c that's a tiny C compiler because the GCC takes a little bit longer if I don't have time. And now it's compiling the C code because why not? And then we're going to run it. My point here my friends is that your modern HTML5 browser is really really powerful and you don't need to babysit it you don't need to hold its hand you don't need to serve it resource constrained or limited payloads you can do an unnatural amount of things that said that said it lives inside of a sandbox and that sandbox precludes it from calling different services on different hosts. This is part of the security model for browsers which makes sense good enough you can get around this a couple of different ways that you can do a service that exempts across origin requests to that service a policy that allows the clients that want to talk to it to call it or you can just proxy all requests back and forth because the first option would require you to change every single microservice to accommodate every new client which is as we say kind of a non-starter once you get more than a handful of services up and running. So let's look at our micro proxy approach we're going to use Zool now I'm going to comment out a few dependencies for efficiency here now Zool is a project from Netflix it's we integrate it with Spring Cloud and with Spring Cloud to discover client abstraction now Zool was named of course for Ghostbusters I don't know if you've ever seen that movie but you'll know that Zool is the gatekeeper or the underworld to hell the proxy to hell now I think he's from hell that was kind of the implication it's been a while since I've seen the movie if I'm honest my friend he said you know Josh that's not really fair we don't know that he wasn't sure it was stated in the movie either he might come from a very nice place he might come from Bali like it's not fair to judge so I understand and I take it back anyway he comes from somewhere he's a proxy to that world wherever that is so now the application is up and running on my edge service and because of the integration and the awareness that Spring Cloud has it's up and running on port 9999 and while we could specify arbitrary routes Zool in conjunction with Spring Cloud and Spring Cloud Discovery Client sets up automatic routes for us so 9999 forward slash reservations forward slash reservations there we are so there's my edge service on this port on the service ID and this context path this is the same service as I've just called or just stood up a few minutes ago on 48,000 so notice that when I go to 8,000 I get the original service when I go to the edge service on 9999 I get the proxy service same payloads we have at this point first of all how does it know which instance to route the request to how does it pick that instance right now there's only one instance in the registry so it sends a reason that it's going to use that one instance but what if there were two or five or ten or a thousand or n greater than one in this case it has to make the decision on the client and does this using something called Netflix ribbon Netflix ribbon is a client side load balancer it codifies as objects the strategies that are used to make the routing decisions the benefit here is that you have an object you can plug in the strategy yourself now you can do more interesting things by the way, does anybody know how to turn the air conditioning on in this thing air conditioning the spice of life thanks so there's a load balancer there's a client side load balancer that has different objects and different strategies for doing load balancing by default it's doing random robin which is probably what you wanted you can do all sorts of interesting things you can do weighted response based load balancing or you can provide your own strategy that does something for example data sharding you might want to pin requests from a client to a specific node maybe you have an OAuth token a stateless OAuth token you want to take this OAuth token and route it to a specific node maybe you're doing something stateful like streaming video on a certain node maybe it's a Netflix use case you want to pin all subsequent requests to that particular node because streaming video is stateful you can't load balance that whatever your business use case is you can program that, you can design it define it and then reuse that strategy in ribbon and it gets plugged in for you automatically in this case by the zoomicoproxy the other thing you may be wondering is how did these URLs get rewritten and in this case there's a little bit of sleight of hand the downstream service my rest repository is smart enough to look for the incoming request and look in the headers and it examines the request headers and it uses one of the headers which provides the origin URL to rewrite the links and you can do the same thing in your own custom services you can see the origin URL so from the perspective of the client it has no idea that this JSON originated on anything besides localhost 99.99 good enough this might be enough if you've got if you've got ubiquitous sort of HTTP and JSON this might be enough, you might be done you might be able to go home you should probably probably use some sort of authentication mechanism maybe HTTP basic or OAuth you should absolutely, no questions asked use HTTPS SSL especially considering that it's a requirement that HTTP2 so you're going to have to get there one way or another that said maybe that's enough maybe we've just done all that we need to do to support our HTML5 clients if that's the case then good but sometimes clients require different views of the data they require different parts of the API they require different views of the data synthesized from different services etc and this requires this requires that we do some sort of translation or mediation or transformation right so what we're going to do can you can you stop thanks what we're going to do is build a edge service that calls the downstream services and does some sort of translation or transformation this is called an API gateway an API gateway is like a micro proxy except it does transformation or translation so we're finally proxying things back and forth we're going to build a very very simple API gateway something that just calls the downstream service so reservation API gateway REST controller private final REST template and we're going to map this as a REST controller like this at request mapping I'm going to map it to forward slash reservations I'm going to say at request mapping it's called value forward slash names so now I'm going to have an endpoint here that's going to return a collection of names and right now I'm not going to return anything but what we need to know is that I'm going to use the REST template spring REST template to make the REST call to the downstream service to get the JSON and then we're going to turn it back into a collection of hyper media resource envelope objects with a payload of type reservation whose reservation name we're going to unpack and then put into a collection and send back so we're going to create a simple endpoint that just returns a collection of names very very simple I need this REST template to be aware of my client side load balancing the REST template is not by default aware of that so we need to configure a REST template that knows how to do load balancing that's a fairly trivial affair we just need to create the right beam like this and say at load balanced at load balanced is a qualifier annotation that spring will use to spring cloud will perceive and it will use it as a flag that you want spring cloud to configure an interceptor on the REST template an interceptor that is going to be smart enough to preprocess the request extract from the host parameter or part of the URL the string and then pass that to the registry then get back a collection of service instances and then pass that to ribbon to do client side load balancing so it will make the decision based on all the different services that are in the registry all in the blink of an eye so we will see that here we will say this dot REST template dot exchange HDP reservation hyphen service forward slash reservations get no so we are not going to make an HDP put or post so we are going to say no there is no body and then finally for this final parameter we need to tell the REST template what kind of payload what kind of data we want to send back right we could say that we want the JSON from the downstream service this is the service ID by the way not DNS we could say that we want the JSON to be turned into a string in which case the response from the REST template would be a string we would get the JSON delivered to us as a string we could say that we want the data to be turned as a map whose payload whose keys rather are the attributes of the JSON and whose values are more maps we could say that we want the value to be turned as a JSON node this is a document object model like structure and we can programatically interrogate the tree if we like but what I really want more than anything in the whole wide world is to get back a collection of spring HODOS resource objects whose payload is of type reservation and this won't work for two reasons the first of which of course the most natural is that we don't have this type right this type doesn't exist on the class path I could naturally copy the type from the service implementation but for a couple of reasons this is a poor fit the first is I would be unnecessarily coupling my client side representation to the implementation on the service side that would leak details perhaps the JPA implementation details as well both of which are undesirable the other reason that I don't want to copy the implementation from the service side is because I can't necessarily be sure that I have an implementation to copy from the service side I've heard but I have never seen or confirmed a very interesting theory my friend a theory in the same way that we all regarded for the longest time it's something that hasn't been since substantiated by science but again it's theoretically possible so I broach it here because I want you to be prepared for something that may happen in theory in theory again skeptical skepticism is warranted but in theory the service implementation could be it could be implemented in something besides spring and job and again I've seen that's never happened so it's not really an issue but it could be one of those other theoretical non spring Java technologies that I've heard about but I've never seen and if that happens and we can't take for granted that we even have a spring Java type to use on a client so there's that the other reason this doesn't work is because of well Java's lack of a generic type type system we need to deep dive into that we need to take a moment and take a look at some basic Java isms what my friends is the generic parameter for this instance variable at runtime what is T at runtime that one, yeah there's no so as a compromise made by the Java language designers in Java 5 more than 10 years ago Java does type erasure it burns away the type parameters for instance erasure as far as Java is concerned you wrote list X equals new array list there's no generic parameter not even object because object is a constraint it has subtle nuanced differences between that and untyped array list you can get around this by creating a subclass you can say class X extends array list of string right and then say list of string X equals new X T in this case is a string but it's not it's not a not exactly what you want right you can also do an anonymous subclass at array list curly bracket curly bracket there you go trim to size so there I'm creating a subclass I'm overriding it right overriding so this actually works but again kind of ugly if you understand what we're doing here that we're creating an anonymous subclass to capture generic information to reify the generic parameter then you know what we need to do this hack, I'm sorry design pattern is called the type token hack pattern and it's supported well supported by different libraries different technologies it's by no means something that we innovated in the spring ecosystem it's just something you have to do with Java the parameterized type reference the parameterized type reference you know builds upon what I just showed you it requires you to create a subclass if I fail to subclass it, it doesn't compile I don't have to override any methods it's just that I have to subclass it it's abstract right in and so doing I can then ask the PTR I can say give me your java lang reflect type and java lang reflect type of course is the parent of java lang class which is all we need to answer this question so I can give the rest template that parameter there and I can say return respondentity.get status code, no thanks, value, no thanks headers, nope, I want the body I want the content I want to stream over it, I want a map from R to R of reservation name I want to collect the results into a list and then return that back okay my friends we have a very simple api gateway three lines, three admittedly very very very gnarly lines but three lines and if I restart this we should be able to confirm as much now this will work just fine but it ignores some of the realities of distributed systems we very conveniently ignore the fact that while this will work fine and the happy path being that we have one or more instances of that service available it ignores the fact that something is going to go terribly wrong if we have no services over which to load balance what's going to happen if I try and load by load balance by zero what happens if I divide by zero so to speak if I have zero instances of the service it's not going to be good I'm going to get a big fat java stack trace of my phone my iPhone users aren't going to like that at all so we need to be a little bit more honest with ourselves and understand that failure is a statistical inevitability it will happen it's not a matter of if it's a matter of when in a sufficiently distributed system it is futile to try and build a system on the predicated idea that your systems will always be available as you add more capacity the likelihood of something being down at some point goes up if you have a few dozen nodes that can be minutes easily per day if you have a few hundred nodes that can be hours per day it can be days per year if you have enough nodes at some scales it becomes something that's a statistical guarantee and high performing organizations understand this they internalize it there are people that wake up every day every day they go to their Google underwear and their Google t-shirts and their Google socks they go to the Google shuttle they go to the Google headquarters while talking to their Google colleagues and checking their Google mail on their Google phones when they get to the Google headquarters they may talk to other Google colleagues while having Google gourmet breakfast at the Google cafeteria on occasion as it's one to happen accidents will happen they accidentally spill some of the voluminous Google Kool-Aid on their Google t-shirts that Google Kool-Aid does get everywhere after all if they spill the Google Kool-Aid in their Google t-shirts that may necessitate a trip to the Google laundromat if they go to the Google laundromat well that'll delay them even further and in the meantime they're gonna have to stroll about in their Google gold sequined bathroom waiting for their Google t-shirt to finish and then having done that they may since they're not quitters get back on their horse and go back to the Google cafeteria and get on with breakfast and after that they may then decide since it is getting on path three o'clock in the afternoon to go to Google work and this may mean getting on a Google bicycle or Google shuttle or Google bus or Google boat to the Google data centers where they will walk the Google data center aisles and find completely and utterly unsurprisingly to them more than two million enterprise grade production servers deployed in Google data centers all around the world that some of their Google servers have Google died and they don't care they don't care that some of their services died they didn't care at eight in the morning when they got the alert that told them that they don't care because they didn't build their services to be sensitive to the loss of a few nodes this is why they're Google and we need to do the same thing we need to understand that failure will happen Netflix understands this fact as well they have a suite of software called the Simeon Army the Simeon Army are basically little agents of chaos they purposefully loose them on production during daylight hours, during office hours when people are on hand, able to respond these things do terrible little things in production they kill minus nine processes RMRF, database and disk partitions they've even got one that as I describe it to you now, gives me goosebumps it's called Chaos Calm and it purposefully kills a whole data center availability zone they do this to themselves during office hours to make sure that if there's a problem in the failover strategies that there's somebody on hand to be able to respond to it they'd much rather find out any kinks in the process during office hours when there are people there than at four in the morning when people are trying to sleep these organizations and many others besides they have a mantra, a motto they say, you build it, you run it you're an adult, you can do whatever you want to solve the problem at hand which by the by I think speaks volumes about why organizations like Netflix use Spring Boot and Spring Cloud at scale you're an adult, you can use the technology you want to solve the problem but if something goes wrong you're going to get the phone call you build it, you run it you see we live in a different era right now we live in a different time gone, gone, gone are the days when we had two different teams operations and developers no longer are the developers exclusively charged with delivering business differentiating functionality and then throwing their product over the wall for the operations to deploy and no longer is operations exclusively charged with ensuring functioning a production environment no longer is operations the scene-eaters for so long we would take our terrible, terrible buggy code and throw it over the wall and those poor, poor schlubs down the stream would have to make sure it worked in production and if they ran it then it's their fault now but no longer are they the ones responsible for saying no, we're not going to do it they have to help the business move forward too their job is not to say no to everything because it's all going to fall apart we live in a different world now we have a word for this we have the word DevOps DevOps as I'm sure you all know I'm sure you all know this DevOps is an ancient very ancient Malaysian word, it means empathy empathy for operations and developers empathy for each other on the team it's a very ancient Malaysian word in this world it is no longer enough just to build code that works okay it has to be corner case friendly it has to understand that it has to embrace the fact that failure will happen and it has to do the right thing in the face of failure that is a key tenet of a cloud native system so looking at our example here we've got an example that's doing load balancing across different nodes what are we going to do if we have zero instances of that service it's going to throw a big fat stack trace, it's going to throw a big fat exception and we're going to get a big fat Java stack trace and we need to do the right thing to counter that I want to make sure that if something goes wrong that we have a fallback behavior so I'm going to introduce what's called a circuit breaker and we can do this because we have on our class path the Netflix Histrix Circuit Breaker I'm going to say atNable Circuit Breaker and I'm going to create a fallback method and the high performing websites they do this kind of thing all the time they'll say, oh well, you went to the search engine service but the search engine isn't available so we'll give you some machine learn recommendations from across the web here you are it's better than nothing it's certainly better than a big fat Java stack trace in the browser experience this will give us what we want for the read use case it'll give us a fallback it'll give us an empty collection for the read, but what about a write what happens if we try to write the data to the downstream service how do we post that data how do we confirm or guarantee that the data is being delivered to the downstream service now before we do that let's take a look at what's happening here I'm going to kill the reservation service poor thing and as I do that you can see it's hesitating you can see it's hesitating and I get the I get the array list eventually command R refresh etc so it's timing up and then I get the exception then it gives me the fallback if I deluge the downstream service there if I deluge the downstream service eventually it just stops trying and it gives me the fallback directly this helps protect our downstream service we all know of course that if a website isn't responding the best thing you can do is to refresh the browser a lot is that true? no? well it doesn't work in distributed systems either so this defeats that effect this defeats the unnecessary traffic that's helping that's preventing our service from coming back online if you've got something like CloudFundry a platform that is built to operationalize applications then CloudFundry will move having an earth all day and all night to make sure that if you say I want 10 instances of the service running at all times that darn it there are 10 instances of the service running at all times but we need to build our software to be to be resilient in the face of topology changes so now let's go back to that question of what about rights how do we guarantee that if you have a downstream service and we want to send data to that downstream service how do we guarantee that that data gets there eventually you can't really circuit break that if I post to my edge service my downstream service isn't available what do I do this is a fundamentally classic you know ancient kind of question this is something that's not a new idea it's not a new problem what we need to do is to get two nodes separated by network partition to agree upon state classic problem there are many different ways to solve this problem many many many many different ways to solve it one solution that doesn't really solve it is to use distributed transactions if you ever feel the urge to use distributed transactions that ever strikes you as an idea I urge you to stop go home drink some absinthe pet the cat or the dog hug your family and reconsider your terrible life choices and then come back perhaps the next day or next month after you've reconsidered and try again for you see distributed transactions as modeled by JTA which is the client side middleware for the X Open protocol are a terrible idea they just introduce another single point of failure and they gum up the works unnecessarily in an otherwise fast-moving system avoid them at all costs there are superior alternatives newer but still superior alternatives there is for example the saga pattern the saga pattern was introduced in the 80s by 30 years ago it's still a better idea the saga pattern says that if you can design or model your system as a set of interleavable that is to say reorderable transactions and then guarantee or rather assign for each one of these interleavable transactions a compensatory transaction that is to say a transaction that semantically sets the system back to a well-known state and if you can make it so that those compensatory transactions are edempotent that is to say you can re-apply them multiple times then you can build a system that will guarantee consistency eventually think about the kayak.com use case kayak.com or orbits.com or whatever it's a website where you can go and book a hotel, a car, and a flight all at the same time if you have this use case well first of all the X Open protocol doesn't apply these services are all based on risk they don't implement X Open so that's a non-option anyway so if you have these different transactions I call the hotel website I call the car or the flight website I call the hotel car flight I call three if one of them fails I cancel the bookings for the car or the hotel and the flight that's an example of using the saga pattern and that works there's lots of different implementations for that in our case we don't even need to go that far we can use something much simpler we can use eventual consistency eventual consistency is a five dollar word for messaging we're going to do it right behind we're going to publish a message from the producer to the downstream consumer this is messaging, it's classic 40 years plus we can use spring integration spring integration has at its heart the concept of a message channel is a Java Util Q basically it's a named conduit through which messages pass and you can compose message channels, you can compose interesting solutions by stitching together components in between these different channels I might have a component that takes messages in from the outside turns it into a spring framework message passes it through message channel sends it out the other side of a message channel does some sort of enrichment or filtering or splitting or aggregation etc etc spring integration by its very nature is meant to handle integration use cases so it'll work with all manner of different systems whether they're necessarily event based or not all things are event based but the API may not be so friendly to being event based so it'll let you talk to an FTP server or email or iNAP or Twitter or a file system or anything or a message broker we have to wire up this configuration we have to wire up the inbound adapters and the outbound adapters which is fine but this is a little too low level for our purposes we're not going to use FTP are we we're not going to use FTP as the broker between microservices I'm not going to email my right from my edge service to my downstream reservation service either so spring integration however useful it's a little low level we can move up the stack and the abstraction a little bit if we're willing to take for granted the fact that we're going to use a highly concurrent, highly efficient message broker something like RabbitMQ or Apache Kafka or Redis even if we're willing to do that then we can get a lot of benefits we can use something called Spring Cloud Stream Spring Cloud Stream makes declarative and a matter of convention and configuration the wiring of these different channels we define the channels and then work with them declaratively we don't have to worry about how these things get stitched together so let's take a look at that right now now I've got here the reservation service where's my reservation client in my reservation client I'm going to define I'm going to go back to my build here I'm going to bring in these last two dependencies for now I can say interface reservation client channels and I can say that I want to define a channel that's going to be going out I'm going to send messages on the output channel so it'll be called output message channel output and this is a one channel I could have as many channel definitions in this interface as I'd like but in this case it suffices to leave it as is and I tell Spring Cloud Stream that I want this definition to be hydrated I want these channel definitions to be created for me by the framework by the abstraction and I know that I'm going to use RabbitMQ because I've elected to do so here in my stream binding definition implementation I'll show you in a minute how this configuration actually corresponds to the broker itself but for now it suffices to say that Spring Boot has automatically configured a RabbitMQ connection factory we could of course customize that by providing properties and application of properties and then we'll look at that in a minute now in my code here I can now create a new endpoint public void write request body reservation reservation I can say at request mapping method equals post and I can say private final message channel out and in my constructor here I'm going to inject that channel that reservation client channels this.outClientChannels.output okay and in my code here I can say let's create a message of type string message equals message builder .withPayload reservation.getReservationName .build and then let's send the message on the output channel like that so that's just basic spring negation messaging code I'm publishing a message on the channel I'm not worrying too much about how that channel is being connected to that downstream service on the other side on the service the consumer the reservation service is the consumer of the data I need to bring back in my spring cloud stream rabbit and queue binding which I do have and I define the input channel instead so I'll say interface reservation service channels and I'll say atInputSubscribableChannel oops input and I'll say atNableBindingReservationServiceChannels.class and here I'm going to say message endpoint class reservation processor public void onNewReservation message string message okay service activator inputChannel equals input and all I'm trying to do is whenever there's an incoming message I'm going to write the data to the database using my repository so I'll say this.ReservationRepository saveNewReservation message.getPayload etc so that's pretty straightforward I think now let's go ahead and start up the reservation service and we'll take a look at the configuration that's making this all work so we can visit the reservation service configuration or the reservation client configuration here and we can see that it says spring cloud stream bindings output destination equals reservations output is the name of the channel definition in the broker reservations is the name of the exchange or the sorry output is the name of the channel in the Java code reservations is the memory in the broker upon which both the producer and the consumer need to agree this has to be the same for both the client and the service you'll see that we've got that mine of course spring cloud bindings input destination equals reservations these other lines are what are more interesting you see by default spring cloud stream bindings are publish, subscribe one to many, broadcast so as I publish a message as I produce a message to a spring cloud stream the consumer will by default broadcast that message across all the different sorry all consumers will get the same message if you have 10 consumers listening for messages on the exchange then you'll get 10 different 10 different copies of the same message which we don't want so this is what this is, this is a group it says for all the consumers in the same group in this case the reservations group deliver the message only once to any one node so you'll get at most once semantics instead of load balancing or instead of duplicating broadcast messaging you'll get point to point semantics here we're saying that the subscription is durable this says if the downstream service isn't available then the broker will re-deliver it as soon as one of the instances becomes available this guarantees that even if the service isn't available the message eventually gets delivered which is what we want in this case so that's the service what about the client is it possible to get the air conditioning turned on or do we talk about that is that a no is it just me or is it warm in here it is hot right, open the doors open the door open the window, I mean just anything I'd like to just break them please, it's staggeringly siphoning the hot okay I mean thanks maybe the air conditioning is just for those people out there not for us is it going on out there is the air conditioning going on out there yes oh it's just not in here yeah open the yeah if we open the doors that'll be that's a hotspot that's a hotspot you're sitting at a hotspot air conditioning room right the temperature distribution is not uniform right you're going to be sitting at a hotspot so too bad are you air conditioned oh I just thought half as much as you did that's true but yeah I'm not running WebSphere like my computer's not hot I shouldn't I should be cool too anyway goody so now I've got I've got the let's see we've got local host 99.99, reservation names there's my reader right, that's the reader let's look at the writer, so I'm going to send some of my favorite doctors to the downstream service via the edge service I'm going to post some data to the edge service that we just stood up the API gateway endpoint so we'll say quote quote reservation name and I'm going to send a few of my favorite doctors so doctor who and then I'll say minus h content hyphen type jason and I'll send it to the local host 99.99 reservation names what did I do oh, reservations, thank you so there's that doctor who and we'll send a few more of my favorite doctors so Dr. Seuss Dr. Strange there they are so that's working in the happy path but now what happens if we kill our poor reservation service yet again okay, goodbye now those three doctors are gone they're no longer there because we've killed the process and remember it had an in-memory embedded SQL database so that data is gone I'm going to now send three more of my favorite doctors Dr. Pollock he's the founder of spring.net, a spring framework contributor co-founder of spring MQP and spring XD and spring cloud data flow I'll send Dr. Sire is a founder of spring batch he's a spring framework committer co-founder of spring boot and spring cloud and also spring MQP and then we're going to send Dr. Subramanian now Dr. Subramanian is just the coolest cat I mean they're all really cool but he tends to see some non-spring technologist Dr. Subramanian is legit there we go so now I've delivered those three doctors but they're not going to be delivered to the downstream service as the downstream service doesn't exist so what's going to happen we're going to see that reflected hopefully in the re-delivery of those messages when the service starts up now consider that we have these 12 records here so the command line winner is going to get run when the application starts up we're going to see those 12 records registered in the database we're going to also see that those doctors were delivered before the command line winner gets run so we should see here doctors, Paulic, Sire and then interleaving because of concurrency and then Dr. Subramanian so there we go so that works we've now built a system that is resilient in the case of failure for both reads and writes we've done something more robust I'm a big believer in style though so let's take a brief moment to clean up this code a little bit to make it a little bit more elegant what I want is to simplify the code, the surface area of the code that's being used to talk to these downstream services right now right now I've only got two endpoints I've got a post that I'm doing that's sending a message via RabbitMQ and I've got a read that's using the rest template it's only one endpoint for both rest and for messaging it's not so bad but what happens if I had more than a few endpoints more than a few interactions as it's going to reasonably happen in a distributed system I'd have to rewrite this logic for every single client of that service which could become very tedious very quickly and one option of course is for the teams that build the service to also build a client but that risks having too much magic or too much business logic the client code the team that builds the API may treat the client as a crutch the service should have all that it needs to do its work if the client is enforcing business logic in the Java code then it's going to be very hard to port that to for example another language to use that same API so you want to make sure that the client is as minimal as possible so for this reason Amazon has famously said no teams should build clients for their own APIs others will do it organizations like Netflix are kind of they have the same reservations but they understand that sometimes it's useful for the team that ships the API to also build a few useful client bindings but they don't want too much magic in the client code as well so they make it as easy as possible to build those client interfaces automatically one way they do that for REST is to use something called Fane Fane is a way of building declarative REST clients which means to pretend to play dead for example if you're if you see an animal in the forest like that then that animal is feigning dead he's pretending to be dead like WebSphere feigns utility it's not utility but it tries to so we're going to create a declarative REST client here we're going to say interface and we're going to say that we want to call an endpoint by calling the read method and that endpoint is going to the call that we're going to make is an htb get call and it's going to call the reservations endpoint and it's going to call the reservation hyphen service and this will benefit from the same kind of load balancing and all that we've seen before it benefits from spring clouds awareness of the registry and of ribbon so that's automatically done for us and we just need to activate feign here by saying at navel feign client now of course I've done the wrong thing I've built my feign client on the service which makes no sense so let's copy that here and voila ok so put that over there and we'll go to our build and we'll make sure that we have spring cloud feign here this is the service let me get rid of that feign dependency there because we don't eat it now there's the client this is the feign client that'll work for the reed use case and we can now throw away the rest hi boss thank you green tea too you have no idea poor surgical this poor poor guy all day I've just been like because I quit coffee I don't drink soda this has minimal green tea minimal black tea minimal caffeine whereas black tea has a lot so this has been and you mean my you want one more? maybe later ok sure it's like you read my mind ok so now I've got a rest template here I no longer need that I've got my feign client so I'm going to say reservation reader nope wrong reader and let me see I can get rid of all of that and I can say return dot read I need to inject that reservation reader into the constructor so there's this we'll say reservation reader and there we go we'll say reservation reader down here this dot reader equals reservation reader but what about the write the writing has been done using messaging so I'm publishing the message to my downstream service in here too it's only three lines so that would very quickly add up if I started to have more than a few interaction models so I can get the same effect I can't use feign directly but I can use spring integration messaging gateways a messaging gateway an enterprise application integration parlance is a facade for messaging logic so that the client can use the downstream messaging system without being too intrinsically aware of it so we can get the same thing ok no we have to 10 minutes anyway we can build a writer so we can say reservation writer void write string rn and it's a gateway so the gateway is going to send messages on the output channel the payload will be sent on the output channel and it's going to be it's going to be workable because we're going to have integration component scan on the class path and then that means we can throw away this code as well so all that stuff goes away I can say this dot final reservation writer this dot reservation writer dot write and sorry this dot reservation writer equals we're just passing the writer so so there we are now that code becomes market to the simper as well we can say this dot reservation writer dot write reservation dot get reservation name and there we go so that's the same code it's just I think it's a little easier on the eye and it's the cognitive overhead of understanding what's happening is far diminished we don't have to think too much about what logic is happening here we can just look at the declarative interface and at this point there's no real risk in the teams that built this API if they shipped a client that was based on those because the hard part is not being done by something you have to parse or understand anyway so we can restart now while that's restarting let's take a look at the last that last tenant of a cloud native system and I talked about it a little bit earlier but we need to revisit this idea of observability you see thus far we've built a system that lends itself to agile and easy evolution we've built a system that benefits from elasticity and the dynamicism of cloud environment and we've built a system that is resilient and robust in the face of service outages and topology changes and we have to a point by using the actuator built a system that is observable that is to say I can monitor it from its outputs but we haven't captured emergent behavior of the system you see the map is not the terrain if you're walking around here in beautiful Singapore is that the same thing is that the same thing as looking at a google map of Singapore I submit that there's a lot more interesting stuff happening on the ground walking amongst the people walking amongst all the wonderful hawker stalls and smelling all the great food and listening to the hustle and bustle of the people in the city or in the malls that is far more vivid and more interesting than the google map of Singapore the map is not the terrain it's not the same thing the same is true for your system the architecture diagram for your system is very different from the emergent behavior that you can only hope to capture by observing the system in production and we need to do that we need to understand what the system is doing not just individual nodes but the interplay between services in the system one natural place to do that in the future represent the connective tissue if you will between one service and another and that's great for services over which you have the ability to influence or inform observability monitoring and security and so on but it's even better for services over which you don't have that control you see if I have a circuit breaker that circuit breaker may be something that's connecting to another service that may not be available to change how that service is behaving or change its resilience qualities or to add monitoring to it then I can at least add monitoring to the circuit breaker I can protect my system that's what we're trying to do here so we're going to build a dashboard that'll show us the flow of data through that circuit breaker local host 8888 spring.application name equals histrix dashboard and we'll rename this bootstrapped up properties we'll open up configure this rather to use the dashboard and what this dashboard is going to expect is a service end event heartbeat stream heartbeat stream that the circuit breaker on the edge service will emit for us automatically that heartbeat stream is never ending oh for that matter there so that heartbeat stream is never ending it goes on and on and on forever it has no end it is without cease it has no bounds no limit it continues forever like the skies like the seas and the stars and the oceans like the bugs in your code just infinite just unlimited whatever you do whatever you do do not and I cannot stress this enough do not curl this end point now what we're going to do is we're going to put this in our histrix in our histrix dashboard here I'm going to paste this here like so, so there we are we're going to hit monitor and having done that we can go to the edge service and drive some traffic to see what's happening now as I drive traffic on the left you can see a moving average spiking ever upwards on the right 21, 28, 52, etc and things are happy this is a happy path 0% of the errors 0% of the requests that are going through are having any errors there's no trouble right now so it's working and as I let go of the gas I can see the moving average going downward if I were to kill eventually my circuit breaker would open up and I'd see the fallback mechanism kick in and this would say open so this is one way to get visibility as to what's happening in the system now my favorite approach to doing this is to use distributed tracing distributed tracing in theory is very simple what you want to do is for every request that enters or exits a node in the system you want to ensure that there's a unique ID by which you can correlate the flow of a message through a system a unique ID that you have fixed for every message on entry or exit at every ingress or egress point in the architecture and this sounds it sounds simple but in point of fact it's actually quite difficult you see think about all places where messages enter or exit the system right now in spring MEC in the zoomicoproxy via the rest template via the feign client even in this very simple system there's a lot of places where we'd have to write the right logic to intercept all that behavior and we'd have to do so in every node in which we want this behavior, this effect so this isn't a good idea it's not something you are in the business that's trying to solve anyway this is a concern that has to be dealt with and it's useful for understanding what's happening but it's not something you should be writing we have an abstraction called spring cloud sleuth like a detective with sleuth spring cloud sleuth is a sorry distributed tracing abstraction and it's already in play on the class path of my applications by virtue of the fact that I've got here spring cloud starter zipkin and we'll get to what zipkin is in a second but you can already see here on the console the results of this we have the service ID, the trace ID and the span ID the trace ID is the aggregate journey it's the unique ID that represents the aggregate journey from A to Z from the beginning of our system to the very end of our system from the beginning of the journey of a request to the very end of the journey of a request the span ID on the other hand is a unique ID that represents each hop in the journey from A to B from B to C, from C to D from D to E, etc this is interesting if you've got something like cloud finder it's that simple to drain your logs for all your different services like for example Elasticsearch or Splunk or PaperTrail and there you can do log archeology you can follow the flow of a message by sifting through the logs and finding what these log IDs are now I'm a big believer that a picture is worth a thousand spans instead we're going to use the the zipkin distributed tracing technology so I'm going to say zipkin server zipkin UI config client Eureka and hit generate so there we are at navel zipkin server application you okay? so now that's going to spin up on port 9411 9411 faster people are waiting okay so right now it's not aware of our services yet but we can make it aware by driving some traffic through it there we are now if I click on this you can see it knows about my client it knows about my service let's make some posts here Dr. Supermanian Sire Pollock so now if I refresh that we can see that it knows about the different endpoints that have been hit in the different services I can click on the reservations endpoint here hit find trace and I can see that for example less than a minute ago there was a request that had five spans and you can follow the flow of the message through the system you can see that the aggregate journey the total journey took 26.2 milliseconds and that the message started at the reservation client going and I like to read it like this I say read it from the service ID on the left going to the endpoint on the right so it's at the reservation client going to the reservation names endpoint at the reservation client going to the names method going to the reservation endpoint at the reservation service going to the spring data rest collection resource endpoint and if I click on individual parts here I can see the log of the message through the system I can see the ingress and egress log I can see metadata about the request itself for example the tags that are used to uniquely identify the service I like microphones so I can see the individual tags now these tags identify the request the nature of the request and you can contribute custom tags as well so that you can easily correlate business transactions to request if you like this also works for the messaging stuff right so here it says output I'm going to go to the message channel called output I click on this and it says that I made a request that took 37 milliseconds my goodness that's very slow let's try another one that's more representative of 2 minutes ago 7 milliseconds, yeah so this is the Rabrim Q Rabrim Q post going from the reservation client to the reservation's endpoint which then went to the right method which then went on the output channel and then eventually it arrives on the input channel but there's a bit of a gap here can anybody tell me what that gap is usually these are overlapping timings that makes sense right it's what happened why is there a gap why is there a gap between the output channel and the reservation client and the input channel and the reservation service the spring innovation messaging channels what? yeah kind of I mean Rabrim Q where network in this case means Rabrim Q so the message was delivered on the output channel and then it transited between the producer and the consumer transited through Rabrim Q and then it got delivered to the other side so we don't have any instrumentation in Rabrim Q but you can kind of figure out what's happening there this is going to happen for anything generally async normally in a synchronous transaction the start of the transaction necessarily takes longer than the sub transaction underneath it like if I have a transaction that starts from here it takes 20 seconds and then 5 seconds into it I call another service and I synchronously wait for the response naturally the outer transaction is going to be at least as long as the inner transaction with async that's not necessarily true or if it is true you don't know it until you see these gaps that's what's happening here in the Rabrim Q now this information is useful you get this pretty graph which I like the graph shows you the ontology I can see that the reservation client is used by the reservation or uses the reservation service I've made 17 calls etc but this is not for customer service to find out what Jane did on the website 5 years ago most organizations don't keep more than a week's worth of this information on hand and even that should be a subset of the information that's in the system by default spring cloud sleuth captures 10% of the request in the system twitter who originated Zipkin Zipkin is their open source distributed tracing platform twitter captures something like 1 out of every few million requests so the goal here is to find patterns emerging in the behavior of the system how much money did we make in the last quarter no idea but I can tell you what the average latency on the website has been for the last hour this gives you that visibility that systemic emergent behavior my friends I think that's a kind of a good place to cut it you see we've looked at the four tenets of a cloud native system we've looked at how to build a system that is agile and lends itself to easy evolution the elasticity of a dynamic cloud environment we've looked at how to build a system that does the right thing in the face of failures and we've looked at how to build a system that is observable I wish we had more time so that we could actually cover something I really feel like we've just skimmed the photo which itself is just of water we didn't even skim the water we just skimmed the photo of the water but maybe next time how do we more time for example distributed secure sign-on single sign-on federated sign-on using spring cloud security for OAuth how do I handle protecting my different microservices and rejecting requests that are otherwise unauthenticated we might be able to look at spring cloud cluster whose functionality has now been moved into spring innovation to handle concerns like leadership election and leader node worker node promotion and the emotion if we had more time we might look at how we can take our microservices as we've built them here using spring cloud stream and compose them together to form more interesting solutions remember that spring cloud stream input and spring cloud stream output are basically commoditized transports what else do you know that has a commoditized transport that composes very nicely standard in and standard out come to mind if you've ever used bash then you know that it can build interesting solutions by composing little atoms of functionality the same holds true for messaging based microservices and we have a technology called spring cloud data flow which allows you to orchestrate these kinds of messaging based microservices into more complex stream based stream processing solutions but we don't have time fair enough I'm happy to take any questions and I do want to know before you ask me questions did you like any of this did any of this seem relevant or useful to you ok good because I'm a big fan of this stuff and I would love it if you liked it as well I'm obviously a big fan I'm wearing a spring t-shirt and spring underwear of course of course I like it of course but you don't have to take my word for it there are small companies small mom and pop shops that are using this stuff to great effect there's a small company in Los Altos in the States called Netflix who are using spring boot and spring cloud at scale to build services and they've talked about this several times in public there's a small business in next door China called Alibaba Alibaba is using spring boot and spring cloud at scale in their systems several different businesses several different businesses under Alibaba are using spring boot and spring cloud there's a Baidu also in neighboring China Baidu is a small search engine they're the number 3 search engine after Google and Bing and Bing of course powers Yahoo so Yahoo doesn't count well it doesn't I'm just saying so they're using spring boot and spring cloud in several places as well at huge scale we talk about web scale screw that China scale so these organizations are using this stuff very successfully if Amazon.com is the Alibaba of the west then I suppose Rakuten.com would be the Alibaba of the further east in Japan and they're also using spring boot and spring cloud we've talked about it Baidu and Rakuten are also using cloud foundry so is Yahoo Japan so these are organizations that have the money the brain trust, the resources and the motivation to solve these problems but they're still building upon spring boot and spring cloud for new stuff and wherever possible they're taking old stuff if they can and moving to that you see these organizations have to solve these problems they need to deliver better software faster and after all the name of the game, the goal here is to differentiate through better delivery of software and there's a lot of things that you need to do to get that done safely and effectively and I hope we've underscored that there are solutions to address some of those problems some of those things that you need to do if you have more questions I'm happy to answer them I'm online as I mentioned before there's also great guides at spring.io for such guides I don't need to dwell too much on that fact here at the spring user group and I just wanted to say thank you very much for having me today it was a lot of fun that was fun any questions, thoughts, feedback sure