Hello, I'm Steve Nunn, president and CEO of the Open Group. Welcome to Toolkit Tuesday, where we highlight the various components and leading experts of the Architects Toolkit, a collated portfolio of the most pertinent technology standards for enterprise architects. During the series, I'll be calling on a number of recognized experts who will bring their particular insights on how to most effectively use the various tools in the Architects Toolkit. We'll have a mix of interviews, panel sessions and pre-recorded presentations along the way. While all standards of the Open Group are designed so they can be adopted independently of one another, the greatest value for an organization can be derived when they're used in unison. The sum of the parts should be greater than the whole. In the Architects Toolkit, we have collated a portfolio of the most pertinent ones for architects together, all in one place. For most of these tools, certification from the Open Group is also available, so practitioners can demonstrate that they have the skills required, and recruiters can take the guesswork out of the recruitment process, all backed up by our Open Badges program. Minute EA, give me a minute and I'll think about it, just one minute. A minor or minute detail may seem unimportant, but precision and small details can often be easily overlooked. They can also make a massive difference, think marginal gains in all those sort of 1% adding up to make a big improvement. So in 60 seconds or thereabouts, I want to provide a smaller or minute thought that could lead to a big idea, maybe sow a seed, gives us something to think about at the back of our minds throughout the day. I'd love it if it gave us some inspiration or food for thought on a parallel topic, or helps us to see something from a different angle to solve a problem that we've previously been stuck on. My one minute thoughts are going to be mostly around architecture, architects and architecting, but hey, isn't everything? Thanks, I hope you enjoy. Welcome everyone, welcome to Toolkit Tuesday. That was Paul Holman from IBM. Many of you will have seen Paul at some of our earlier sessions, so we're continuing his season of EA minutes as we go through, so watch out for more of those. He's got some great stuff to offer us. Wherever you are in the world, thank you for joining us. We've got a lot of people signed up for today, and our main theme today is microservices architecture. We'll get to that in just a moment, but just a couple of things that those of you who are regulars will probably tire of me saying, so I apologise, but not everyone is. We're using the WebEx tool today, as you'll know, if you've got this far, and the way that we would ask you to do questions for our main speaker today is through the Q&A channel. That's the Q&A channel rather than the chat channel, so if you can't see the Q&A channel, if you go to the three dots that are in the bottom right hand corner of your screen, you'll be able to see the Q&A channel, click on that and that will bring it up. That's where you would ask your questions, please. We do have a limited time for questions, but we'd love to get some, and that's where to get them, please. Please use the chat channel, as I can see some have started already, as is our tradition. It's nice to see before I mention it. Use the chat channel to communicate with your other folks on the broadcast today. Typically, we let people know where we are, where we're joining from in the world. It's very much a global audience, and we love to see all the different countries and places in the world and anything else you want to share with your fellow attendees today. Without further ado, we're going to move to our main topic today, microservices architecture. To talk to us today, we have one of the major contributors to our recently published series guide on microservices architecture, and that is Mr Peter Maloney. Peter is a senior engineering fellow at Raytheon company, and his long career has taken him from developing and applying solid-state technologies to radar system applications to his current role as a system architect. He became interested in enterprise architecture and particularly SLA as a result of the ever-expanding need for providing access to increasingly complex data products to a diverse group of end users, with the resulting needs for collaboration throughout management and security. He's a Raytheon certified architect, which is a program accredited by the Open Group, and a three-time winner of the Raytheon Excellence in Technology Award. He holds one patent and has authored more than a dozen papers. He's also currently the co-chair of the microservices project within the Open Group SOA working group. So a warm welcome from the Open Group on Toolkit Tuesday to Peter Maloney. Are you there, Peter? Yes, I am, Steve. Thank you very much. Can you hear me? Yes, I can hear you clearly. Thank you. So over to you and then we'll come back for some questions. All right, very good. So as Steve already indicated, I'm here today to talk about the Togaff series guide for applying Togaff to microservices architecture. This guide took about a year and a half to develop. It actually was just published a couple of weeks ago, and I've got a reference to the guide number at the end. So again, thank you all very much, and glad to have you here today. So the intent of the guide is basically, of course, we're trying to contribute to the Open Group mission of bond-related information flow, and particularly we're trying to provide some guidance on how the architect can use the Togaff standard for developing, managing, and governing microservices architecture or any architecture or MSA as part of the scope. And that's actually an important statement, and I'll come back to that in a couple of slides. I've actually been involved in the microservices project since about the summer of 2014, I think. It's gone by pretty quick. While many of the characteristics of an MSA are very similar to SOA, in fact, our official position in the services project is that an MSA is a subset of SOA. This particular guide is dedicated for MSA only, and so I'm going to really focus on the differences in characteristics between the two. So the agenda today is, for us, I'll just briefly define what a microservices architecture is. Again, it's a distinct item from a SOA. I talk a bit generally about the use of the Togaff standard and how it applies to MSA, and then move on to enterprise architecture in a distributed world, and that is the world in which we find ourselves living today. Increasingly, cloud-native or cloud-centric, increasing numbers of users and also developers in a global environment. So that means we have to really look at the implications on the architecture and the architecture development process in that very distributed environment. So I'll go through some of the impacts that doing an MSA will have on that, first on the role of the stakeholders in that development, and look at the individual types of architectures within the microservices, business, information systems, and technology architectures, and finally, I'll turn to the question of governance. Governance is very, very important, of course, in any sort of application and architectural development. It's going to be even more important in a microservices environment, again, because of the distributed nature of everything, and then I'll sum up and we'll have some time for a question and answer. So first, what is the microservices architecture? So in the microservices project, we've defined it as an architectural style, defines and creates systems through each or small self-contained independently deployed services that align very closely with business capabilities. So we've come up with three key features we think distinguish an MSA from another type of SOA. The first and really the most important of these is service independence. Each microservice has to be independent of all other microservices and all other microservice instances. So each type of microservice is developed, deployed, and it can be changed independently. Going along with that need for independence is the concept of single responsibility. So each microservice is mapped to a single atomic business function or activity for which it is responsible and it's not responsible for anything else. And last that goes along with that is self-containment. So microservice has to be self-contained. It has to be able to access whatever data that it needs to carry out the particular function or any other resources, knowledge of business rules, etc. And the reason why this is important is or these characteristics are important is that what that does is allows you to put together is services that are because they are independent, they are also gives you resiliency and scalability that you really can't achieve any other way. The important thing is because those instances are independent, so I can scale them up and down as needed by my business. In the event of failure, I can simply spin up another instance to take over for the one that's failed with minimal interruption or other interactions. And again, this is what really gives me a resilient architecture capable of staying up regardless of what's actually happening to any particular individual instance. It also means I have technology independence because I can develop those services independently as the development team best sees fit. I can deploy them very flexibly, the same reason, and they are also reusable once I've defined and developed the microservice since it's independent and self-contained. I can go use that in another context if that's the easiest way of meeting my business needs. Okay, so Togaf and the MSA, so obviously the Togaf architecture development method is still going to apply to MSA. The details will be different in a few areas, and I will go through some of those on the slides that follow. The most important thing is right at the beginning in this preliminary phase, you're going to make the decision to use the principles of microservices architecture. And that's going to be the independence, the single responsibility, and self-containment.  Even if you're enterprise, it's already service-oriented. Again, the independence, particularly in single responsibility, a thing that you have to take into account as part of your architecture and make that commitment to. It also has some organizational impacts, which we'll discuss a little bit later. The other thing that goes along with MSA is typically these cycles will be shorter than would be the case in a more centralized or monolithic type of architecture. We are trying to in a microservice project remain agnostic to development methods, so we're not prescribing the use of DevOps or agile development or any of that, although those both align very well with microservice architectures. And of course, one of the things that comes out of that is the planning and development intervals can come very fast. You're talking about in agile, you have sprints that are a couple of weeks to a week or a couple of weeks and maybe a month at most long compared to a much longer development cycle for waterfall type development. So we have to bear that in mind as well as we go through this. So the role of stakeholders in MSA.  MSA brings you a lot of wonderful things. Principles of independent and self-containment mean I have scalability, I have resilience, I'm able to recover quickly in the event of failures, but that also limits your ability by capabilities at higher levels in the enterprise. For example, choreography of microservices. You cannot use microservices themselves to do any sort of choreography. If you do that, you're going to introduce dependence upon one microservice upon another one. As soon as I have dependence, I'm violating that principle and I'm running the risk of cascading failures in the event of a single instance or single microservice fails. So if you need to do choreography, you're going to have to do it at the application level or some higher layer in your enterprise architecture. What MSA does give you gives you a very highly reliable and scalable set of capabilities. There are a subset of that overall enterprise architecture and coordination and alignment with those stakeholders across the enterprise is going to be even more important than you thought it already was. Okay, so looking at the role of stakeholders again in MSA, so the decision to use an MSA to meet your business requirements, that's an enterprise level decision by the enterprise architects. The actual implementation of that MSA and the technology choices that go along with that is the responsibility of the project architect and they have to keep that all in context. So a typical set of stakeholders is shown here in the stable. So the project architect is responsible for the implementation of that particular capability but they always have to obey the guardrails that were laid down by the overall enterprise architects and reflected in the architecture review board. So if you need to violate one of those defined architecture principles, you need to go to the architecture review board and explicitly ask for a waiver for this particular case. So you can't allow anything, any deviation from that if it's not given a waiver by the ARB because if you do that, you're going to simply open the doors to technological chaos and incompatibility. So those architecture principles that are going to laid out, that's going to be part of the documentation that you're going to produce coming out of preliminary phase. And again, the architect has a free hand within their domain but they have to remain within the guardrails that are laid down by the enterprise architects. Okay, so as I said, so in MSA, each microservice is responsible for implementing a single atomic business function. In order to get to that point, you have to take your business processes and your capabilities and go through a process of decomposition that arrives those atomic functions. Getting to the point where they cannot be, whether you're irreducible and you can't break those down into one or more additional functions that would be carried out by additional services. The requirements that go along with those functions are more likely to be distributed in an MSA implementation as well. And typically you have a federated organization as well as a federated architecture. I forget whose principle it is and said, you know, a system architecture tends to reflect the organization of the enterprise that's creating it. But it's going to be, the reverse is true as well. If you really want to have an organization that reflects the system architecture in this case and federated is going to work best in this kind of environment. So the information data system architecture, architecture is really is broken down into two other architects, sub architecture is the data architecture. The application architecture are not really going to dwell on the application architecture here. That's not terribly different from a SOA, for example. But the data architecture is constrained by the data necessary for that particular business function that's being supported by the MSA. It's also possible you're providing data to other functions or you're being supplied with data originates externally, whether on an occasional level or constantly. But in the MSA, most of that data is going to be distributed. So the data distribution models have to reflect that. And again, you're going to be relying on resilience and scalability. So your data needs to do that as well. Have those characteristics, as well as your microservices, or you're going to lose the benefits of doing an MSA. And again, this is from a purely architectural standpoint. You can always choose to violate one of these principles. If it's necessary for your particular application, you just really need to keep in mind deliberately that you're doing that. But for example, if you've got a third party database vendor who's guaranteeing you seven nines uptime, and you're going to rely on that as a central point of failure, that is a decision you can choose to make. But you have to be aware of the implications that go along with that. So technology architecture. So of course, the decision to use microservices provides some or all of your capabilities is a technology decision. And the scope of that, the functionality, the use of your data, the requirements, the architectural principles, the patterns you're going to implement, and whatever standards you're going to enforce upon yourself, as well as the deployment mechanisms and what you're running in for an execution environment. That's all going to define what your technology architecture ends up being. And that's all covered in TOGAF phase D in pretty good detail. Again, the independence of these services and the idea of a federated architecture. So you can have independent development of your microservices. So your development teams and your project teams can make decisions about what programming language they want to use, what libraries, tools, et cetera, they're going to use, how they're going to deploy. And you can leave those decisions to the development teams and the project leader, project architect, as long as they remain subject to the overall constraints of the technology and enterprise architecture is defined by the enterprise architects. And finally, governance, right? So an MSA is an inherently distributed architecture. So the government's requirements really have to reflect that, even compared to a regular service oriented architecture, which isn't necessarily distributed even though it's using services. So really a federated governance structure is likely more successful than a fully centralized one because you really want to allow those teams to develop their own rules for governance and their own processes without having to reach back to a central authority every time. And the way to do that without having chaos resulting, of course, is that you really need two parallel tracks for this. So each team has a self governance path that they are applying only to their particular service within that MSA. But there also still has to be an enterprise architecture governance tracks which supplies the guardrails that are established for that enterprise. And again, if that team has a really, really good reason for violating one of those principles, they need to ask for a waiver and be granted that explicitly. Because you don't do that, you're really running the risk of now you've set up an organization that's institutionalized the process of technological chaos and instability because they are not required to adhere to an overall standard and you run the risk of incompatible tools, incompatible communication mechanisms, incompatible deployment, and things will get ugly in a real hurry. So finally, so just I've tried to go briefly through here some of the highlights of the key differences from the use of the regular Togap standard for an MSA as opposed to a SOA. And this is really about helping the architects achieve their final objectives, developing MSA solutions that effectively can divide ads or responses in being able to meet the needs of their specific business capabilities. And just a brief advertising here. So the guide which was just published, the series guide, is G211. It's available for publication in the Open Group Library. We have a couple of additional publications we put out. Our foundational publication I think was back in 2016 on microservices architecture goes through the derivation of how we came up with these three key principles and other principles that go along with that. That's W169. We published a paper on the benefits of DevOps methodology for microservices solutions. Again, not trying to prescribe it here, but it is undoubtedly true that distributed architecture and the DevOps techniques go very well together as well as agile development. And we published a, which is actually also another guide on microservices architecture for the Internet of Things that's also available from the Open Group Library. So thank you very much for coming today and I'll be happy to try and answer any questions in the time we have remaining. Peter, wonderful job. Thank you very much. A big subject and a lot to cover in 15 minutes. So thank you for giving us that overview and the pointers to the guides themselves, the documents. So very easy to find on the Open Group website at the Open Group Library. So several questions coming in and not much time. So I'll go straight to one, Peter. You talked about governance a bit, obviously towards the end, but at the beginning about choreography. And the first question is, at some point, some choreography from a higher level cannot be avoided. How do you cope with that as part of an MSA team? Yes, and that's an excellent question. And you're right. Again, why I said that an MSA can only provide part of the solution to your enterprise architecture. At the application level, you are undoubtedly going to have to do choreography. Typically, doing it at the application level is the easiest because, again, you have to keep those individual microservices independent from one another to allow for individual instances to fail, scale back up, scale down in response to what your demand is. So typically, you handle that at the application layer. And again, this is part of defining a communication with microservice through an API. It's the simplest thing to do and rigidly adhering to that. Can you say a little bit more about what you mean about distributed business requirements? What are they? So really, what that amounts to is, when you look at it, a federated organization, all I'm really trying to say is that each of those business organizations that is developing their part of an MSA or their MSA needs to have the responsibility and authority to create what their own requirements are. And, again, it's about federating that responsibility and not forcing them to constantly reach back to a central authority in order to get permission for requirements development. Again, not prescribing the methodology, but in order to respond quickly, which includes your requirements development for your microservices, you have to push the responsibility for those requirements down to as low as level possible so the developers are able to work at a local level with a requirements team as well. If this all really comes down to system and software engineers ought to be part of the same team. I don't know why we feel they need to discover that over and over again, and then ignore it next time around. But that is an absolute tourism in this business which we suggest. Yeah, no, absolutely. Absolutely. That is an often learned lesson. Again, on governance question, how much freedom should be given to teams to select anything in their technology? In other words, how much governance is ideal? Yeah, that's a good question. I'm not sure I have a really good answer to it. Part of that is the degree of compatibility that you can maintain with different technologies. Obviously, if at an enterprise level, you've selected a particular distributed database, for example, whether that's CacheDB or any one of a dozen other rival candidates, that's going to have to be everybody's choice. You can't allow that sort of flexibility. At some point, there are some things that are going to be necessary for each one to adhere to. Other than that, there's not a simple answer to that question, I'm afraid. No, sure. There are obviously lots of variables in there, lots of variables. We do have a ton of questions coming in, and I'm afraid folks, we're not going to get to all of them, but if you could pick one thing, Peter, what would you think is the biggest advantage of implementing an MSA in a system? Yeah, I'd say the single best advantage of using an MSA, again, is because of servicing independence, is the resiliency that it gives you, particularly for mobile applications where connectivity can be up and down, you have the risk of connections hanging up, instances failing, you really have a hard time beating an MSA in terms of the resiliency of that architecture. So when instances fail, you simply spin up another one, and you can continue almost seamlessly without the user, who's the person we're really trying to hear, ever being aware that something went wrong in the interim. I think that's the most important advantage that MSA gives you. Okay, and one last one, and apologies to those whose questions I don't get to, but domain-driven design, do you see that as bridging the governance function between MSA and enterprise architecture? I think it can. I think domain-driven design is a valuable technique, and again, part of the ability to federate these decisions includes breaking it up into the key domains, and particularly since requirements will tend to align along domain boundaries, I think that can be a very valuable technique in allowing this sort of federated, self-governed activity to take place. Right, I appreciate that. We'll leave it there, Peter. Clearly, this is a topic where there's a lot of interest and a lot of questions, and we may need to do another one of these on the topic and get some of these answers. But in the meantime, Peter Maloney from Raytheon, I appreciate your contribution and your guidance on MSA. Thank you for joining us. Well, thank you very much for having me, Steve. I really appreciate the opportunity to spread the word on this, and it's been a lot of fun and very educational, so thank you again. Yeah, it's great to hear. It was fun along the way, too. Okay, folks, that's the end of Peter's presentation. We're going to have another short video from another one of our resident experts on here on Toolkit Tuesday, Terry Blevins. Terry's Toolkit Tuesday tip up next. My name is Terry Blevins with the Toolkit Tuesday tip. I have given many messages about ensuring that you as an architect focus on real-world customer challenges, and of course, I stick by that. But having said that, you as an architect must also deal with issues that are not exposed directly by listening to high-level customer challenges. These issues are often in areas that customers overlook because they don't directly relate to their business functions. You as an architect not only have to ensure these issues are dealt with, but you may even have to convince the customer of their importance. These issues are often referred to as non-functional requirements. A few non-functional requirements are present in the open group vision of boundless information flow achieved through global interoperability in a secure, reliable, and timely manner. There are, of course, many more, and your job as an architect is to make sure these requirements are dealt with as appropriate for a given scope. Another important point is that these non-functional requirements are often at odds with each other. For example, security is often at odds with performance and usability. It seems like I might be opening a can of worms, but you're asking, what's the practical tip here? Well, here, there. First, don't ignore. Use your experience to identify those non-functional requirements likely to pertain to a given scenario, and do not ignore the temporal nature of those requirements. For example, interoperability issues come up over time as new things come into the enterprise. Second, do explore. Use trade-off analysis to identify how these non-functional requirements interact or interrelate, and use risk analysis to identify which have priority. Third, model to demonstrate how the non-functional requirements directly impact the delivery of functional requirements. Make it tangible for the customer so you can get their buy-in on the importance of dealing with them. So you wanted to be an enterprise architect? Well, these are the kinds of issues that you have to deal with. For more information, please check out the Togaf Library. Keep architecting for enterprise value. Thanks for watching. Thank you, Terry. Thank you. A consistent message of keep architecting for enterprise value, an important one. That's it for today, folks. A quick word about the next one I could see in the chat. There was a question, when's the next talk? It's Tuesday. It's nice that you're interested. The next one is on February 22nd, so two weeks today. We generally run them every two weeks. And the topic there will be modular open systems architecture, which is another style of architecture. It's being greatly picked up and endorsed, particularly in government circles in some of our activities there in the open group, but more broadly applicable too. And to present on it, we have a real expert there, Dr. Steve Davidson of MITRE Corporation. So do join us in two weeks' time for another toolkit Tuesday. Meantime, take care wherever you are, and we look forward to seeing you next time. Thanks for your participation and giving up your time today. Bye for now.


Oh, I'm sorry. Oh, no, no, no, no, no, no. So we're going to switch hats to S-254. Oh, boy, she said. Yeah. Well, it's fun. I'll be in back at the same room. It is. It is. It is. So what I'm going to do is. Oh my God. Take the. Easy way out. And. What was that? You should have in your file. The EP draft. Yes. They did 310. Nice. Okay. This one's an 830. So there's 3.1, which I've just submitted this morning. But there's also. I'm sorry. Yep. All right. I'm going to have to work. Then switch this up. I'd like to explain what I'm trying to do. Become clear to me. That with the amount of time that we have. And give the efforts. The last few years to improve fair and partial. Policies. Use of force and guidelines. Proof police officer. Training. I still support the idea of the elimination. But. It's clear to me. That. To try to do that this year. Given the. Length of time. Committee. Even though we've had extensive discussion. Some say that we're, you know, what I've heard is the way we do it now has. Worked well. Settled by the. Various departments in this state. We don't have all good information about those. But we do know that. There have been a lot of. It's a controversial topic. I believe we need to get it right. And as a citizen. Legislation. We deal with a lot of different issues. Just like the one just previously. I think we need to get on the back and forth with the lawyers from each side by commissioning an independent report. On the state of Vermont. And I was working now. I don't think we have to have a study. But what I really am lacking is when I hear from one side. Oh, no, they do, they do have it. It's like my argument with my town manager at Bennington. Well, they already have access to suit. Look at all of a sudden. Well, yeah, but what, you know, what about the other case. And so I hear from the other side. So I've been working with that. Try to craft something. The other thing I heard was there is some doubt about Zulu. And whether it covers ministry. Municipal police and covers. Sheriff's. We know it covers state troopers. That was clear. So it just attempts to do a draft, three point one attempts to do. Is. Say it's our intent. Zulu covers all. And it also contains. Not a study, but a report.  The reports that he and I have clicked out. And it also contains. Not a study, but a report.  The reports that he and I have clicked out. That's a report on the access to justice. And. And we're hopeful that he would get assistance from. Various groups that have been here. On both sides. As well as. It looks like. Project that might be rightfully. Center for justice. So this, I think. I mean, I realized that. The proponents of the bill. Will not be happy. I realize that the opponents may not be completely happy that the. Issue is still out there. But for me, anyway. We need to understand is, does Ramon have a problem? You know, we looked at no, not one. No. Not. And this committee determined that it was not necessary to change that law based upon the testimony. It was clear from both sides. This one hasn't been that way. And I'm like to get some clarity. And if we don't have a problem. Then we should. Then we don't need to change it. But if we do have a problem, as I suspect we do. Then I'd like a little bit of a better basis. I really want to thank that. Then over back. With me on this issue. And done some tremendous work. I don't know where we'd be without them. You know, listening to both sides. But this is what we've come up with. So I'll let Ben. Read it over them. We'll take a break at 10. Come back and talk a little bit about it for about a half an hour. See if there's something that. Why are you bringing it up? Can I make a suggestion? On lines. 14 and 15. Or it's against any Vermont state. County. Law enforcement officer. I think it should be all certified law enforcement officers. Because they're like UBM has a. It's on agency. And so this wouldn't cover. It should be all certified law enforcement officers in the state. Regardless of who they are.  Sure. Oh, yes. Absolutely. You should see what they have in their trunks. I walked by one of the UBM police cars. The trunk open. It was like an arsenal. A lot of riot. Yeah. Anyway, just so that it makes sure that, and there might be one or two other. Kind of quasi. Well, yeah. And any certified law enforcement officer. Just a suggestion. Good idea. No, no, they can't. So this is a, so Ben overgrossed me for the opposite legislative council. We're looking at draft 3.1 of a strike on that. It has two 54. This is almost completely new compared to the last version. It still creates under section one, a new chapter 190 under title 12. This is now a term though violations of article 11 of the Vermont constitution by law enforcement. It creates a new section 5607 entitled standard to recover that damages. And some section eight is essentially intended to codify it as a low standard. And as Senator White said, to make sure it applies to all Vermont law certified law enforcement officers for violations of article 11 of the constitution of the state of Vermont. And that is the search and seizure provisions. So this would cover roadside stops. Anything sees during those stops trying to, you know, arrest of individuals. So any potential violations that would come out of those types of actions. And again, this is just to make it clear that this is the standard. And it was a thousand Zulu and to make it uniform across the board for all law enforcement officers for article 11 violations, which again is what the Zulu, the holding in Zulu concern was an article and violation. It does say that the common law under subsection B, that the common law defense qualified and you shall not be a bar to any action seeking damages brought against a law enforcement officer for violation of article 11. And that's because under Zulu, it's a burden that the plaintiff must prove to obtain damages and a limiting analysis that the court undertakes. So it's just to make sure that it's being concerted and that qualified against not bar to that new burden that the plaintiff has to prove under Zulu. Can I just stop that line? Correct me if I'm wrong. The memory says that line is taken exactly from the Zulu case. I don't, I can't recall the moment it's a direct quote. And you're talking about subsection B. Yeah. Okay. But my intent was to capture it as closely as possible. And really just. I just want to, this is a good place to say this. When the bill began. It was creating. Right. Expanding what Zulu had already put in place to cover all certified law enforcement officers. As opposed to just state police troopers. Correct. That is what I understand now is the intent of the bill. Correct. Brilliant. Thank you. It is. Anyway. And moving on to what would now be in session 5608. This is a record of the case. This position. This was the old section 56 and version 2.1. And this is the one where. All judgments and settlements that an agency would pay. Must be maintained. Which would include a copy of the underlying complaint. And it would be subject to public disclosure. Unless an exception under the public records act applied. But any record that would be disclosed would include the name of the law enforcement agency and any monetary amount paid under the judgment or settlement. And then there's language that would ensure that all person identified with information contained in judgment. Or settlement. Shall be reacted prior to. Disclosure. And it would be subject to public disclosure. Unless an exception under the public records act applied. But. Any record that would be disclosed would include the name of the law enforcement agency. And it would be subject to public disclosure. And this is an effort to make sure. To really track what's going on in Vermont. With law enforcement agencies and these claims. And also to ensure that. You know, people that are included in the. In the, in the cases. You know, their information is not necessarily disclosed. So. So. Does the officer's name. Is the officer's name disclosed. It would be all personality information. I would be so. The officer's name would be included within that. So it would not be disclosed. That's some type of us. The officer's name.  You read in the paper that there was a settlement reached. For example. More center family. And it tells us the amount. But most of. Many of these settlements. Privately. And we don't know they exist. And people are saying, well, yes. They. They are already getting. They already have access to justice. Well, how many of them are there? What's. This will just give. People the ability. Without. The name of the. Or. Or the. The name of the person. In the case. In the case. In the case. In the case. In the case. The name of the person. In the case of. The representative or the center family. That was made public like that. And. Even though everything is redacted. If your chief will. Nose there is a suit against the. Brow� police department. You know. But, but I think this is good. Yeah. For you to deal. And then. And then- What we're interested, what I'm interested in, and hopefully the committee is, what is the current status of how many of them are there? We got such conflicting information on both sides of this. Counties. We move on to section two. This is the report that Senator Sears discussed in the news. The report on access to civil justice remedies and law enforcement qualified community in Vermont. What this does is essentially, the office of legislative counsel by November 15th this year would produce a report to the Senate Committee on Judiciary, the House Committee on Judiciary and the Joint Legislative Justice Oversight Committee concerning the impact of the doctrine of qualified immunity on access to civil justice remedies for people wrongfully harmed by bad faith policing and violations of civil rights in the state of Vermont. In report or in particular, the report shall identify, one, the origins of the doctrine of qualified immunity if it's present in interpretation by the state courts of Vermont. Two, the existing constitutional statutory and common law causes of action for addressing the alleged misconduct of Vermont law enforcement under Vermont law. Three, existing immunities from suit concerning allegations of Vermont law enforcement misconduct under Vermont law. Existing defenses to liability concerning allegations of Vermont law enforcement was conduct under Vermont law. Existing statutory and common law limitations on damages and the applicability of the doctrine of qualified immunity to Vermont state-level law enforcement officers, county-level law enforcement officers and municipal law enforcement officers. Just on that last number six. So on page one, or I'm sorry, on page two, page one at the bottom, subsection B, we're removing qualified immunity in violation of article 11. So is number six in the report meant to talk about what remains for applicability? Essentially, yes, because I would just be for in the previous section for article 11 violations, but it can apply in other situations. And I think the intent behind this is to give the committee, all the committees that we've received in the report the fullest picture possible of what the state of the law is in Vermont, just to analyze and see if there's any action to be had going forward. Because it could be clarified to say the applicability of the doctrine exclusive of violations of article 11. I understand that's what you mean. And to your point, Senator White, from before I can change this just say all certified law enforcement officers. Yes, sir. And then subsection B with the Office of Legislative Council should have the administrative technical and legal assistance of the Attorney General's Office, the Defender General's Office, the Center of Justice Reform at ELS and any other stakeholders interested in assisting with the report. There's something to be said about compromise that everybody hates and everybody loves. Well, it's been one of the most difficult bills I was explaining to a reporter yesterday. Well, I think of the three that are some of the more difficult bills that I've dealt with, one was civil unions and the chair of this committee. And the Supreme Court has said we had to act. So we didn't have a choice really. The other was Act 60, which was the big decision. I can't remember a little girl from a town in Madison County. But anyway, that was another, the Supreme Court said you had to act. In this case, nobody said you had to act. And it's been difficult because of the very emotional and honest disagreements on both sides. I remain an arm and deep perspective, the folks from ACLU, the JDS, the folks from the White Park community have spoken out clearly in favor. I also have a great deal of respect for Commissioner Shirling and other members of the Department of Public Safety, Matt Bernanke and State Troopers. So I hope that this will at least allow Vermont to look at this in a different way, but also make sure that it's clear. I'm suspect that Supreme Court would, if there's a statement, but to make it clear that Zulu applies to all law enforcement. So I'm happy to hear from folks in a few minutes but are not to be briefed because we basically have a half an hour to deal with it. We'll get back here at 10.15.


Hello people of the internet, my name is Johnny and welcome back to yet another FNAF news video. This one's gonna be a bit quick, I'm walking on some other videos right now. But I just had to pop in, get a video out for you guys today because we just got our first look at the fourth Tales from the Peterplex book. And this might be one of the most interesting covers we've ever had for a FNAF book. It sure as hell has got me excited. And if you're excited please hit the like button, if you're new subscribe we talk about FNAF news all the gosh dang time. And subscribing to this very channel is the best way to stay up to date with what's going on in FNAF. So very quickly, what is Tales from the Peterplex? What are the other books in the series about? When are they coming out? Well, just like the Fazbear Fright series, the books in Tales from the Peterplex all contain three separate stories. And also a prologue story at the very end that connects all the stories together in some way. Though what's different about Tales from the Peterplex is that they somehow connect back to Security Breach or the Mega Peterplex. And that's why it's so interesting to theorize about what the stories of Tales from the Peterplex are going to be about. So far we have five confirmed books in the Tales from the Peterplex book series. The first one, Lally's Game, comes out in a few days actually on July 19th. Some secrets are better left hidden. A forbidden artifact from her fiancé's past beckons to Selena. Jessica leads a double life from her friends and co-workers in the children's wing of a hospital. Maya can't resist the temptation to explore an off-limits area of Freddy Fazbear's Mega Peterplex. But in the world of Five Nights at Freddy's, everything comes with a price to pay. Now this happens a lot with new books, especially in FNAF it seems like, but people have already gotten their hands on the first book in Tales from the Peterplex. And I keep saying this but hopefully I can get a video out before the book officially releases based on those early reviews of the book. And I will also try my best to make some summary videos of these books. The second book in the book series is called Haps. Have you ever wanted something badly even though you shouldn't? Steve's dream of a video game programming career in starting a family led him to take a job that seems too good to be true. Aiden and Jace Cantor's are scaring some young kids in the tube maze of Freddy Fazbear's Mega Peterplex. So hopefully now you realize that some stories directly connect to the Mega Peterplex, maybe even directly to the game lore we see in Security Breach. And all Billy wants is to become the perfect ideal of what he knows he really is, an animatronic. But in the world of Five Nights at Freddy's, be careful what you wish for. Tales from the Peterplex number three, Somnophobia features our lovely dear friend, the daycare attendant. What are you running from? Sam is afraid of just about anything and everything that's unhealthy ever since his father died. Something about the Springtrap costume in the role-playing area of Freddy Fazbear's Mega Peterplex chills Luca to the core. Okay, first stop, number one, why would the Peterplex have a role-play section in it? And number two, why does it feature a Springtrap costume who wants to be role-playing as William Afton? And Grady's fear of being trapped in small spaces makes working as a Peterplex technician extremely challenging. But in the world of Five Nights at Freddy's, our deepest fears have a way of chasing us. Now, I've talked greatly about those first three books in previous episodes of Johnny Blox here on the channel, so I'll leave them linked down below if you want to go see what my thoughts are, some possible theories about what the stories are going to be about in those first three books. But now, let's finally move on to the fourth book that just got revealed to us today. Ladies and gentlemen, tales from the Peterplex entry number four, Sub-Machanophobia. Now, unfortunately, we don't have a description for this book just yet. So we don't know what any of the stories are going to be about. We just have this cover to go off of. And oh my God, this might be my favorite cover we've ever had for not only a tales from the Peterplex book, not only Fazbear Frights, but probably even all the FNAF books. I freaking love this one. It looks terrifying. So if you couldn't tell, Sub-Machanophobia, which I hope I'm pronouncing, right? If I'm not, I look like a complete tool, is the fear of submerged machines underwater. I'm sure you've all seen images that show off this fear. The most popular one that comes to mind for me is one of the animatronics from Disney's 20,000 Leagues Under the Sea ride. It is terrifying, legitimately terrifying. So the fact that we're getting a FNAF story based off of this fear and this concept is crazy to me. And I'm all for it. And then the cover, you can see not only a submerged animatronic, one that looks, you know, an awful lot like that animatronic from that 20,000 Leagues Under the Sea ride. But you also have a mermaid who maybe is animatronic. I don't know. It's hard to tell. It looks like maybe they're bleeding from their eyes that could also be oil from the animatronic. Actually, I think they're missing an arm. So yeah, they have to be an animatronic. So you've got a destroyed animatronic like dragon sea serpent type character. You also have a damage like what appears to be a princess mermaid. And then you also have this terrifying scuba man coming down. I don't think that's going to be our protagonist. He looks absolutely too terrifying. So who knows, maybe this could be a drowned worker who worked at this amusement park that had this ride similar to 20,000 Leagues Under the Sea. And it has since been abandoned where all the animatronics and characters have been left to rod. They get decomposed. They're destroyed. Maybe our adventurer stumbles upon this amusement park. Who knows, maybe it's a special version of the mega pizza plex that has a, you know, aquatic theme to it. It's all abandoned. They explore. They find these terrifying animatronics that chase them. Maybe they find the scuba man who's trying to kill them too. Seems like we've got a lot of threats in this story, but I am so, so hyped for it. The book comes out in December, I believe December 27th. And I think I forgot to mention it, but the first book comes out in a few days on the 19th. The second one comes out September 20th. The third one comes out November 1st. I just said the release date for the fourth one. I said earlier on in the video that a fifth one has been confirmed and that is correct. It's coming out on March 7th of 2023. No title, no cover for that just yet. Who knows, maybe it's coming pretty soon. Hopefully the description for book four is coming pretty soon. Because as amazing as this concept is, you know, sub-mechophobia, as terrifying as these characters look on the cover, we don't know anything about the stories. But it really, really does have me so, so excited. I'm so glad we're finally getting, you know, a story based off this fear because it's a pretty popular one, at least in the FNAF community. And especially since there was a lot of people who wanted a aquatic themed DLC for security breach back when we thought, you know, it was called Chowda. And Ray McCaffery was tweeting out about going to all these aquariums and stuff like that. People really wanted a aquatic themed DLC. So hopefully this can serve that purpose. This can fill our terrifying aquatic, you know, sized hole in our hearts with security breach and the pizza plex. Because these look absolutely terrifying. I cannot wait. So tell me, are you excited for it? Sub-mechophobia, book number four entails from the pizza plex. What are you hoping the story is going to be about? What are your theories? How do you think this all connects to security breach? Like I said, I'd love to make a video going over some of the early reviews for Lalli's game pretty soon because the book comes out. I'm kind of running out of time to make that video. And also I do want to make some summary, you know, some proper summary videos about these books, you know, when they come out. Because I started doing that with Fazbear's Frights and then I completely stopped doing that. Who knows, maybe I'll come back to that. But the books are definitely something we need to tackle on this channel because we have not done that a whole lot and we need to get on that. But anyways, that's going to do it for this quick FNAF news video. Again, tell me your theories in the comments down below and I'll see you all on the flip side. Goodbye.


Good evening. Again, I'm Jorge Motiel. I'm an organizer with the Industrial Areas Foundation. I work with COPS Metro and the San Antonio Sponsoring Committee, a new organizing effort here in San Antonio. And it is my responsibility tonight to help us follow the money trail when it comes to what we call the business of family detention. Can you hear me back in the corner? Yeah. All right. Even louder. So I think most of us are familiar with this curve. It shows the trend of undocumented immigration to the United States. And as you can tell, it reached a peak around 0708 and has been tapered down or has gone down since then. Not surprisingly because of what's happened to the economy. People want to come here to get jobs and if there are no jobs, they don't want to come here. However, something totally different has happened to the costs of detaining immigrants in this country. And that is the curve has not changed. In fact, it has continued going up and it broke the $1 billion mark back in 2011 and it has continued rising ever since. Let me tell you a story about a quota. The U.S. Senate added language to the Department of Homeland Security Appropriations Act in 2010 that said the following, I's immigration shall maintain a level of not less than 33,400 detention beds that went up in 2013 to 34,000 beds. What does that mean? It became known as the immigrant detention quota or the bed mandate. We're telling eyes, we're giving them money and we're telling them you must have this many beds and in some instances congressmen are telling them you must fill them. And what we know is that there is no other law enforcement agency in the country that has a quota from Congress except for ICE. How did that come about? Well, there was a lot of lobbying to make sure this happened. One of those responsible is CCA, Corrections Corporation of America. They spent between 2008 and 2014 $10.5 million lobbying immigrant detention and immigration reform issues. Not only that but of that amount CCA spent 61% of dollars lobbying the subcommittee that dealt with DHS appropriations. So if you're wondering how you get that language in there, this is how. The Geo Group, another private prison company, spent $460,000. They need some catching up to do in lobbying between 2011 and 2014 but they're just getting started. And as a result they got some contracts. What we know is that 62% of the ICE detention beds are operated by for-profit prison companies and that's up from 49% in 09. Geo and CCA together operate eight of the ten largest detention centers in this country and get this following the quota, the implementation of the quota. They grew their share of the total ICE detention system from 37% in 2010 to 45% in 2014. They were rewarded by the implementation of the court. You still with me? Now, how does this translate to dollars and cents? Well, there was a significant payoff between the implementation of the quota, the beginning of the implementation of the quota and now CCA's of the profits go from $133 million to $195 million. The Geo Group did a little better. They went from 42 million in 07 to 144 million. Let me save you the math. That's 244% increase in profit. Now, this is a little harder to see but it shows the stock price of these two companies over the last five years. If you'd invested with them five years ago, you would double your money. Not a bad investment. Now, it's interesting, though. Early in the summer, something happened. Maybe this little IWC group of rebels started messing with the detention quota. They're both going down a little bit. CCA lost, went from a high of 40 to a 31 and the Geo Group from 40 to 32. So maybe they're a little concerned about the talk about reducing family detention because that's lost business, right? Well, we'll see what happens but don't be too sad about their economic future because, hey, there's some alternatives to detention or is it an alternative form of detention. The monitors are brought to you by the BI, Incorporated, a subsidiary of the Geo Group. So if you want to reduce the tension, that's fine. We'll come up with a different form of business and we're going to sell making money. And this is a tagline in the website, a leader in offender monitoring. So we're referring to these women as offenders. I started reading when I started thinking about this issue, I started reading this book called Why Some Things Should Not Be For Sale. I think it's relevant to this. This question is relevant to what we're talking about and I want to leave you with a line from this book. Markets not only allocate resources among different uses and distribute income among different people, but particular markets also shape our politics and culture, even our identities. Efficiency, we're going to use that word a lot when we talk about private business. Efficiency, it's clearly not the only value relevant to assessing markets. We have to think about the effects of markets on social justice and on who we are, how we relate to each other and what kind of society we can have. Deborah Sats, S-A-T-Z. I heard this really terrible line one time when they said, you know, when you deal with drugs, the problem with dealing with drugs is that you sell the drug and then it's consumed and it's gone. When you deal with people, when you buy and sell people, which the cartels in Mexico have begun doing, you can sell them over and over and over. It's terrible to think about that, but it sounds like GEO and CCA have taken a page out of the bulk of those cartels. Thank you.


Okay, so just talk real loud. You want to start? Okay. So thanks for joining me today. My name is Michael McEwen and I work for Red Hat and I'm going to be talking about building Apache Spark cloud services in Python. So to start with, let's talk about what is a cloud service or at the very least what I mean by a cloud service and what I want to talk about today. And so I'm kind of relying a lot on definitions that have been put out by the cloud native computing foundation. And this is an organization that's like a governance body that is kind of helping to govern projects like Kubernetes and Prometheus and what they say is that a cloud native application should be something that's containerized. So, you know, we're talking about building for containers. We're talking about deploying to Kubernetes. So this is obviously a strong part of the story. They also say that a cloud native application or service should be dynamically orchestrated. And so what this means is that you can use the container with a container platform to, you know, migrate between instances. It really should be generically useful in those situations. And then they also say it should be micro service oriented. And this is a really nebulous topic. It's a little more philosophical or ideological. But the way I take it is I kind of look back to the Unix philosophy of old that a micro service is an application that kind of does one thing and does it well. And so really what you're talking about is building purpose built applications that you're going to put into a container and deploy to the cloud. And here's a link to their FAQ. They've got some great language that kind of talks about what these platforms mean and what these applications are. And so I highly recommend checking that out. Now, since we're talking about the cloud or we're talking about cloud native, the platform that I'm using is OpenShift Kubernetes. And how many people here are familiar with Kubernetes or OpenShift? Okay. Pretty good audience. So, you know, this is like the generic diagram. I'm sure many people have seen this. You know, you've got kind of your container network and everything. But what I'm most interested in is this part over here. Because I want to use this as a developer and I'm kind of interested in how does it connect to my source code repositories and how can I get kind of automated build features in there. So, although all this part is really interesting from an infrastructure point of view, what I'm kind of curious about is this part. And this is what I want to talk about today. So, uh-oh. Looks like we've got images out here. I'm going to switch browsers quickly because this is, apparently I have an issue here. Sorry about this. Okay. So, what I want to talk about is Apache Spark. And how many people here are familiar with Apache Spark? Okay. So, again, a good number of people. So, this diagram might look familiar to you. And this is basically a general outline of what an Apache Spark application might look like. You know, we have what we call the driver process, which is where all your user code lives. And then there are a number of executors that help perform the distributed work. And underneath all this is a cluster manager that controls how these pieces interact with each other. And so, I'll go through some of this a little quickly because it seems like a lot of you guys already know this, but kind of the fundamental abstraction at the base of Spark is something called the resilient distributed data set. What we refer to as the RDD. And this is really the primary or base level abstraction that happens for all the data. And what it is is a partition, lazy, and immutable homogeneous collection. And so, what does that kind of mean for the resiliency and the distributed nature of these things? Well, they're partitioned, meaning that when the data comes in, Spark makes partitions in your data set. This makes it easy to distribute that data. Second, they're lazy, meaning that any calculations that need to be performed on those data sets, they don't happen until they absolutely need to return a result. And finally, they're immutable, meaning that you can't change the data set. You can make a new one, but you can't modify the one that's being distributed. And these pieces together help to build this resiliency into the system. So, if one piece falls out, it's very easy to recalculate that partition and send it back to do work. And this really helps Spark to become kind of a stable platform to do these type of calculations. So, what does this look like in action? Let's say you have an array of numbers, and you want to do a calculation to say which one of these are even. How many of these numbers are even? So, the first thing I would do is tell Spark to parallelize this data set. And that means it's going to distribute these things, and it's going to create the data set for me. So, this is our RDD of each number in its own partition.  And this will give me a new data set. And in this case, I'm just looking for any number that has modulo 2 equal to 0, meaning it's even. I want those in my new data set. And so, I end up with 2 and 4. And I count it. Now I know how many even numbers are in my data set. So, this is a really simplified look at the operations I might do. But this is how these RDDs are built up, and how they might be distributed to do work. So, when we take OpenShift, and we take the model of Spark, this is what it might start to look like at the infrastructure layer. So, we have our nodes that are the physical nodes where the cooblets live. And inside those nodes, we have our container pods. And you can see that, you know, over here we have a Python application. And maybe it's got some Spark containers. You know, maybe these are the executors, and this is our master. And maybe we have more applications, a Mongo database. So, you can start to see how we can use the platform to distribute the different processes that are occurring for us. So, what is a Spark application? And this is kind of the way that I generally reduce these things. You have source data that's going to come in. You'll perform some sort of processing on it and return results. Now, source data and results can be very nebulous in many ways. You know, source data might be coming from a database. It might be coming from a file on a file system. It might be coming from a stream. It might even be coming from an API call where data is pushed in. Likewise, results could mean the same thing. So, some of these are abstractions that you'll have to deal with. And what we're looking at here is a very simple Python Spark application. And this is going to do kind of what we just looked at earlier. It's going to take in a data set and then it's going to parallelize that data. It's going to run our little you know, evens counting function on it and it will count it and return that. And so, if I run this on my desktop this is maybe what it would look like. I would have the spark submit command which is a tool that comes with Spark. And I tell it to take my application and in this case I'm going to say, for all the numbers zero to a thousand tell me how many of those are even, right? So again it's very simple process and it starts to run and the JVM goes and it spits out a bunch of logs. Now, as an application this is probably not that useful to me because what just happened, what do I have as output? How many numbers were even?  Okay, so how do I use this in a cloud situation? How do I go from using this locally to taking it to the cloud? So again, I start to think about how am I going to design my application in a way where it can become a microservice. So I go back to this pattern of ingesting data, processing data and then publishing it, right? Ingest and publish can be very nebulous and they'll depend on the systems that you're designing for. You may have a database that you're reading from to ingest data and publishing might mean sending a call to an API that another service exposes. Likewise, that could be inverted. You might have a service that calls your microservice on an API and publishes data to a database. These will change depending on what you're doing, but in general, you'll be operating on this type of model. So as you think about building these applications, you need to consider the structural needs of what you're building. Depending on what cloud you're using, it may be Kubernetes, it may be Mezos, it may be something else. You have to know how will I deploy my application? How will I command that application from the outside and then how will I control it? What tools are provided to me by the platform to do that? Likewise, where is my input and output data going to come from and go to? These will be dictated by the systems that you're working in. There's really not a good way to generically cover all these, except to say that you're going to have to consider these pieces. What I'd like to talk about is kind of three common architectures that I've come across and I have a feeling that many of you will come across these as well as you build applications. We'll talk about on-demand batch processing, continuous batch processing and then stream processing. First we'll talk about on-demand batch processing. Generally what this means is I have some event that occurs which picks off processing and then of course I take some data in that I want to process and I create results. This is something that happens every time it's triggered and that could be a cron job, it could be something that comes in on an API call, maybe a user visits a website and that produces the action, but this is one pattern you're going to see. When might you want to use this pattern? In general, these are kind of some of the top things that came up to me as I was thinking about this. When you have non-deterministic request windows, so think about a user who visits a website like Amazon. They're going to be clicking through products and just looking for something that maybe they'd like to buy and every time they click on a product there's going to be return some sort of rating, some number of stars maybe says I suggest this product for you or I don't suggest this product for you. This is the type of area where maybe an on-demand call is what you want to use. You may not have these things pre-calculated, you may have some model that contains this data but you'll need to filter that at the time that a user actually hits it and you won't necessarily know ahead of time that that user is going to do that. Likewise, when you have quick results to calculate, something that can be returned very quickly because if you have a situation where you don't know when these things are going to happen or you have a user visiting a website they're not going to want to wait a minute for some long processing cycle to happen before results come back. So this is another situation where you might think about using this technique. And then likewise, when you have a lot of situational dependencies, so again think of the example of a user visiting a website. If your processing depends on a user entering information before the request can be performed, this is a situational dependency. So this is something that you can't pre-evaluate or maybe is very difficult to pre-evaluate. This is another scenario where on-demand might be what you want. So what does this kind of look like? I'm using an example here that I'm calling the Hello World of Spark because it's been in the Spark code base. I looked just the other day at like Spark 0.1 Alpha. And this example was I think one of the only examples that existed in there. And it's a Monte Carlo method for estimating pi. So we're talking about throwing darts at a dart board, counting how many land inside a circle versus how many land outside, and that ratio will approximate pi for you. So this is just a function that does that. And you can see this part here. We're doing a similar type of operation that we did before. We're paralyzing a range of numbers. In this case this is the number of random points I'd like to use to calculate pi. Then we're mapping a function onto it and we're reducing that. And this is the technique for returning pi. So now I've got a function that every time I call it can give me that answer for pi. And what I might do is I might embed that into an HTTP server. So if you're familiar with the framework called Flask for Python, this is an HTTP framework. This is what it might look like if I embed this function into a return. So now I've got an HTTP service, a REST-based service that can basically take a request and give back pi whenever I need it. And this is just one way to look at it. You might use GRPC, you might use some other kind of remote procedure call mechanism, but this is just one way to look at it. So let's move on from that and talk about continuous batch processing. And I look at this as a type of scenario where you have data sitting in some sort of data source. Your processing is always occurring. There's no need to trigger it with an event because you want this process to always be running and always be producing results to wherever your store is. And when might you use this pattern? So if you have data that updates very frequently, let's say you have a database of users who are always giving ratings, for example. You want to be creating recommendations, let's say, off those user ratings and you want to be doing it continually because that information is always happening. So there's probably never a time that I don't want that to be running. And one way you might use that is when you're creating machine learning models for evaluation, right? So if we think about a recommendation engine where users from all your products like Amazon, for example, they're always rating what they like and what they don't like. Can you initially be creating models so that you can evaluate how those models are performing against what you have in production, right? So you want to say, as users add more data, are my models getting better? Maybe they're getting fresher? In the case of a recommendation system, you probably always want it to be fresh because users are continuing to add information. And then another way to look at this too, or a situation you might use it in, is what I'm calling life cycle management processes. So think about a lot of the work that gets done in distributed computing is data engineering. It's transforming one schema into another schema or taking one format and turning it into another format. This is something where you might just want it to be continually running. You say, I've got users who are putting in data nonstop. Maybe it's a text data. And I want to make sure that no one's putting in words that are, you know, on our band list or something, right? So you want this to always be running, always be searching for results that are coming out of that. Now what might this look like? So this is a piece of code from a recommendation engine that one of my colleagues who's actually sitting here helped to write Ruri in the back there. And this is part of a model generation service. What it does, as you can see at the top, there's this while true loop. It does some sort of database selection here to pull the ratings out of our database.  And what I really want to highlight is this part here, we see where we've taken the new ratings from our database. So since the last time I created a model, I want to see all the new ratings that have been added to the database. I create an RDD out of that. And then I'm going to do some processing here where I create a model from that. And so this is just always running. And if it sees changes to the source data, it creates a new model. And it puts that model into my database where I'm kind of storing these models for later usage. So the last kind of pattern I want to talk about is a stream processing pattern. And this is something that I find very intriguing. And I think it's a really cool way to work with data. But this is where you have a stream of information that's always occurring. So think about Kafka or AMQ or those type of message bus type applications. There's data that's always coming in. And my processing, this is a little different than the continuous batch. My processing is reacting to every time data comes in on that bus, you know, within some sort of window. And then again, those results are stored somewhere else. Now, you probably would want to use this in a situation where you have like real-time event processing. So think about an IoT situation where you have sensors on maybe public transportation, right? So you want to follow where all the buses in your public transportation system are going. Are any of them running late? Those type of things. So you have a continuous stream of information coming in and you always want to be updating on what's happening. So that may be one area. Another is if you're working with systems that are built on a broadcast messaging system. So I like to think about the Fedora message bus. How many Fedora users do we have in the audience here? Okay, a couple. Fedora, since it's a large community system, they have something called the Fedora message bus. And it's a federated message bus system where you have messages coming in from build systems, maybe for all the different packages that are being created. Messages coming in from mailing lists, you know, all sorts of information being aggregated. And this is a situation where, well, I would obviously want to build something to that message bus, because that's the architecture of the system I'm working on. And then likewise, another level to this is what people are calling Kappa style architectures. And this is a way to look at stream processing where you have a stream that's your input data, you have processing happening, and then your output is always going to be to another stream. So you're kind of creating these message bus scenarios where one topic might be your clean data, and then you have a process that runs and maybe changes the schema or pulls out some specific information and lays it to another stream. And this allows you to build up really complex hierarchies of applications that don't necessarily need to depend on each other. All they really need to depend on is the message bus. So Spark has a really cool API called the structured streaming API. And this is kind of what it looks like. There's multiple ways to do stream processing in Spark, but I think the structured streaming is really interesting. And what you do is you kind of build up this set of instructions. You tell Spark where will my information come from, and that's kind of up here. I'm telling it a Kafka broker. And then I tell it what to do. So I wanted to select, you know, in this case this is real simple. It's just taking every value that comes in on the stream and casting it to a string and then rebroadcasting that. So this is just really kind of looking at a simple example. And then what happens is all those strings that come in get grouped by the value that they created and counted. So what's happening here is this application is doing like a word count on a stream. It looks at every string that comes in. It groups them by similar and then counts them up, right? So you can say imagine you had a stream of words coming by. This is just going to count them. And then at the bottom here you can see I'm telling you where I wanted to put the output of that stream and then I just tell it to start. And in this case I'm just storing it in Spark's memory storage and I'm giving it a name so that I can query that. So at this point it's doing a bunch of work but how do I get the results out of this? So what I might have as a function like this that allows me to look at Spark's internal SQL in memory representation and then I can run a query on it. So in this case the query that I'm running is I just want to know the top 10 entries of all the things that's counted. So I want to know the top 10 most frequent things that are happening. But I could call this function on demand when I wanted to get information out of it. So that kind of brings me to the next way you could do this, right? And this is an example of what the Kappa style architecture would look like. You have a stream coming in, you have processing happening, and then you have a stream that it lays it out to. And if we look at this kind of similar function that we just looked at, the top part of it's all the same. It's all doing the same work. But if we look at the bottom part where it's writing the output of that stream, I've told it now just to put it onto another Kafka topic, right? And so what this means is that I don't have to worry about having a routine that pulls that information out using an SQL query. I could actually have another microservice that just listens on that second topic. And then it would be automatically giving me all those counts. And I could do whatever I needed to there. I could aggregate them or I could do something different at that point. Okay, so, you know, we've talked about some different patterns you might use. How do I take these from that desktop example and bring them into the cloud? You know, I want to take it from source code. I need to turn it into a container. And then I want to push it into my orchestration platform. And at the same time, I need a way for the user to still get in and out or myself to get in and out of it. So the group that I work with at Red Hat, we have a community project called RedAnalytics.io and we've created some tooling, a project that we call Oshinko. And this allows us to use some of the source to image type workflows that are in OpenShift to say, I'm going to take my source code from a Git repository. I'm going to use the source to image to build that into an image that can run. And then when it gets deployed a Spark cluster will go with it and be bound to my application's lifecycle. So in this way, I don't even have to manage Spark anymore. I can use the workflow that I'm used to in OpenShift going right from my code, making pushes to my code and that might go through a CI testing framework. And then when it's successful it gets deployed onto OpenShift and a Spark cluster will appear and be bound to it. So I'm going to assuming nothing else goes wrong, I'm going to try and demonstrate for this view real quickly here. I've got a small GitHub repository here and this is a tutorial that you can find on Rad Analytics. And this is a web microservice that's going to create that Spark Pi for us, right? So it's going to create an HTTP service that I can on-demand query to get a Spark calculation. And you can see my repository, in this case, my repository is pretty simple. I've got to read me, I've got my app file, which is not overly long, it's just a little flask application. And then I've got the requirements like any Python application might have. So what we're looking at here is OpenShift, this is my project and I've already taken the liberty of loading into this the Rad Analytics templates that I'm going to use. And so what I'll do is I'm going to select from my project the template that I'd like to use. So I'd like to launch Apache Spark Python. So I select the template that I want, kind of click through the description. You can see now it's asking me for a bunch of information here. So maybe I'm going to call my service Spark Pi. And now what it wants is the URL for my GitHub repository. So I'll just copy this from here. Now, there's a bunch of other options I could use here, maybe if I wanted to build from a branch or there was a subdirectory I was building from or, you know, these options help you control how the application gets deployed. But my application is written in a very simple manner, so I don't need to fill in most of these. And likewise at the bottom I could adjust how the Spark cluster gets deployed. I could change the options that go to it. So I'll click create. What we see now is OpenShift. This is being built on OpenShift. And so if I look at the logs for this, you can see this is kind of a standard Python build process and then it's pushing it to the internal registry. Now you can see at this point my pod is up and running, but what's happening is the Spark cluster is being deployed with my pod. So this is automatically being bound to my application and it's kind of giving it a random name. Now the last thing I need to do to get to this is just expose a route to it so that I can get to it. And now if I click on this, hopefully it'll work. Okay, so I hit the root endpoint and it tells me, you know, the Python, Flask, SparkPy server is running. I need to add this extra thing to get more information. So if I add this extra route, see now there's kind of this weight going on. This goes back to the quick result side of this, right? What's happening? All I can see is a little spinner going, but eventually it comes back and gives me a really bad estimate of pi, okay? So like don't go to the moon with this or anything, but it's fun to play with, right? And, you know, something I want to point out here that I talked about before, which is now that my application is linked to this GitHub repository, you can see I've got this thing called a build here. And I don't have my webhook setup, but if I did, what I could do is push a change directly to my Git repository, the webhook would hit OpenShift and this would actually rebuild automatically for me. Now in this case, you know, I could hit start build, like if I made a change to the repository, I could hit start build and it would run and then deploy it again and attach it to the Spark cluster again. So as a developer this is really nice because I can really easily test my changes out and even in a private project I can do this. So let me switch back here. Okay. So you saw when I made that web request do the work, it took a little while to come back, right? And this is where one of the problems you're going to run into when designing these type of services is kind of the synchronicity issue, right? So I make a request for pi and now that service is off doing something, right? And if I tell it to use too large a data set, this could take minutes to come back, right? And you don't want that result coming back later and the user just walking away from the terminal or something. So to mitigate this in our designs what we like to do is start to separate the API concerns from the actual processing concerns. And this might be a very common way to look at this would be our main process, the API, whenever I make a request for a new pi estimate, maybe instead of giving me back the pi estimate, maybe what it'll do is right away it'll give me back an ID. And then I can use that ID to query the results of what's happening. Or perhaps the application on the other end of this uses a web socket, right? So I make the request and now the main processing loop can push the information back to me when it's ready. And in the meantime my application can display some message saying this is, you know, work is happening or whatever. So, you know, depending on what this API is, you'll have kind of different ways to mitigate this. But in general what you want to start doing with microservices is pulling apart these concerns to make it easy to deal with the other ends of them and to address issues like this synchronicity issue. So what might this look like in Python? Well this is like the main process, right?  This is pretty compact and I don't want to go into every line of it, but what I want you to take away from this is the top line, which says import multi-processing. The Python multi-processing package is really powerful. And I would say that if you're going to start doing these type of things, read the docs on that package because the primitives in there are very easy to use. And so if this is the main process side, this is maybe what our processing loop looks like, you know, and so we've got this big thing here where there's, you know, and this is coming from a service that actually responds to incoming requests for recommendations basically, right? So you have a user who wants to get a recommendation, just to kind of do it for you. And you know, it's a big piece of code, whatever, I've taken out pieces of it, but the main thing to look at here are these areas that are probably really difficult to see because they're in red and there's a lot of light washing it out, but you can see these primitives, this response, request, response, response, these are the cues that I use to communicate back with the main process. And so whatever I'm doing here, I'm using these primitives and again, this is coming from the multi-processing library and I really recommend checking it out. Another big thing that you're going to run into, especially if you're doing Python programming for Spark, is dependency management. So right now, Spark has some really good features for JVM languages that need to distribute dependencies. There's to that Spark submit command, there's an option called packages and you can give it, you know, Maven targets and it will pull all those packages in and send them out to the entire cluster so your applications can use them. With Python, the tooling is a little behind the times. So let's say we've got this application and, you know, the filter, this is maybe like, let's say we're building another service to tell us how many evens exist in the dataset, you know, our classic hard problem here. And you can see at the bottom here is where I'm actually doing the work, you know, I'm telling it to kind of make the paralyzed data, filter it and count it and this filter evens function is doing some sort of database action, right? So what this means is that filter evens function is going to be distributed to the Spark cluster. And so what that means is each executor is going to need Pi Mongo in place because what's going to happen is something like this. My main application wants to talk to Mongo, but now I've distributed code that also talks to Mongo. And so that Mongo library needs to be on every one of the executor nodes. And this is a situation where you might have to manage these dependencies yourself until the Spark community kind of catches up with this and gives us better tooling for doing it. So, just let's recap a little bit here. You know, we talked about this design pattern that I really like to use in just publish process. This is kind of the general pattern I like to get into. We talked about some different types of architecture patterns you might get into when you're designing your applications. We talked about, you know, the on-demand batch, the continuous batch and the stream processing. And then we also talked about the Oshinko source damage project. So this QR code up here is a link to this slide deck. Please download it. It's open source. It's just a reveal.js project. You're welcome to use what's there. Here's my email and my blog. And please check out redanalytics.io. We've got a bunch of tutorials there. And lots of this material you've seen here is on that site. So at this point I'll take any questions and thank you for your time. So you have any questions? Barma, don't hit me up too hard here, okay? Yeah, I'm talking about the first pattern the client would call a REST service and the response comes back immediately. So that's the code you're describing. So what happens, the client continuously pulls and the second, the function that you showed us, it is the multiprocessing package. Is the client continuously needs to connect to be the same invocation of that request for that process to happen? Or can there be another status function independent of the first one that goes back and gets the data directly from the spot? What is that? So the first example I showed is a very simple example, where you're just hitting it and it's trying to make the request. And if you double up on requests at that point, you're going to break that because that application is really simple. But once we start to separate the API the first way that I think about it for a REST application is I say I'm going to make a request to the REST server to start doing work. And the REST server gives me back an ID and that ID number is the ID of the work that's being done right? So my processing loop there, the second process, it may have a queue of work requests that have come in and it knows about the IDs and so when it does each bit of work, it can take that ID, it can update the status for what's being done and the main process will always be able to see what the statuses are with those IDs. So my client could keep requesting, I make the first request to do work and I get an ID back. And now I just query that ID and the main process can tell me no it's not ready yet, no it's not ready yet, okay now it's ready, here's the results. And so this is really what I mean about separating the concerns. Your architecture will dictate how do I want that processing loop to kind of queue up information and maybe I have several processing loops, maybe that work gets distributed in a better way. But really what I want to do is hide those concerns from what the user is seeing, right? So that they can do that, they can keep querying to say what's happening and they're not going to overload the spark work that's being done. Does that kind of make sense? Thank you for solving the same problem. Cool. Any other questions? Can you tell us more about the Oshinko source to image builder process? Sure. So what you saw with the demonstration I did was me exercising the Oshinko source to image tooling, right? So what the tooling does is it first pulls a source code from your Git repository and then it creates a container inside of an OpenShift that will do the build process, right? And so that's that log that I showed. It was pulling the code and it was starting to build it, right? Once it builds that code and it's successful, it deploys the built container to OpenShift and then the Oshinko tooling is actually inside your built container and what it will do is it says did you request to use a spark cluster that already exists because you could do that, right? That's one of the options I went by. If you didn't request that, then what it will do is it will spawn a spark cluster for you and you could specify the configuration for what you wanted to do and then your application will run and when your application exits the Oshinko tooling will kind of catch the exit on that and it will delete the cluster that goes with it, right? And then your application may go away or something. So that's kind of a general look at the steps that happen to do that deployment. And it associates, you know, like a service with it and exposes ports. The same behavior you would expect from the other source to image tooling, whether you were using, you know, JavaScript or Java or Python or something like that. Okay. So you described Oshinko a whole bunch without mentioning like CICD or continuous integration or deployment or anything like that. Can you speak up a little bit? You described Oshinko without mentioning CICD or sort of, but the way you describe it, and I don't know that much about it, it's a very similar workflow. If, like, and we do something similar in a Jenkins stack that uses Helm charts to deploy stuff to Kubernetes clusters, what are, like, why didn't you work that into a more classic CICD platform and how is it better? So the reason I didn't work it in is because I'm not really diving into the pipelines features that exist with an OpenShift, but that would, what you've seen here is kind of the lowest level of application development, the next level of application development that would happen if I were going to take this into a more production situation. First, I would put testing between, you know, when my application gets checked in to Git, you know, there could be a test running there, and if it gets rejected, then, you know, the commit doesn't get merged and it doesn't kick off a new build. Another way to look at this is I could use the pipeline functionality that exists with an OpenShift to create a pipeline that says first check the code out and build it here, then run the tests here in OpenShift, and then if all that works, then deploy it, and I could say deploy it to another project or something. So part of the reason I didn't get into talking about it today is because those primitives start to exist at a higher level, but once you've created your applications in this manner, you know, they become very easy to kind of mix and match, and I think if you're using Jenkins with Helm and Kubernetes, you're doing something very similar. What OpenShift kind of adds to that is this, you know, kind of UX around building the pipeline for you, right? So if you're not a Jenkins expert, or even if you're not familiar with Jenkins, the pipeline tooling allows you to specify those things using a language that's a little bit easier than diving into, you know, a Jenkins configuration or a Travis configuration or something like that. So really that would depend on what you're doing with your application design, but it starts to exist at a higher level than just creating the applications. And so the Oshinko tooling helps because when we're automating these things, we don't need to automate the Spark cluster creation. We can just let the tooling take care of it for us. So when it runs the tests, it actually spawns a cluster dynamically, runs the full integration tests, and then can report on whether it's succeeded and then, you know, deletes the Spark cluster after it's all done without having to kind of build it into my Jenkins file or something like that. I think we're running low on time, so... Yeah, okay. So, thank you. If you've got more questions, come bug me afterwards, you know. Yeah, bugging for questions after. Thank you, Michael.