Hi Al, today we are going to discuss about how to choose a right database for your microservices. If you see the industry trends from the last one decade, the architecture got changed and microservices architecture got introduced in most of the applications. This is Dr. Vamsi Mohan from India and today we are going to discuss about how to choose a right database for your microservices. Before going to that session, I want to share my details about my profile, my experience and my credentials as well. I am a co-founder and CTO in Hubber Technologies which is based out of India and I am next hundred CIOs for the year of 2020. And also I got CXO Excellence Award for the year 2021 and I am top 50 global thought leaders and influencers for RPA in data center technologies and cybersecurity. Coming to my credentials of academic, I hold PhD in computer science and engineering and holding several patents in data transmission and cybersecurity. I did my post graduation in computer science and technology and I hold a master's degree in management from IMA Hemsworth. I like writing and publishing papers and I published 40 plus national and international papers successfully. And also I have successfully driven several industry academia initiatives with various tier one and tier two technology institutions globally. Apart from that, I am an industry speaker in several national and international conferences. Coming to today's topic which we are going to discuss is the agenda will be looking like this. We are going to discuss about the introduction. We are going to discuss about microservices versus monolithic architecture. And we will be talking about micro front ends. How it is different from the traditional front end technologies as well as we are going to discuss about how microservice architecture tackles the drawbacks of monolithic architecture. As well as some portion which we are going to talk about monolithic databases and also some of the microservices database patterns like a database service or a shared database services we are going to discuss about it. And as part of choosing a right database for microservices which we are going to discuss about and also some of the different database models which we are going to talk about. Basically, the poly got persistence models as well as multimodal databases which we are going to talk about in this session as well as what are the different drivers behind choosing a right database for microservices are also we will be going to discuss about it. And finally, we will be talking about the recommendations what kind of a database is suits the microservices architecture as well as followed by the conclusion what kind of a database is right for your microservice applications. Coming to the introduction part of the microservices is as we discussed microservices are gaining popularity from the last one decade with respect to the infrastructure as well as with respect to the loosely coupled architecture. And because of that the loosely coupled architecture as well as the downsized development and the testing setups it came as a familiar in the market as well as it helps to go to market strategies quite easily. And most of these microservices based applications it takes less time to go to the market as well as we have discussed about the it is actually it is the rejection of tightly coupled or a tightly connected architecture apart like not like a monolithic applications or a monolithic architecture it is not completely not tightly coupled and it is a completely a loosely coupled architecture. And because of these some of some of the specialized features like a self-contained and autonomous features actually created the specialty for this microservices and widely popular because of these autonomous and self-contained behaviors. If you see on my screen I have actually shown two different diagrams one is the diagram one states about your monolithic architecture which is completely tightly coupled like the user module threads module as well as the post module is completely coupled as part of your monolithic application. In this application the entire application is tightly coupled and at a time you have to deploy. However in the second diagram it is completely classified decoupled into three different bundles code bundles I mean to say is basically the users threads as well as the post services. And with respect to these kind of a loosely coupled architecture if any module like a users or a threads or a post is not working properly you can you can the developers can work on only particular module or a particular microservice bundle and the other two microservices or a functionalities will work as is as intended without taking any downtime or application outage. And coming to the architecture types of microservices versus monolithic as we discussed monolithic architecture is tightly coupled and because of the tightly coupled the entire functionality will be bundled as a one single bundle or a one component which will be hosted or which will be deployed as a code bundle in the container or a server. And because of that if any feature or any patch you are introducing on top of this code bundle you have to take the outage and at the same time you have to take a complete deployment of this entire code bundle. And this creates time consumption with respect to the deployment as well as it proportionally the testing efforts will increase. If you see in the diagram of a monolithic architecture the user interface business logic and data access layers are tightly coupled as well as all three layers are connected to the single database. All these calls the user interface business logic and data access layers are completely coupled and as a as considered as a single bundle and it will be a huge size with respect to the code bundles as well. Unlike monolithic microservices are lightweight components as well as it is a loosely coupled architecture because of that the entire functionality will be distributed into different code bundles or a different smaller functionality applications which we call it as microservice applications. These are all differently classified and deployed each component as an independent autonomous unit. With respect to that if you see on my right side of the image you will be able to see different microservices as a different applications which will be hosted and at the same time these microservices are connected to the different databases lightweight databases. These lightweight databases which we call it as a micro DBs or micro databases. Basically how it works is microservices will be directly talking to the concerned micro databases which is a lightweight and because of that there is no need for there is no need for there is no need for entire bundling the entire activity and microservices are having microservice application is also known as a microservices architecture which organizes as an application as set of interconnected services. Basically it is highly maintainable and testable products as well as it is a loosely coupled architecture and independently deployed code bundles where the independent lightweight components can be deployed across in the containers and organized around different business capabilities owned by the smaller teams to develop and test and host these applications. Basically if you see different kind of a functionalities will be split into based on the business functionalities it will be split into the different modules as well as the applications those those modules or applications will be considered as microservices. Coming to the microservice patterns we we call it as micro frontends which we are talking as but of the table of contents like what is the difference between actual actual frontend code as well as the microservices frontends. In case of microservice frontends it is a it is again it is a lightweight component and because of that each micro each microservices application will be having their own micro frontends as well as micro databases. These micro frontends are also a lightweight unlike monolithic frontends. Monolithic frontends basically the entire the frontend layer or a UI layer is coupled with the with the business logic and hosted as a single component. However here the micro frontends will also be considered as a independent deployable unit or a bundle which can be deployed independently which will talk to the microservices I concerned microservices as well as it it surfaces the business requirements. Basically if you if you see on my screen on the on the right side the source control of the source control of three different microservices or micro applications is interacting to for surfacing a production application. If you see micro frontend A is Vue J is which is a which is a JavaScript component which is directly talking to the microservice and it is directly hosted to the production. As well as the microservice B or a micro frontend B is having the AngularJS and it is directly talking to the another microservice as well as the microservice micro frontend C is developed with ReactJS which is interacting with the business component of a microservice C and and it will be independently deployed in the production as a different components and all together it will be working as a single application however it is deployed as a different independent components. Coming to the benefits of the micro frontends is basically the micro frontend architecture is very simpler the simpler and that is why it is easy to maintain and manage the code and also it is independent development teams can collaborate on the frontend applications and they can they can develop easily and also they can provide a mean of migrating from old app by having a new app running side by side with it basically how it works is in case of migration or UI transformations without disturbing the old application we can create a new app parallely for the parallel run and till the completion of the entire development we can parallely develop as well as deploy in the production as an app and once these frontend application is completely developed and working fine in the production you can dismantle or wipe off wipe it off the older application of the older view basically it helps for the your parallel run as well and how microservice architecture tackles the drawbacks of monolithic architecture some of the components which we are talking about is flexibility as we have discussed microservices architecture is more flexible and different microsales can be developed since it is a smaller core core basis it is easier to manage the code as well as deploy and upgrade easily and it is it is coming to the reliability microservice architecture is reliable and even out of out of 10 features if any one feature is goes down the entire application doesn't go down we can we can fix the issue and and whatever the feature went for went for down or or the fix which you have applied those component are those microservice you can deploy it and it starts interacting with the other microservices through different ways of communication and also we were talking about the development speed how fast these microservices will be will be interacted and how fast we can develop as we have discussed these microservices are smaller in size and the code bundles are very lightweight because of that the development the feature development is quite faster as well as it don't take much time for the hosting as well as deployment as well as loading these components to the developer system even even testing of a particular feature is also quite faster because of the less code or the the lightweight component and and one of the advantages is building complex application basically you can develop as a independent different components and integrate or you can you can establish a communication between these independent components and it will surface the entire application application functionality basically you can you can develop as a smaller components and you can you can deploy it and once once the communication mechanism is established gradually it the complex application you can develop it easily even the independent components can be further broken into the smaller independent tasks which can be deployed independently as a microservice and also with respect to the scalability scalability is one of the advantages for the microservices each microservices can be scaled individual and as we discussed previously it is in smaller in size as well as caching of some of the data is very effective because of the less size and the cacv a kind of a features like a continuous integration and continuous deployment will be there quite faster and easier as well because smaller in the in the code the continuous integration of your the developers code will be directly checking into the SEM system as well as once it is checked into the SEM system the continuous testing and continuous deployment cycles are quite faster because of the less code as well as the scalable of in size and we were talking about micro front ends and how it is different from monolithic front ends now in this in this slide which we are going to discuss about the monolithic databases a monolithic database is a framework where data is stored transformed centrally as a single data store monolithic data architectures are managed by one team as well as whose business domains are relatively simple and the data is quite rarely changing if if it is an in case of a dynamic changing of the requirements as well as well as dynamic changing of the data models it is quite difficult to manage this monolithic databases as well as coming coming to the scalability of these monolithic databases or it is the data architecture can't scale indefinitely and most monolithic databases don't auto scale it as an application were workload as well as data volumes increase exponentially monolithic databases gradually becomes low when the time passes and also it is expansive and hard to maintain these kind of monolithic databases the ability to consume and harmonize the ubiquitous data in one place managed by the central team is also a difficult and it impacts on the on the team morale as well as the team it increases the team burden as well as monolithic databases often have high latency and throughput that naturally arise from the concurrent database read writes from the different disconnected teams with monolithic data architectures it is also challenging to respond to new needs without altering the entire data pipelines basically the agility is quite less compared to the microservice databases and here it is it is a one complete database which will be directly talking to the application because of that if there is any any particular new requirement comes it impacts on the application as well as it impacts on the database as well eventually the the third monolithic architecture lacks of a modularity and suffers the homogenization of the technology basically this is this is one of the advantages when we are talking about different databases talking to the single application it is quite complex in case of a monolithic databases basically your application is directly talking to a one one database only it cannot interact with the different databases even if it is a hybrid kind of a monolithic databases when we are talking about even the databases should be the similar kind of the same databases like a oracle or a mySQL server or mssql server or a db2 kind of a application it should be homogeneous not one application can talk to a different kinds of a databases this is one of the drawback with the monolithic applications as well as a monolithic databases eventually a monolithic database would naturally result in inpatient data consumers disconnected data procedures and a backlogged engineering team burdens by towering the technical depths basically what what will happen is because of the complexity the team morale will go down as well as some of the technical depths will increase and it cannot be scaled according to the the agile requirements we were talking about different micro database patterns basically database we call it as a database dedicated databases to the microservices which is an ideal scenario for the microservice based applications the idea is simple each microservice has its own data store unlike the entire database and because of that assume in my screen you can see the service a is interacting with one database a small small lightweight database as well as service b is interacting with the another data another small database and service c is interacting with the another small database because of this the smaller in size of the database it will be it will be quite easier to interact also the performance of these the performance of these request response system is quite possible and coming to coming to the heterogenicity of different kinds of databases interacting if you see on my screen service a is interacting with the my school database and the service b is interacting with MongoDB database and service a is interacting with the customer database basically you are the same application even though the front end or the microservice is developed with one particular technology or environment it can easily talk to different databases I didn't get that could you try again and coming to individual data stores are easily scalable on my on my screen you can see different databases and it is easier to understand the services within the data as a whole finally the data database per service we are able to use polyglot persistence we can use different database technologies for different microservices in case of database per application and this is an ideal scenario when we are talking about microservice architecture and it is especially for the scratch development of applications are a greenfield kind of applications and the second kind of a application is a micro database shared database with the different microservices this is this is this kind of a scenarios will come as part of your transformations with the database is core for the business activities and we don't want to optimize or normalize the database and the front end code as well as the microservices will be optimized the entire application code is normalized or optimized into different microservices and still it is interacting with the one database even though you can you can reap some benefits with respect to the application leaves from the micro front ends and microservices but when it comes to the database the the cons basically the button is still there with the with the existing database because the database is not normalized or refactored with respect to the micro databases and also all different kinds of a microservices will be interacting with the single database and because of this the number of requests will be more to the single database and microservices with shared database can't easily scale unless unless it is it is not recommended unless we cannot avoid the situation as well as the database will be a single point for all these applications and if anything happens to the database or a schema automatically all the applications will stop working working and the application will directly go for a outage and coming to coming to the concept of how to choose a database for microservices is basically various databases if you see from the last one decade we have seen multiple new databases introduced recently today database market is brimming with different possibilities and you can pick the best database for your needs however the database choosing of these databases also depends on the application structure as well as what kind of a frameworks you are using we can by deciding on a method for constructing different database models and and and when we are talking about these different database patterns or different database types one is a polygot persistence as well as the second one is a multimodal databases which we can discuss in the subsequent slides on my screen you can see what are the different kinds of a databases comes as part of your relational database relation database databases or a databases schemas as well as what are the different databases basis or a data stores comes as part of your non-relational schemas polygot persistence basically we are talking about one is two one similar kind of a thing which we we have discussed as part of your micro dbs or a micro macintz basically if you see on my screen service a interacting with one database service b is interacting with one database as well as services is interacting with another database because of that the basically the service the services or application layer is a quite simple and lightweight which is interacting with the lightweight database and because of that different the the processing is quite faster as well as the microservice architecture enables difference kind kind of data store technologies with respect to the different technologies basically applying the polygot performance and each development team can choose the persistence technology that suits their needs if you see on my on my screen service a is interacting with the Cassandra as a core application data as well as service b is interacting with a key value pair of a database which is for the difference data as well as service c is interacting with document database which is for the content and service d is interacting with the graph database basically if you see different functionalities are sitting in different microservices as well as these microservice are interacting with different data a smaller chunk of the data in the form of a micro databases and and as part of this example you can see different kinds of a databases as well basically one one microservice is interacting with the Cassandra and one microservice is interacting with a key value pairs as well as one microservice is interacting with document them kind of a kind of a services and the the another way the another kind of services are multimodal databases you can use multimodal approach it allows database to support more different kinds of databases if you see for instance you can see my the diagram which i have shown as part of the screen the service a service b service c and service d there are four different services and the first three services are interacting to the the core database and the service d is interacting with different uh uh graph database even though even though that the database is we basically the types of the database is completely different uh you can see service a is a Cassandra uh interacting to the database and service b is interacting to the key value and services interacting as as part of the json and service d is directly interacting with the graph database the multimodal approach offers what polycode persistence count but operationally it is a very simple when you only have a one platform it's easier to manage the system if each service has its own model of interacting with the data the polycode performance fits perfectly for the microservices and your application if the if your application is not too complex and it is better to stick for this kind of a mechanism and you can actually in in in hybrid kind of uh approaches you can combine both these uh this kind of uh uh approaches and uh we were talking about uh different different non-relational database models as well as we were talking about uh some of the multimodal and polycode models what kind of a database lies as part of uh these kind of a patterns uh non-relational databases is basically it is a no SQL or a no SQL kind of a different types of a databases if you see on my screen some of the uh widely popular no SQL uh databases I have given as part of uh this example like a MongoDB or S&R uh Neo4j or Redis kind of uh databases are widely popular as part of your non-relational uh relational databases and uh second kind of uh databases are uh document databases the databases uh the database of this type of uh store query data as a JSON like documents document databases offers and in due to data models and uh and basically the uh you don't have to uh run any joins or decompose data across the tables document maps the objects in the application code itself because of that uh the entire entire uh uh dependencies will be taken care in the application code itself and uh and since since document database are distributed systems they are easily scalable and document database are perfect for the content management and the storing storing catalogs and you can apply uh for the blog applications or e-commerce platforms popularly uh these kind of a document databases we call it as uh uh widely popular as a MongoDB or a uh Cloudant database or examples for the document databases and another another kind of uh uh databases key way key value databases uh which we used to write probably one ticket back uh the key value pair kind of uh uh operations or when we when we have implemented certain data structures uh these key value uh pair of our data we used to manage in the traditional old systems here also is the same same kind of operation mechanism it happens uh the key value data uh suits for the session oriented applications and uh there are some of the databases like uh Radish Amazon DynamoDB as well as Oracle NoSQL databases are widely popular uh for this key value pair kind of for databases basically the key will be assigned uh a unique key will be assigned to the data or a value and uh when when we have searched when when when the search criteria triggers automatically uh the value will be picked as part of the data and it will be visible to the end user uh this this is uh this is more more on the conventional method the key value databases there are certain data uh algorithms are available for uh for uh extracting the data and another another kind of uh databases uh graph DB basically graph DB uses the graph data models this is this is also uh we used to uh apply an in as part of our data structure methodologies uh which the data is uh is these data entities are connected in the in the form of nodes and uh relationship uh stored in the edges the main value of the graph is stored and uh navigate those relationships and it is quite faster as well as uh it is there is no uh unlimited uh relationships you can establish it with it and uh basically uh it is it is widely used for the fraud detection as well as social media networks and uh recommendation engines and some of the databases like uh orient dp new 4j or amazon nipchoo are popularly widely uh accepted in in the industry as part of the graph DB and uh and another another kind of uh uh databases are column-based database and relation databases column-wise uh based database uh data is stored in in the form of columns basically uh some of some of the pure uh database management systems will store uh this kind of uh databases you can you can use column-based uh databases building and uh data warehouse uh kind of uh tools and uh there are there are certain tools like uh apache kessender apache hbase are widely popular uh and and uh these columns store is a great solution for the big data uh processing and uh relational databases which we are talking about uh some of some of the these uh relational databases the data is uh stored as part of rows and columns and uh and this is this is a standard technique or uh uh the standard way of storing the data and uh some of some of the databases like a db2 mysql server or uh post-bis SQL or a rackle kind of uh uh application or databases are widely popular as part of this and uh a rational database is uh excellent fit for uh services that manage low volume of a stable data like uh high volumes basically uh the volumes are limited in case of a relational databases and uh drivers behind choosing a right database for microservices is basically uh there are different kinds of uh classifications or a categories which you can think of one is a fmr data basically the temporary data store uh which uh for supporting the user request as well as the caching techniques uh kind of operations you can use as part of fmr data and the second kind of a data is uh transient data uh which where you can store uh the logs messages and signals uh and as well as the time series data uh will come as part of the transient data you have to uh you have to choose a right database for uh uh right categories and also the operational data uh what kind of uh uh operation data you are storing as part of it and uh for this json graph or uh relational database are quite uh uh quite friendly as well recommended for this this kind of operational data and the fourth one is a transactional data which is having the uh uh acid kind kind acid kind of uh controls where uh the transactions will be happened and committed and stored as a persistent uh uh data records uh you have to choose uh on what kind of a data you are managing it or uh you are dealing with it and based on that uh a right for a right kind of a data right kind of a uh database type you can use it basically uh one of the uh one of the advantages with microservices are it can easily talk to different databases uh like like uh if you have a app model data you can use one kind of a database as well as when uh you are using the transactional data you can use uh another kind of a database uh and operational data you can use another kind of a database basically uh you have a quite uh you have a quite flexibility to choose your uh requirements and you can choose your own databases and coming to the uh recommendations and conclusions uh basically what we strongly suggest is uh basically uh give the choose a right requirements for the right right databases for the right requirements and it is it is not uh like a no SQL or we cannot say no SQL might be a right uh option or rational relational databases are the right approach uh as we discussed in the previous slide uh for the right requirements right database can be chosen and also you can give flexibility to the different the team capability as well and uh with that uh one is the implementation flexibility as well as the team flexibility you can achieve uh with respect to the my this microservices strategy thank you very much and uh it is a pleasure sharing my experience with you all thank you


What's going on everybody? Got Kyle and Jake here with Layton Sports Cards and we are ripping one box for Colby M Gonna be one box of 2022 tops f1 sapphire Good luck Colby. Let's see what we got All right, sounds good Matthew. I see the order gotcha, sir Gotcha my man be about 30 minutes about 30 minutes and get that going for you Good luck Colby one box of 2022 f1 sapphire four parallels here Let's see what we got and good luck Matthew appreciate you sir a Checo Perez base ghastly, Magnuson and Checo Well max for sapping already won the uh won the championship. He's got six races to go Already won the championship. There's Lando. There's an aqua a nice one there Lewis Hamilton Winner round 15. Believe that's Russia to 99 nice little aqua Lewis Yuki and max winner belgium and next year these cars gonna have to have max on every single winner card Franz Toast George Russell George Russell and Chuck Leclerc Why was he sort of down there? The car is just good. He's just the the best driver in the best bar. I guess. I don't know Has to be something to do with the car. I think but he's also the best driver Lewis Hamilton There's Lewis nice one there on the uh short print Lewis with the flag Vettel and Lewis triple Lewis packed there Yeah, he won last night during the uh sprint race not even during the race today. They got a race at one He won during the sprint race Um Sebastian Vettel There's vesti future stars and Danny Riccardo Yeah, he already already won He does own f1 time being there's a gold Gold is going to be Sebastian Vettel a nice one there They're retired man now Sebastian Vettel nice true gold to 50 Well Sebastian portrait Magnuson and Lando Yeah, it's pretty crazy. It's pretty crazy for sure Winning that early on in the year Checo Perez this one is Yuki Sonoda on the um sepia Sepia to 100 nice Yuki Another Yuki and max on the winner last pack Say frederick vesti got lime green this one said Vettel again nice one there $1.99 a couple nice um Vettel numbered And Marcus Armstrong All righty there Colby that'll do it for your one box of f1 sapphire. Thank you so much. We'll get them right out to you


What's up everybody? Ricky Caruth here. Look, 2021 is right around the corner and I want you to hit the ground running. On December 11th at 4 p.m. central, I'm going to go through my 2021 goal setting and planning session. I'm going to reveal my new 90 day action plan, help you set up your goals in an entire year so that you can have the best year of your life. Right now I currently have 17 pending deals. I'm almost to 100 deals as a single agent for the seventh year in a row and I can't tell you how excited I am for 2021. I think next year is going to be one of the best years we've ever seen in real estate. All the fundamentals of the industry are telling me that this is going to be one amazing year and I want you to be fully prepared. So click the link below, join me December 11th, 4 p.m. central so we can all be on the same page. I'm super fired up. I hope you're fired up and I'll see you at the event. Let's go. Thank you.


FNN the Tiger financial news network. Good morning everyone. This is the tiger financial news network 10am market update on this very last day of August August the 31st Wednesday. We're looking at the Dow up $9 at $31,791 ups just made a negative. This is after the dropping that it's had it's really done a little bit more than I want you want to the downside. Made a low of $31647. I see the $31600 to $31500 in the very near term. That's kind of what I'm watching very closely, especially on the weekly chart for Friday's close and we'll see what happens. We're looking at the S&P right now. So the sell mode in the daily chart of the Dow continues. The sell mode in the daily chart of the S&P continues up $9 to $39.95. There should be some buying on this last day of August. I think portfolio managers have had a lot of days in which to reorganize their portfolio. So maybe they're thinking of September or maybe there'll be some new buying. We'll get to see in a moment. QQQ is up $2.33 or $3.40. It's quite a bit above yesterday's a low. I didn't update that. The low yesterday was $298.44. $298.44. It's up at $3.38. Just attempting. I'm beginning to like some of the action in the Q's at this particular point. I don't love it. I'm just saying I'm liking just the near term trying to find support. I think it's getting a little bit of support here. We're looking at the SMHs. The semiconductor is ready. They go together but in a way I'm trying to treat the Q's as something separate the NDX100 a little bit separately to the semiconductors. Semiconductors need to find some support very soon because this is a really important moment for them at 216 up to 7 cents. Looking at the gold. Gold contract took a driving yesterday and again today down 8 at 1728. Has the year taken out the low of July but we're watching this closely. Looking at the dollar. Has the year made leg deep to the upside. Very nice action up 22 ticks at 109.01. I made a new multi year high three days ago. It needs to go just a little bit above that to start your leg deep. I'm looking at crude oil. Crude oil. Wow. Down $1.50 at 90. This is just stuck in the low range. Has it broken down yet? Wow. It starts to trade under 84. That'll be a big problem in the TLT. This is bonds. Bonds are trying to rally up 30 cents at 113.24. Fuzzle, Chapman, Tycho, technicians are coming up. I hope we can stay great programming here all day. We'll be back and check out my opening gold day news video.


Oxford Bibliographies is a great resource to start your research for religion 325H5. You will find a link to Oxford Bibliographies in the description of this video below. The entries in Oxford Bibliographies are annotated bibliographies that link to primary and secondary sources. Keep in mind that for this assignment, you need secondary, not primary sources. Use Oxford Bibliographies to find sources for your research and cite to those sources. Do not use Oxford Bibliographies itself as a source. Do not cite to it. Keep your keyword searches simple. Let's imagine that you are interested in Zoroastrian along with Jewish and Christian apocalyptic texts from the 1st and 2nd centuries of the Common Era. Let's type A-P-O-C-A-L-Y-P with an asterisk at the end of the stem into the search box. Put the asterisk onto the stem of a word to retrieve all forms of the word. Apocalypse, apocalyptic, apocalyptical, apocalypticism. You will retrieve all of the articles that mention some form of the word apocalypse. Some of the articles are listed under subjects that fall outside of the scope of interest for this example. Over on the left hand side are a complete list of subjects. Let's click Clear All to select them. Now you can browse the list and select the ones that relate to your scope of interest. For this example, let's click Biblical Studies and Jewish Studies. Scroll down to the bottom and click Apply. Now we just have the articles with the subjects Biblical Studies or Jewish Studies and they are sorted by relevance. The top result is the article Apocalyptic Literature. Let's click that to open the article. You can navigate the article using the headings listed under in this article on the left hand side of the screen. Click the black triangle twice to expand for the full list of headings. Keep in mind that you want secondary sources that analyze primary sources rather than primary sources. Type anthologies as well, since these are collections of primary sources. Let's click on the heading Social Contexts. Read the titles and the annotated descriptions of the resources to see if any interest you. To access the resources through the library, click the little plus sign next to Find This Resource. Then click Get it UTL. You will be directed to the library website. Let's look at the library record for Graba's article, The Social Setting of Early Jewish Apocalypticism. If the record shows full text availability and a blue hyperlink, then that means you can open the resource and read it online. If you have any questions, you can contact your liaison librarian or reach out to research specialists by clicking Reference and Research Help. You will find hours to meet with us in person at the library or using the online chat.


In today's series of lectures, we will be looking at factorial design of experiments. Some of the popular references for this topic are shown in this slide. The first one is the book by Box Hunter and Hunter, Statistics for Experimenters by John Wiley published in 1978. The prescribed textbook Montgomery and Runger applied statistics and probability for engineers. Fifth edition, Wiley India 2011, there is also the more detailed book on design of experiments by Montgomery, design analysis of experiments, eighth edition New Delhi, Wiley India 2011. And there is also the reference to the paper by Box and Wilson on the experimental attainment of optimal conditions, Journal of Royal Statistical Society, 1951. So we will be looking at the history of statistical design. There was a phase in 1930 where Sir Fisher introduced some very interesting concepts. He was working in a research institute involving agriculture and he came up with some novel ideas. He brought in the concepts of blocking, randomization and repetition, factorial design and analysis of variance. So these remain the backbone of statistical design of experiments. So we are really indebted to Fisher for bringing these concepts. The next phase was the development of response surface methodology. We will be talking about this particular topic pretty soon and this was more suited for industries where you do a set of experiments in a narrow range of operating conditions and you want to know in which direction you should proceed in order to maximize the process yield or minimize the reaction time and so on. So essentially this becomes an optimization exercise and you are looking at the particular search direction to find the optimum location. So this is a good combination of optimization and statistical design of experiments. Just imagine you are lost in the forest and you really do not know how to come out of it, what direction you should go. So similarly the experimenter also faces this situation. He is comfortable with the R and D results but where to move further especially when there are large number of variables influencing the process, the response surface methodology is a very effective tool. And unlike the field of agriculture, the results in the industry are pretty much made available quite soon. So we bring in the concept of immediacy and you also have to plan for the next stage of experimentation. So this brings in the concept of sequentiality. The response surface methodology was widely adopted in the chemical and process industries and during this period there was a general lack of awareness of this kind of methodology. They were comfortable doing experiments in a very systematic manner. They did not really worry about the large number of experiments or even if they worried about it they really did not know how to reduce the number of experiments and they felt that by reducing the number of experiments they would miss unvaluable information. And also there was lack of user friendly statistical computing tools. Well this is not really a drawback in my opinion because many of the statistical applications for real life problems involve reasonable mathematics. I would not say simple mathematics or complicated mathematics. Reasonable mathematics I think which should be done but I think to popularize this technique on a large scale especially to industrial people. There was a pressing need for software tools where people could plug in the data and get the results. Then came in 1980s the philosophy of Taguchi. This created quite a stir not only among the industrial people but also among the academicians. There was a big debate on the Taguchi principles and the basic idea in Taguchi's method is to have a robust experimental design which will produce products with minimum variability and the technique involves the identification of levels of factors or settings of factors that will force the process towards a particular mean value with minimum spread around this particular value. And also the experiment or rather the process has to be designed such that it becomes insensitive to variation from uncontrollable environmental conditions, raw material or component variations. So this became also popular and took the statistical analysis to the discrete parts industry like semiconductors, automotives, electronics, aerospace manufacturing etc. And there was also considerable research or flurry of research activity to find alternatives to the Taguchi's method and quite importantly the concept of statistical design was also implemented in the academic institutions. So this gives a hint that sufficient theory had developed by let us say around 1970s late 1970s. For example you saw the book by Box and Hunter it was in 1978. So by 1970s and 1980s it became a full fledged academic course with sufficient theory. And Montgomery in his design of experiments book proclaims that the experimental design has to be integrated in engineering and science to attain industrial competitiveness. So there must be lot of industries which can benefit by the application of design of experiments. So if students are trained in this area they can contribute in an industrial environment to minimize or towards minimization of costs, time, manpower requirements and so on. So look at this course rather than as a theoretical subject as one which should be implemented for saving of time, money, manpower, energy and so on. So let us come down to the factorial design, what are the advantages, it is a neat structured and systematic approach to experimentation. It gives you a set of rules or procedures which are simple to understand and easy to implement. Okay, there is nothing complicated if it was complicated nobody would follow it. And it can be subject to rigorous statistical analysis and the results interpreted through these statistical means are accepted by the scientific community or even if there were ambiguity or subjectivity that is quantified in terms of the P value which we saw yesterday. So people were more comfortable doing one variable at a time experimentation. They will keep all other variables at their fixed values and then vary only one variable. After varying and completing the experiments with this particular variable then they will keep this variable constant, take up another variable, vary that while all other remaining variables are kept at their fixed values. So this is a one variable at a time approach that looks very logical but when you look at the factorial design of experiments it is much more compact, involves lesser number of runs for obtaining the same level of accuracy. And another important thing which we will be encountering very frequently is the identification of the interaction effects through the factorial design of experiments. What is interaction effect? How does it influence the process we will soon see. Another beautiful or elegant advantage of the factorial design is its orthogonal property or characteristics. I like this very much and what is orthogonality and how does it make the design elegant and simple. We will soon see, first my objective would be to give the introduction to the factorial design of experiments and then we will look at the orthogonal design experiments. Basic idea in the orthogonal designs is the different effects are sort of designed in such a way that their contribution of sum of squares to the overall sum of squares become independent of each other. This may be a bit difficult to understand at this moment in time but just let us keep it in the back of our minds and come back to it in a short while from now. When you have a factorial design which minimizes the number of experiments, sometimes you may have a large number of factors to study and even a factorial design would make things cumbersome. So what you can do is divide your factorial design into several fractions and carry out the fractions in a sequential manner. Even fractions give you valuable information so you do not have to complete all the fractions of the entire set but you may stop at any point you want once you feel that you have got sufficient information. So it is very flexible so it does not force you in any way it makes life simple when you have less number of factors and when you have large number of factors it also helps you to proceed until you are satisfied. The factorial design of experiments let me put to you bluntly is not implemented in its original form most of the time because even factorial design of experiments may involve large number of runs and especially when you have let us say 5 variables that would be something like 32 runs and always you know that you have to carry out repeats to get an idea about the experimental error and in such situations even 32 into 2 or 32 into 3 leads to 64 or 96 runs which are probably too many and so it is important that we sort of streamline the entire methodology and designs are available to reduce the number of runs but they are all built around the factorial design concept. So it is very important for us to understand the factorial design and once we become familiar understanding other designs become quite simple. So it forms the basis for the optimization procedures like the response surface methodology which I talked while earlier the optimization of the process conditions factorial designs play a very important role here and another beauty about factorial designs is it can handle both qualitative and quantitative variables together. What I mean is if you are carrying out experiments in a chemical reactor and you are looking at variables like feed, flow rate, temperature, pressure these are quantitative variables. You may also be looking at catalyst A and catalyst B unless you quantify the catalyst in a detailed fashion and incorporate it into some mathematical model we assume that these are qualitative variables. We want to see whether catalyst A is more effective than catalyst B in affecting the yield from the reaction or well you can even do experiments with machine 1 and machine 2. So these 2 machines would then represent 2 levels of the qualitative factor machine type. So these are some examples of qualitative variables your cleaning clothes in 2 different washing machines so they become qualitative variables. So the advantage of factorial design is it helps you to consider even qualitative variables in addition to the usual quantitative variables. So the factorial design is a very flexible one and many variants of the factorial design are available we will be looking at a few popular amongst them and it would be a simple matter for you to look up at the remaining ones and understand them. What do we get out of a factorial design we get a empirical model which gives the process response in terms of the different factors their interactions tells us which factors are important which interactions are important again I am using the word interaction without really formally defining it please be a bit patient I will come to it shortly. As was cited from Montgomery's book statistical design is compulsory for industrial competitiveness it is a very important thing. So what I would like to tell the students especially research scholars and students who are working in the laboratories various undergraduate laboratories that it is okay to have scattered in your experimental data you may not get exact reproducibility of your experimental runs because there may be several factors which may be affecting your runs you do your best through blocking randomization to try to get rid of any systematic variability and even if you have variability beyond this let us so be it okay and you use a proper statistical tools like the design of experiments to find out whether the factors in your experimental work are comparable with the experimental noise in which case the factors are considered to be ineffective or the factors are contributing way over the experimental noise effects it sort of helps us to segregate the contributions from different factors even if the two factors are acting in a combined manner that effect is also isolated and importantly all these effects are isolated from the experimental noise or the random errors and then it helps us to compare the effect of changing the different variables with the variation from experimental error and helps us to make the necessary conclusions. Again it involves hypothesis testing and we have to postulate the null hypothesis and alternate hypothesis and based on the experimental evidence we have to make come to a decision. So it helps us to tap the rich informative content from the experiments using limited number of experiments that is the beauty of statistical design of experiments. So a few definitions are in order at present test conducted in as reliable manner as possible so that credible conclusions may be drawn for process modification or improvement experiment. This is one such definition most important thing is an experimentalist or an experimenter should approach the experimental work with an open mind he should not have preconceived notions that the experiment is going to behave in a certain way. If that is there then he will not have that curiosity or that open mindedness to accept results as they come and try to find reasons for that doing experiments also is an admission of the fact that our theoretical framework for analyzing complex processes is not fully developed or it is highly complicated they may not yield correct exact solutions or solutions in closed form analytical expressions may not be obtainable the numerical simulations may be very expensive. So it helps us to fill in the gap also experimental work is essential to validate the high end and numerical computations okay. So that the credibility may be built on those simulations the model equations used and the assumptions made they have to be validated with experimental data. So inevitably wherever you go you cannot avoid doing experiments doing simulation work with computers is not the end of it it is essential to do modeling of varying levels of detail but eventually you have to back up the simulation results through proper experimentation. So what is a factor factor is the variable of interest which is being controlled by the experimenter and he can set it at different levels for example if the experimenter is looking at washing machines performance he can alter the speed of the washing machine drum and so that would be a factor and he can set different speed levels for this particular factor. The factors are chosen to be independent of each other in other words if I change one factor setting if another fact is automatically affected by it then the two factors are not independent for example if I increase the power to a machine and the machine speed increases as a result of it then I cannot say the machine power and the machine speed are independent factors. So we have to look at factors which may be varied independent of each other level the particular setting or value of the factor it may be quantitative or qualitative quantitative means temperature is 30 degree centigrade 50 degree centigrade and so on. Qualitative means machine A machine B or catalyst A catalyst B and so on. Design what is the design it is a strategy adopted to do experiments in an efficient manner okay. So design sort of gives you the matrix of experimental conditions or settings where the experiments have to be carried out it is a kind of a blueprint for doing your experimental work and choosing an appropriate design is important it is not as simple as it looks many times I have seen students who are very enthusiastic about the statistical design of experiments often are at a loss as to what is the appropriate design for their experimental work okay. So this is a very important feature and we have to look at the factors sorry the reasons which go into the choice of an appropriate strategy or an appropriate design. So factorial design the levels of the factors are varied in a systematic and efficient manner and output that all combinations of these factors are found now we come to the important 2 power k factorial design a factorial design involving k factors only and only 2 levels of each factor. So we represent factorial design usually in the form of 2 power k sometimes students may get confused they think that they are only analyzing 2 variables this is not correct you can analyze any number of variables you want you are in fact analyzing k variables in the 2 power k design okay and what is the 2 doing there the 2 represents the number of levels for each factor. So if you are having 2 that means each factor will have only 2 levels a low level and a high level okay if you are having a 3 power k design again you can have any number of factors you can call them k number of factors and the 3 represents the levels of each factor each factor will have a low setting medium setting and a high setting. So the 2 represents the number of levels in the factorial design here it is referring to 2 levels of the factors. So number of treatments this is not very common we came across it quite frequently in the single variable experimentation now we are looking at multivariable experimentation so we can have any number of factors or variables and the number of treatments refers to the complete and unique set of experimental conditions. If there are 2 levels of catalyst 3 levels of concentration and 2 levels of pressure then the experiment has 2 into 3 into 2 which is 12 treatments when there is only one factor investigated in the experiment with p levels the number of treatments is p that we have seen. So when you want to find the number of treatments do not count the number of repeats when calculating the number of treatments and it is not commonly used when there are several factors. So in factorial design what we are doing is we are not varying the variables one at a time sometimes the variables may be changed together okay so in doing so you are reducing the number of experiments without losing on the information aspect. So as I said earlier if there are several factors then even a 2 factorial design may become infeasible so a fractional factorial design is required. So I think it is time for a small break now please sort of think of those difficulties you were facing when doing experimental work and also reflect on how the stated advantages of the factorial design may help you in your experimental work. We will continue further with the discussion on the advantages of factorial design in a short while from now fine. So the factorial designs can measure interaction between factors thereby accounting for deviation of responses from the expected additive effects of the factors sometimes even though you are having independent factors when they act together they may act either independently or in a combined fashion it may be a synergetic fashion if you want to put it like that what is meant by a synergetic fashion the net outcome of the process would be over and above the independent or additive action of these 2 factors for example you are having factor A and factor B which are completely independent in all respects and if you vary them in a factorial design the response would be a constant plus beta 1 XA plus beta 2 XB. So this shows that over an average value there is contribution from factor A and contribution from factor B but on top of it if you also have a contribution like beta 0 plus beta 1 XA plus beta 2 XB plus beta 1 2 XA XB that means in addition to acting independently the 2 factors are also combining into XA XB and influencing the process so this is termed as interaction and for attaining the same level of precision of the effects of the factors the factorial design requires less number of runs than the one variable at a time approach. So you can see that this is a simple figure where the process output is Z, so Z is given by 2 X1 minus 3 X2 okay so I am plotting X2 and X1 and these are values of constant Z so Z is equal to 2, Z is equal to 4, Z is equal to 6 and Z is equal to 8. So this is a kind of a contour plot where you are shown the variation of both the variables on the X1 and X2 axis and these lines represent values of constant response and these are linear. So if you look at this Z is equal to 2 X1 minus 3 X2 so Z is affected twice by X1 and 3 times by X2 and X1 and X2 are acting in opposite senses increase in X1 increases the Z value but increase in X2 actually is found to reduce the Z value but the important thing is the 2 factors X1 and X2 are acting independent of each other and this is another example where you have Z is equal to 3 X2 plus 2 X1. So you can see that again X1 and X2 are acting independent of each other, Z is a constant and you can see that there is a negative slope okay. So when X1 increases X2 decreases but when X1 increases and X2 increases you can see now that the Z value is increasing. So you can have different types of relationship between the factors and the process response and in these 2 examples we have shown the 2 factors to be acting independent of each other but in many processes there may be considerable interaction involving the 2 factors which also has to be accounted for. So I will just give an example to demonstrate interaction between factors using the Minitab software. Let us look at scores of a batsman in different innings okay let us consider 8 innings these may be 4 test matches or more depending on how many innings were completed in a given test match. So we are looking at the batsman using a light bat and heavy bat I do not know how many of you would recollect sometime back maybe 10 years back there was lot of discussion on a star batsman using a heavy bat and what was the impact on his shoulders and arms. Anyway you can have a light bat and you can also have a heavy bat nowadays heavy bats are becoming more popular and let us also look at the type of a drink the batsman may have had before coming out to bat it is coffee or tea so he is either using a light bat or a heavy bat or may have drunk coffee or tea before coming out to bat okay it is a very fictitious example just to drive home the concept of interaction. So we want to see the influence of the bat and the effect of drink on his runs scoring well the runs are not looking very impressive overall the highest is only 44 well it may be a T20 match also so we cannot say anything but these are the runs scored anyway. So we have to see the influence of the type of bat he has used and the type of drink he had taken before coming out to play and see the impacts on the runs he is scoring unquantifiable or qualifiable factors like form all these things may not be considered they are random effects or uncontrollable factors so we will represent them in a cube plot it is not a cube but that is a general term it is a square plot so you are having bat on the x axis and drink on the y axis you are having a light bat here and a heavy bat the average runs scored with light bat and coffee is 26.5 light bat and coffee so 25 plus 28 is 53 53 divided by 2 is 26.5 and that is what we have as the average here. So we are saying that light bat is a lower setting coffee is a lower setting it is completely arbitrary you can set light bat as a higher setting and coffee as a higher setting it is completely arbitrary for convenience we are just putting light bat as low level and coffee as low level tea as high level and heavy bat as high level okay. So when you have 2 factors and these 2 factors are at the low levels we represent it by 1 then here the drink is still coffee but the bat is heavy so factor A which is the weight of the bat or type of the bat is now at a higher level so factor A is at a higher level factor B is at a lower level and so we call it as small A on the other hand if you go like this you are now moving in from a lower level of drink to a higher level of drink from coffee you are moving on to tea but the bat is still at a lower level a light bat okay lower level and higher level I am just using it without implying that a lower level is bad and higher level is good I am not saying that it is just 2 settings of the experimental variables so you are having coffee at lower level and tea at higher level but the bat is still at lower level of light bat. So you are having the label as small B in this case you would easily understand that both the settings are at the higher levels bat is at a heavy setting and drink is at a tea setting and so you are having A into B so this is the general nomenclature we will be following in design of experiments and calculations of the different effects. So we plot a normal probability graph and we can see that A and B are very far from the normal probability line saying that these 2 are having an effect on the runs scored they cannot be at dismissed as random effects A, B is the interaction between the bat and the type of drink and it can be seen that it is not having an effect. You can also see that the type of bat is having a negative influence whereas the type of drink is having a positive influence for example if I am going in from a light bat to a heavy bat the performance is getting lower if I am going from coffee to tea the performance is getting better. So you can see a lot of information given in one simple diagram so this is the effect of the main effects or this is the diagram which shows the main effects it shows that when I am going from a light bat to a heavy bat the mean runs scored is reducing. If I am going from coffee drink to tea drink the average or means runs scored is increasing so it looks like heavy bat is not good and coffee is not good. If you want to score more runs you have to go for a light bat and drink tea before coming out to play. If I am also looking at the interaction plots they are parallel what it means is if I am going from coffee to tea using a heavy bat my performance increases by a certain extent. The performance increases by the same extent when I am using a light bat also of course when I am using a light bat my performance is higher but with the light bat if I am changing my drink from coffee to tea my increase in runs scored is equal or comparable to the runs scored when I am going from coffee to tea with the heavy bat. So the weight of the bat or whether I am using a light bat or a heavy bat it really does not affect the performance enhancement by changing the drink okay. So the same change is observed when I am changing the variable from one level to another level at different settings of the other variable okay that means the first variable or first factor is acting independent of the other factor I request you to think about this a bit. Now we want to demonstrate interaction let us now substitute beer for coffee again this is a completely fictitious example I do not say that batsman have beer before coming out to play just to drive home the point of interaction I am coming up with this example. So you are having light bat again as usual and then heavy bat then instead of coffee the batsman may either drink tea or may drink beer before coming out to play. I do not think he will drink both and come out to play he will either drink beer or drink tea and come out to play and couple of things are noticeable in the previous table of runs scored the runs were pretty close to one another not too high not too low but if you look at this table of runs scored you can see some very bad performances and you can see some very good performances. So even he has scored a half century a couple of times so let us analyze the effect of the light or heavy bat or beer or tea on the batsman's performance. So again we are putting this in a plot where for each setting the average value is reported for beer and light bat the average runs scored is only 2.5. So 5 plus 0 is 5 5 divided by 2 is 2.5 so the average runs scored is only 2.5. Similarly with the heavy bat and beer the average performance is 51 from beer and tea 39.5 heavy bat and tea it is 24. So what do these numbers really mean? So now when we plot the normal chart or graph we can see something very interesting earlier this AB factor was close to this line and we dismissed it as random effect. Now you can see that A and B are lying on the same side of the graph they are having a positive influence on the batting or runs scored whereas this AB combination is having a negative influence on the runs scored very interesting. Now AB is also significant A is the type of bat used and B is the drink taken by the batsman before coming out to play AB is the interaction between the 2. So this main effects plot tells that heavy bat is good if I am going from a light bat to a heavy bat I am able to score more runs and if I am going from beer to tea again I am scoring more runs. So it looks like the batsman should go in for a heavy bat and drink tea rather than beer before going out to play. But these main effect plots are misleading when considerable interaction effects are present. So let us look at the interaction effects before discussing or comparing between main effects and interaction effects. This graph is very striking and completely different from the earlier interaction graph we saw previously. The earlier interaction graph was parallel when you went from one drink to another drink for a given bat the batsman's performance increased by a certain extent and the batsman's performance increased by a same or similar extent when he changed the drink at a different setting of the bat. Earlier he was using heavy bat and even after changing to light bat the performance upgradation did not change when he changed the drink. But if you look at this if the batsman takes beer and uses a light bat his performance is very poor when he goes for tea and still with the light bat his performance increases dramatically. The batsman using a heavy bat and tea the performance is bad okay. Heavy bat and tea is still better than beer and light bat okay. So tea has helped him somewhat but this is very interesting when he goes for a heavy bat and has taken beer the performance improves like anything okay. One may imagine that the batsman may be in extremely good spirits with the beer and also he may be swinging the bat merrily even though it was heavy connecting and making a lot of runs okay. So it may be seen that if a batsman is using a light bat his performance changes by a certain extent when he goes from beer to tea but when he is using the heavy bat the performance changes in a completely different manner. So the effect of one factor on the runs scored depends on the setting of the other factor when the interactions were not present the effect of one factor on the response was independent of the setting of the other factor. Now when we have interactions this is very very important when we have interactions the effect of one factor on the response will depend upon the settings of the other factor or other factors. So depending upon the level of the second factor the first factor shows the effect on the process. So this clearly shows that there is interaction between the factors. A simple way to detect interaction is to see the interaction plot given by the statistical software if the if the relationships are parallel then there is no interaction or very little interaction and if the plots are approaching each other or they have different slopes or they are even intersecting and crossing each other it shows that interaction is present and it has to be accounted for in the model. One variable at a time model will not be able to detect interaction and you have to go in for a statistical design of experiments. So you can see that things became a bit complicated I would not say complicated a bit more tricky even with two variables the interaction between two variables can slightly complicate matters. Now imagine the situation when you are having many variables some of them interacting and some of them are not interacting and then what will you do? You will have to resort to factorial design of experiments not only to identify the main effects but also to identify the interactions.  So we will see after a small break thanks for your attention.


Welcome to our talk. Today, we're going to be talking about a paradox of choice, how to pick an application definition that works for you. My name is Anusha Raghunathan and with me is Kevan Dhani and we're software engineers working at Intuit. Today, we'll be talking about some of the background of why we're even doing this talk and introduce you to the problems that we faced in our platform infrastructure team, supporting the millions of the customers as well as our thousands of developers that build application on top of our platform. And the meat of the problem, the paradox of choice, we will be talking about that and the solutions we came up with when we were faced with this problem. We'll finish up with results and takeaways. Intuit and our infrastructure at a glance. Intuit is a global financial tech company that provides a vast array of financial products and services built on an AI-driven expert platform. If you've used any of these products, TurboTax, QuickBooks, Mint, Credit Karma, Mailchimp, know that they've been running on our Kubernetes-powered infrastructure. We support about 900 plus developer teams consisting of 6,000 application developers and they deliver more than 2,000 services powered on 245 Kubernetes cluster running on 16,000 plus namespaces. Know that these are just average numbers. When we have our tax peak seasons, these numbers go really high. And one of the core tenets of our platform infrastructure team is to accelerate the developer velocity of our end developers. And we try to reduce friction where and when possible. Let's take a look at an average developer's CICD pipeline. An application developer commits code, which triggers build, which triggers test, typically unit and integration tests, and then gets deployed to a particular environment, whether it's QAL, performance staging or prod, and eventually gets monitored using an array of monitoring tools about the health of the application. Now, note that the application developer also has to work with a deployment repo for their deployment needs. What are the configurations for my applications running on various different environments? That's when they are wearing both the dev and the ops hat. Let's take a closer look at our deployment pipeline. The application developer uses customized based configs and overlay configs for their deployment repos. This gets picked up by Argo CD, which is our primary tool for deploying any of the deployment manifests into our end Kubernetes production. Clusters, and they're all isolated using namespaces. Hmm. So what could be a problem with that? First problem, Kubernetes and cloud complexities are exposed directly to our application developers. An application developer pretty routinely has to worry about how do I set my horizontal scaling? What are my min replicas and what could be an approximate max replica for my HPA? Is 15 seconds too low for a health check interval on my ALB ingress object? What's the right max unavailable for my pod disruption budget? What are good CPU and memory limits for my service? What quota should I set for my application's namespace? Now they are worrying not just about building their node or Java application, but they're worrying so much about their Kubernetes objects and their cloud complexities. Problem two, our application developers are very savvy and smart. Let's say they go over the first hurdle and understand Kubernetes and cloud, but Kubernetes deprecations are exposed to them. The application developer now has to understand that ingress v1 beta one, for example, can get very well deployed on a Kubernetes 121 cluster just fine, but when it comes to Kubernetes 122, it's going to break. Of course we don't let this happen, but at the same time, we have to work closely with our application developers to migrate them from a deprecated set of APIs to a new set of APIs. That works. Again, something that causes friction and reduces developer velocity. Problem three, lack of operational input and application definition. Although we expose cloud and Kubernetes complexity, operational input such as, hey, how do I enable HA in my service using an application definition? How can I enable Active Active DR? What if some of my services need external traffic and I need to specify that? How can I do all that as part of application specification? We don't provide that today. So these were the top three problems that were actually plaguing some of our developer velocity from moving ahead. What we did know was that we wanted a desired target state where application developer specifies the application intent. And that's it. We do some magic behind the scenes and that gets deployed into a Kubernetes cluster with the right cloud resources. So we knew what the problem was and we knew what the target state is and we started looking at the possibilities of how to solve this problem and we ran into a paradox of choice. If you were Neo in the matrix, you were given two pills, a blue pill and a red pill. Easy choice, huh? We were given a million pills. Both CNCF landscape as well as our own homegrown and everything in between, there were too many options to actually go figure out a solution for this problem. This is a snapshot of the CNCF landscape that talks about application definition and image build. So we wanted to take a methodical approach in understanding how we can actually solve the problem with a lot of these existing tools that would also fit into a tool chain and our use cases. But before we did all that, let's take a look at what the app specs should look like. Our main requirements for the app spec were it needed to be application centric. There shouldn't be any leakage of cloud or Kubernetes resources into the application specification. And it had to help deployment as well as the operational needs of the application. And this was a pretty straightforward choice. The two choices we had were the open application specification model, which suited our needs pretty well. Or we could go with the templating style model where you had to provide a bunch of input parameters, but there was also a lot of abstraction leaked into the application spec. So it was easy for us to go with an application OAM style specification. So to give a very high level overview, you would take a simplified application specification like shown here in the black box, which had the intent of the developer saying, hey, this is the image that I want. And here is my sizing needs both horizontal and vertical. And I had a way to override these traits depending on my environment. And we would be able to generate the Kubernetes resources. So that was the criteria. With that, I'm going to hand it over to Kevin to talk about the solutions. Test, test. All right. Morning. Yeah. So thanks, Anusha, for handing off. We'll talk about the solutions that we found kind of matched our application criteria. There were four choices actually we found that met with the OAM style specification or be able to allow us to generate resources and deploy those resources. And conveniently, they were kind of bucketed into two categories. One is client utilities versus the control plane utilities. The way we actually wanted to dive into these and evaluate them was POC. Then we took our teams and we divided our team up into the four teams that we're going to basically specify what they wanted to find out about each of these. When we get into the breakdown, the utilities, we're going to go over them really quickly. If you're not familiar with Helm, it's a package Kubernetes, packages Kubernetes configs as charts and provides deployment application lifecycle. It's also a CNCF graduated project. Customize KRM functions. This is client side functions that operate on Kubernetes resource model. KRM configuration supports generators, transformers, validators and customized plugins. These are both client utilities that can generate resources. And just to get an overview, we have a CD pipeline for client utilities. It's very generic where we take the application spec. That's the intent of the user. We use this client utility to generate Kubernetes resources. Those resources would be synced to GitHub and our normal GitOps pipeline here using our CD. When we went through the control plane, just a high level, the open source, we looked at cross plane. So it's open source control plate framework. It helps orchestrate applications and infrastructure. We also looked at Qvilla. It's an open source application engine based on Kubernetes and OAM. And just generically, this is how the CD pipeline would look for either of these control planes where we take the application definition. That's checked in to get the Argo CD would sync that to a control plane. The control plane would then be responsible for continuous delivery and generating the resources to the end cluster. So just dive right through to the results. There is a trigger warning here if you use any of these tools. These are our take on the usage of these tools for our application needs. And we did a comparison. So one of the, we'll just go through all of these as pros and cons. So using Helm for our application OAM like application spec, we found that it was GitOps compatible. It definitely had an active community. There were definitely a couple cons with this chart of charts. If you're ever familiar with Helm, you can get into an explosion of charts where basically how you want to size your chart or your packaging and you might have some packaging with those. There's also a lot of template logic. Both of the logic in the core business logic you'd have to write would be kind of applied at the template layer. Customized care and functions. Similarly had pros and cons. The pros is GitOps compatible. It was also OAM style compatible, which fits our needs. And it has a logical template methodology. So they don't actually do templates and customize. You can use templates with the care and functions, but in a limited way. The cons are it's an alpha stat status. And it's also has very sparse documentation. When we get to the control planes, when we look at cross plane, we found a lot to like. So there was OAM compatible. It was extensible as a multi cloud. The cons are that we have to run these multiple controllers now. I think this is an important point. There's lackluster argocd as of what we looked at this. There was argocd compatibility. However, the state of the CRDs were not actually reflecting the actual state that we were deploying. And there was a limited features. So what we wanted really was application generation. Tools and more, this is geared toward more infrastructure tooling. So when we looked at Kubevilla as well, lots of pros. OAM compatible, extensible, multi cloud. The cons are very similar multiple controllers. So that's more of an operational burden for us. It's kind of an all or nothing. You get to deploy Kubevilla on OAM their way. And actually one of the things that is sort of sold as a pro, as pro would be a con for us is the Q-templating DSL. So implementing all of our templates as this other language would actually be a learning curve for us. So then after we got back and we basically presented all these results, we didn't actually have a clear winner. We had multiple camps of developers. We found a lot to like in each of the solutions. And all of the developers came back with advocacy for that solution that they PLC'd. So we really needed a way to find a better way of getting results and comparing them between each other. So what we want to do is more of a data based approach, qualitative analysis or quantitative analysis. And we want to look at each of these very, from an engineering data perspective. And so what we did is we had presented all these PLCs to our developer groups and we basically gave them this scorecard.  So the weights are kind of important to, specifically our importance, what we find important about each criteria. And we have the four results here. And just from the averages, this is how we ended up. Now that's not going to show that well, but this is where we ended up. And you see on the slide was like, what's happening here in this chart is there's a very clear to kind of be between the client versus the control plane. And clients clearly won, they stood out. Why? Well, you have to look a little bit at each of these criteria that we had and dig in a little bit deeper here. The first thing was the learning curve. As mentioned, there's controllers with multiple controllers for control planes. There's either DSL templating logic, something that would make it hard to do. There was also effort involved. That goes also into the templates. The technical fit was compatibility, whether it's getups compatible or OEM compatible, the upper ability, scalability, those are the client versus control plane. You're going to run one binary versus many controllers. The flexibility was down to the code versus DSL. And all of this came to show off that why did Qvilla and cross-plane show as not going to work for us? So we moved to actually the top two winners here. And we really needed to figure out, so now at least we have two potential solutions here. And if we just take the raw numbers, we would go with Helm. But we really wanted to analyze in a qualitative way why we would need to choose one solution over the other. So again, we assembled the team and we basically said, what's the learning curve for learning Helm and customize? The whole team is familiar with Helm. They're familiar with customize. There is no learning curve there. How much effort is there? This is really what's kind of the key difference here. For us, we already are adopted customize. Our entire control plane and CICD pipeline is all based off of customize today. Re-doing all of the templates in Helm, that would be a very large effort. Some other efforts would be kind of adopting Helm as packages and, you know, proselytizing that across the company. The OAM compatibility, when you get to Helm, for our needs, the style of the application definition that we have is more OAM-like. We want to be able to just specify a CRD like definition and have that processed. And doing that in Helm requires a lot of helper functions. There is no difference between the scalability here. They're both client-side tools. They're both flexible. They do have code, demos and templates. I think there is a key difference between customize and Helm when we have community and documentation. Customized CRM is pretty new and still lacking in community and documentation. But if you recall from the previous slides that we had, more weight towards the top of the charts here. The turn to market, the learning curve, how much effort there is. This is how we ended up choosing CRM. Customized CRM. So why did we choose it? Because it supports transformers, generators, validators, as plugins. Supports our declarative spec that we need. It's also get-off-script-paddle. That's very important for us. We want one source of truth.  So with that, I'll head it back to Anusha with the demo. Thanks, Kevin. We're going to see a quick demo. A short prayer to the demo gods. All right. So this demo is on our customized CRM plugin. And we're going to have a sample app YAML. Note that this is our only interface to our developer. The developer specifies the name of the image and also adds in some hints about the traits. In this case, we're going to focus on the vertical size of this service. And note that the base actually has a vertical size of small. And the override for the QA environment has a vertical size of medium. So once we get that information, we're going to actually use customize build. That's efficient. So what we just did there was build, use customize build and specify that we want to use our OAM customized plugin to build that demo app that we just saw. So what it spits out is actually a whole bunch of Kubernetes native manifests from that simple app YAML file using generators and transformers. So you see that there are several services that we have behind the scenes. We have analysis templates from Argo Proj. We have analysis, we have, sorry, a roll out from Argo Proj, which is actually a deployment replacement in our production environments. And finally, we have an HPA object that gets vented out by default to all our applications. Now let's actually copy this over to our deployment repo. Typically this is done as part of the deployment pipeline, but since we don't have a full-fledged pipeline, I'm just going to do it manually by copying it over to the deployment repo. Let's go to our deployment repo here. There's no diff because it's just the same. Let's go to our Argo CD instance that's running locally on my cluster here. So this Argo CD app is pointing to the URL that we just saw. That was just a clone of this repo and pointing to the manifest directory. Now let's actually make it interesting and say that the application developer decided to change the size of the application vertical size in QAL. So let's go from a medium to a large. Use our KRM plugin to generate a new manifest. Notice that there is a diff in the size. It's increased to a large. Now let's come at this and push it. Now let's go to Argo CD. Notice that we can see that Argo CD detects that it is out of sync. You notice that the diff shows that now we've moved to a vertical size that corresponds to a large. Let's go ahead and sync it and sync is in progress. So the essence of this demo is that the application developer only has to work with the app.yaml file that we just saw and make changes as far as vertical or horizontal traits, which is one example we have a lot more traits that can be added. And the rest of it is taken care of by the deployment pipeline. The complexities of the cloud and Kubernetes are all completely abstracted from the application developer and the deployment is going fine. That concludes the demo. So what are the takeaways? Well, just like us, you might be faced with a paradox of choice when it comes to cloud native solutions, whether it's your own homegrown solutions or something in open source or something in the CNCF landscape. Don't get overwhelmed using a methodical data driven approach can actually lead you to pick a solution that works for you. And abstracting application developers from complexities of Kubernetes is doable and we use a client side solution because we are a huge GitOps based shop at introvert and we wanted to pick a GitOps based solution that was friendly for us and worked with our existing tool chain. So KRM plugins are an efficient way to do this. In fact, we're stay tuned. We're going to write a blog post about more details on the KRM plugin that we implemented and what we demoed today. If you have a non-GitOps based solution, then you could go in for a control plane solution as well like Kubella or Cosplane. And the velocity and innovation of platform teams also increases with application abstraction. When we started the talk, we talked a lot about velocity of the developers, but using an application abstraction also helps the platform teams revise much faster because then they don't have to worry about writing technical service bulletins to make sure that the developer teams are migrating to a new solution. We can roll out a new service mesh or a new CRI or CSI without having to expose the developers directly to it. So speed to benefit is what matters to us. Know what matters to your organization and work on finding the right solution for you. Thank you. All right, I think we have four minutes left, so there's time for questions. If there's any questions, this is a really large room. I'm going to run.  Hello. Yes. Did you try making your own operator as an alternative? That's what we ended up doing when we had the same problem because so you just make an operator which spawns all the other resources. Do we try operator? Is that what your question was? If we tried an operator approach? Yeah, we evaluated stuff we did to show you. We ended up with, you know, the cross-plane, the control, Qvilla were more mature than just our own controller. We already run our controllers as well. So that made us a little bit part of the story you don't know is we are very mature in our pipeline right now. Actually, we kind of be too already in our pipeline and everything's in place. We have controllers running for different reasons for infrastructure or whatever. Adding another controller is just a burden, an operational burden for us. That's why. Hi. Maybe I misunderstood when you were talking about polling your organization earlier.  Platform engineers. That was the platform engineers that you polled about their selection. What kind of specification would it have been? It's the same from the developer's perspective. It was the same regardless, right? Correct. Hello. Quick question on alternatives. Like on the Helm side, did you look at Helm file? And on the other side, did you look at KPT? I think the alternatives were the Helm file or KPT. We are aware of those for sure. I think some of those didn't raise all the way up to our evaluation stage. Like we did look at them and just have a dock. Like, okay, is this a useful thing for... I think what we really came down to is we kind of narrowed down our site into OAM-like specification and we want that specification. When you do that, other things, you're kind of adopting their approach. So do you expect the integration between Argo and Crossplane to improve? Are you aware of any efforts there? The answer is yes. I think there's a talk great for Crossplane and Argo today. I think one of the being, yeah, the lackluster part of it was we were deploying infrastructure stuff and app stuff as part of the POC and we just saw that all it says was complete, right? When it's actually in an error state or it's not in a fully upstate. So hopefully those approve and we're not really not running Crossplane for something, but I think the main point was it was more than we wanted. It was more focused on infrastructure. You actually might want to run Crossplane with Kubella or something like that. We have time for one last question. Hi. So my question is did you have a situation where the OAM kind of didn't fit, you know, a use case that you needed and you needed to expand out of that and if you did have that, how did you approach that? Kind of customizing that in your own way. So we have an OAM-style specification that we have implemented. So the plugin, the customized plugin actually is extensible enough to be able to add our own traits like for example the sizing traits that we have environment related traits and so on. So it's very loosely defined. So it works for our needs right now and the implementation is all in the plugin. All right. Thank you for joining the session and I'll now share with Kevin one last round of applause.


Hello everybody, I am Marga Farre, I'm talking to you from Madrid, from Spain, one of the countries more affected by this epidemic, as you may know. So I want to give you an update of what is happening here, at this, how the debates are going about. As you know also, Spain is absolutely affected by this epidemic, and now we have more than 23,000 death people. That is like we are declining the famous course. So all the debates that we are talking about is about after this coronavirus crisis. So what is going to be the next steps we must have as a country? So there are three debates I think that we must face, or the people is talking about. The first is unemployment. As you know, Spain is a country based absolutely on services and tourism and construction sector. And these are sectors absolutely affected by the crisis. So we expect an unemployment rate more than 20%, which is really a huge amount of people unemployed. It's a weak labor market we have based on temporary workers and also on precarious workers. And our worry is how can we face the situation where the people could be more poor, or poorer, sorry, than we were before this crisis. The second debate is about the reconstruction. As you know the countries in Europe, how can we relocate companies in order not to be so dependent on the industries of the countries? And also how can we change the European social or at least Spanish social model? So that is a huge debate we have. And the third one of course is about Europe. So I would like to say to you that from Spain we don't ask solidarity or of course any kind of charity. It's just only an idea of if we belong to Europe, how can we develop some kind of cooperation among all of us if not what is Europe about? The idea is like if we can exit of this crisis together or not and of course in favor of the workers and the poor and the popular classes, how to do that? So I think the left is pushing hard to reform the government and also from the social sector pushing to put an end to the austerity measures and to leave our countries to use the huge amount of money that is in the financial market today in order to save lives and to protect people. So that is the debate we are having here today and just to wish that we are going to an extra step to how we can exit of this crisis in a better condition than we had before. So I send you a huge hug and I really expect all of you are OK. Bye.


Hi, I'm Geoff Watts and I'm really excited to tell you about my new on-demand courses for agile leaders like you. For the last 20 years I've been coaching organisations and their leaders from small startups to FTSE 100 companies in various industries to change their organisational culture in order to become more responsive and resilient. All of these companies have come face to face with the reality that their world of work has changed and continues to change. It's become much more complex, more volatile, more competitive and more uncertain than ever before and their leaders have recognised that they need to work differently in order to survive and thrive. Now while all of these companies are completely unique and you can't hope to simply copy and paste their approaches onto your own unique circumstances or indeed buy an off-the-shelf one-size-fits-all enterprise agile framework, the good news is that you can learn from there and my experiences to shortcut a lot of the pain that you otherwise might go through. In these lessons for agile leaders courses I've gathered together all of the common challenges and pitfalls that you can expect to encounter on your journey and I'm going to share with you practical tips and practices that I've taught all of these companies. By taking these courses you'll be able to get ahead of the curve and get ahead start when creating a more agile and resilient organisation. Overall there's over 20 modules of real world experience created to help you effectively empower your people, overcome resistance, adopt an agile approach that balances autonomy and consistency, agility and strategy and most importantly become a more effective leader of your organisation in this new, fast-changing, increasingly complex and uncertain world of work. Leading organisational change is admittedly daunting at the best of times but when we're talking about a world full of unknown unknowns where you've barely got to grips with the current state of affairs before they've changed again you can be forgiven for thinking it's a poison chalice but I can help you. These lessons for agile leaders like you boil down all of the common challenges that you can expect, why you can expect them and more importantly what you can do about them and make it a rewarding and enjoyable journey as well.