told me that the key to a successful session is quoting Kelsey Hightower. Usage matures products faster than hype does. That's definitely true. One more quote just to be sure this session is a success. You haven't mastered the tool until you understand when it should not be used. Last quote for now.  Luckily Kelsey said nothing about dumb answers, so you are welcome to ask me questions later on Slack. My name is Regina and I am a huge Github spam. I am also a cloud platform team leading Bank of Olim. Ever since I was a little child I was the only female somewhere. The only girl in class, one of the few women lecturing in DevOps areas and lately I have become one of the few female open source contributors. This is a shout out to you ladies to participate as well. Me and my husband have two boys, which makes me outnumbered in my family as well. This session is about a misconfiguration while working with Customize. How many people are watching this session to hear about a problem that happened to someone else? I will tell you about what caused it, how we fixed it, what we learned from it, and how we improved our methodology as a result. I will be focusing on GitOps for cluster management, especially on managing Argo City Config with GitOps. I will not be going into GitOps for business apps. What I will talk about today is from the perspective of a cluster admin and from the operations perspective, but the insights and the methodology are relevant also for managing business applications configuration. The background, GitOps manage desired states that we have. We manage multiple OpenShift clusters. We have clusters with Argo CD that manage other clusters. We manage multiple tools, products per cluster. And we manage multiple OpenShift operators per cluster. The tools that we manage, some examples are Prometheus, Grafana, Loki, VPA, the Vertical Pod Autoscaler, Litmus. Some clusters have cross-plane installed, some have Argo CD installed, and many, many more tools. We're interested in configuring the same product differently per cluster. We're dealing with lots of OpenShift-specific configuration. We upgrade those tools frequently. And many tools that we manage are Kubernetes plain manifests. Now, you can understand that we manage a lot of stuff, a lot of tools and products. And when we talk about GitOps, the repo structure is key. So let's see what was our initial repo structure and how it evolved. You see that there are multiple Argo CD instances that each manages multiple clusters, much more than the two in this chart. At the beginning, we duplicated that architecture to our Git repos, meaning we had one repo for the configuration of Argo CD's and of the clusters they live in. And each such control-plane cluster desired state was in a separate branch. And we had another repo for the desired state of the cluster's Argo CD managers, also with a branch per cluster. We had a many repo with a many branch, a lot of configuration duplication. When a new product needed to be added to our desired state, people didn't know where to commit it to. This structure was a mess, and we decided to move to a mono repo, mono branch, in order to avoid the duplication. And so that there is one place to look at to get the picture of the overall desired state. The branch per cluster or per environment is an anti-pattern. Don't do that. Never, ever. You can read more at Costi's Capiloni's blog, mono branch for all the clusters configuration. But you hopefully remember that we need to manage many products that come as play manifests, and we are focused on configuring the same product on many clusters differently. One option was to do that with Helm. We'll have heard of Helm and use it. Let's recap what Helm is. Helm is a Kubernetes package manager. It is also a CNCF-graduated project. This means that it is considered to be stable, widely adopted, and production ready, automates software packages' life cycles, such as installing, upgrading, and removing those packages. It uses a template engine, and it enables us sharing and reusing software packages that are called Helm charts. The chart consists of a template directory, a values YAML file with the parameters to be used with the template, and a chart YAML file, which describes the chart. The templates are YAML files with Go template and template functions. For configuring the same product on many clusters differently was to use Customize. I'm sure that many have heard about Customize and use it as well.  My husband asked me, why is Customize misspelled with a K? So let's quickly cover what Customize is and what problem does it solve. A native Kubernetes configuration management tool. And this is why it is spelled with a K. There is a big community around Customize and it is even bundled into Cube CDL. It uses declarative approach to Kubernetes configuration. It enables updating configuration without working by traversing a Kubernetes manifest to add, remove, or update configuration. It uses layering over base settings to selectively overwrite default settings without changing original files and avoids configuration duplication. It enables writing shared configuration once and then overriding it per cluster or per environment. The common configuration is referred to as base and then the overrides of this common configuration are referred to as overlays. Overlays are batches to base manifests or additional manifests. The final Kubernetes manifests are produced upon running Customize build. This command processes the base and overlay files, which are basically fragments of the configuration and produces the final plain Kubernetes manifests. Why we chose Customize? You remember that we managed the configuration of many products of many OpenShift clusters, many OpenShift operators, we need to be able to upgrade those products frequently. So we do not want to fall behind with old versions. We also need to add OpenShift specific configuration to many of those products. Manifest change in those products is externally driven and frequent, as opposed to manifests of self-developed business apps, the change in which is internally driven. We are focused on configuring the same tool on multiple clusters differently. We are not focused on versioned software artifacts in this case. We need a flexible configuration without changing original manifests, because if we change the original manifests, upgrades to the product would get us to merge hell and would be slow and error prone. So keeping the original product manifests separate and untouched is crucial. Customize provides clear separation between what is common and what is overriding. So the original manifests of the product naturally fit into Customize base, and then our configuration changes naturally fit into Customize overlays. Wrapping those tools in Helm chart seemed an overhead. If we choose Helm, we would need to write the Helm templates ourselves for each such product or configuration. Now you remember that Helm template is a combination of Kubernetes manifest fragments and Go template code. So when such tool has a new version that has changed manifests, the merge with our self-grown Helm chart would be time consuming. Because with Helm we wouldn't be able to avoid changing the original product manifests. The technology we use should be always working for us and not the other way around. And in the case of third-party products and tools that come as plain manifests, writing our own Helm chart for each such tool seemed like working for the technology. However, using Customize for that looked like the technology is working for us. We have started with Argo CD long before there was a stable Helm chart. Migrating to Helm chart didn't seem worth the effort until some point in time, but we're constantly looking at it and might migrate in the near future. I want to stress that we are only talking about how we manage plain manifests and we manage tools and products that come as a Helm chart with Helm, obviously. To that we have covered why we chose Customize. Let's see how we managed our Argo CD configuration. We have Argo CD running on different clusters. The Argo CD configuration is self-managed by Argo CD. We managed Argo CD config maps with Customize. There is a part of the Argo CD config map that is common to all the Argo CDs and there are parts that are relevant for only specific Argo CDs and they are different. The config map part that is common to all Argo CDs isn't base and the parts that are specific to each Argo CD are in overlays. It's a visualization of the repo structure. This is the shared part and those are the specific parts that override the shared one using patch. Control planes are clusters where Argo CD is installed. This is Argo CD config map, part in Customize base. We use OpenShift routes and Argo CD needs to be instructed about some route fields to be ignored during diffing. The reason is that the route host field is automatically populated by OpenShift at runtime. But those details are not significant and what is important is that this ignore differences section is responsible for the routes to be displayed as synced in Argo CD. A spoiler. The resource customization section, this one, is a multi-line YAML. It is important for later. This is the specific part. The Argo CD URL is unique to each Argo CD and so is the DEX config. I'll instruct Customize to patch the base manifests with the contents of the overlay manifest using strategic merge. There are also other patch strategies that can be used. Now that you have all the background details, I will tell you what happened on November 13. November 13 is the World Kindness Day and what better day is there to break stuff in production and other environments? So what happened on 13th of November? We received a requirement to implement an Argo CD custom health check for Ingress. The requirement was relevant for a particular Argo CD and so the implementation was to add a custom health check to the overlay Argo CD config map. You can see the resource customization section was added to the overlay with a health section for Ingress. Look at the base Argo CD config map again. The ones who are experienced with Customize can probably spot the problem. The others would have to stay in suspense for a little while. Those are the tests I have done before merging. I ran Customize, Build Results and verified that the word Ingress was there.  All those tests were successful and so I confidently clicked merge. After the merge I verified that the Ingress resource now looked healthy and so the task was done. This is what happened five minutes later. We had a gazillion tickets opened. The business app CD was failing. Most of the apps in Argo CD were out of sync. All the route resources in Argo CD were out of sync. And we now have a monorabel. So the first thing we do when we troubleshoot a problem is looking at the commit history. So I looked at the commit history and surprisingly the last commit was mine. Now fixing the problem with Monoribo is as easy as reverting the commit. Now we will see what the problem was, what impact did it have and what did I learn from that. This slide is for those who are in suspense till now. On the left there is the expected final Argo CD manifest and on the right is the actual final Argo CD manifest. As you can see the route ignore differences section is gone. But I didn't delete this section. So what happened? Well the problem was the multi-line YAML which got entirely replaced by patching the base with what was in the overlay. Let's see what impact this problem had. The actual application deployment was successful. But the Argo CD status was out of sync.  This sync state was never reached and so the status of the CD pipelines was failed. First lesson learned was about overlaying file contents. I'm talking about having a part of a file in base and a part of the file in overlays as opposed to having a copy of the whole file in overlays. It requires using a patch which is sometimes non-driven. The common use case for that is the deployment image overlay. The question is whether other use cases are a bad practice or a best practice. Pros are that it eliminates duplication and it eliminates the change in one place and I forgot to change in another place problem. The cons are that the impact of an overlay change is not obvious. So the conclusion is use it with care. The learn number two, patching multi-line YAML. Multi-line value can be a YAML structure itself. But the multi-line value is a string, it is not a map. And so the so-called keys in the multi-line value are not real keys. And so patching such a key in a multi-line value will result in the entire multi-line value being replaced. So the conclusion is if multi-line values should be patched consider getting the file out of base and keeping it in overlays in its entirety or use helm. Lesson learned number three, understanding the impact of a change. So our goal now was to understand how a change in the overlay impacts the overall configuration. When I ran Cube CTL dry run and Argo CD dry run I made sure that my change was valid from Kubernetes and Argo CD point of view, but this was not enough. The missing part of the puzzle is the diff between the final new and the final existing configuration. This is not a diff in a customized file content that I have changed. This is a diff in the final Kubernetes manifests. I will explain how we added the missing part soon. Had I seen this diff in addition to the diff of the file I changed the problem would have probably not happened and I would have been telling you another story today or just going to the beach instead. Lesson learned number four, Git repo protection. Mono repo. The GitOps desired state of all the cluster we managed is in this repo and this includes monitoring tools, cluster configuration, storage operators, and many, many more. Business apps GitOps desired state is managed separately and they are out of scope for this session. Rebo is mostly customized based. A change can affect a production cluster. A change can affect multiple clusters at once. Having a Mono repo is very convenient for cluster configuration because it enables us using tools like customized to efficiently manage what is common and what is different from multiple clusters. Changes can be quickly introduced to multiple clusters at once and operation can be lightning fast. However, breaking multiple clusters at once can be lightning fast as well. With great power comes great responsibility so this repo must be protected well. So we realized that we needed a merge request approval. Merge request is a methodology of GitLab because this is what we use. It's basically very similar or identical to pull request at GitHub. We realized that we need automated validation and pretesting of configuration changes. Now merge request approvals and automatic tests sound familiar, right? This is what people do for business application development and CI. Well, managing Kubernetes infrastructure and operations has to follow the same path, maybe even more strictly. Especially in a world of Kubernetes is a control plane for everything because then you end up managing not only the cluster as the cluster but many other remote resources as well. The same concepts of working with GitLab, GitHub issues, merge requests, branch methodology, CI tests apply here as well. Just like our code needs to be tested properly and sometimes reviewed, our configuration also needs to be tested thoroughly and sometimes reviewed. This is the last quote of Kelsey Hightower I promise. Automation is the serialization of understanding. So let's take a look at how we serialized the insights from that incident with regard to Git repo protection combined with understanding the effect of a change in customized configuration. Here are the steps in our pipeline. Cube CTL dry run, Argo CD dry run. Changes in base now require an approval before merge. Changes in control plane cluster overlay also require an approval before merge. Changes in critical cluster overlays acquire approval before merge as well. Other changes are auto approved as they're considered non-critical. Regarding the merge request approvals, they will never replace automatic tests but having a review from a colleague is a great way to spot problems otherwise overlooked. Unfortunately, waiting for a review is also a great way to slow down or even block operations. So we chose something in the middle. Widescope changes like touching customer's base and changes to critical clusters require a second eye while changes on a playground cluster can happen without waiting for anyone and can be just merged. This is an example of a critical cluster change merge request. You can see the comment that was automatically added by the merge request pipeline saying that a critical cluster overlay was changed and that a human approval is required. This is the comment. This is an example of a customer's base change merge request. You can see the comment that was automatically added by the merge request pipeline saying that base was changed and that a human approval is required. However, this merge request has no wide scope changes or changes to a critical cluster and so it was automatically approved by the merge request pipeline. It can now be merged by the author of that merge request. But wait, what about the impact of a change in customize? This pipeline is not complete without handling this impact. Do you remember Customize Build I mentioned earlier? Customize Build produces the final manifests of the result of Customize Build using the changes section of the merge request. So the solution was to commit the Customize Build result file along with the changes themselves. So the merge request pipeline requires the up-to-date Customize Build result file to be committed along with the changes. This is an example of a failed merge request pipeline. Now look at the comment on the merge request that was created automatically. You have changed Customize Configuration without committing the Customize Build result. The impact of your change is not determinable. This is a visualization of the implementation. This is a familiar merge request changes view. You can see the small green part that was added and then the huge red part which got overwritten. And now the impact is clear. A change similar to what I have performed on November 13 must not have been merged. The aftermath of a change is now visible and easily comparable using the standard merge request diff mechanism. Basically now after all the tests in the merge request pipeline have passed, I can also see what exactly I'm about to change and I can make a decision, whether this is what was intended or a mistake. And then depending on the scope of my change, whether it affects multiple cluster or a critical cluster or just a playground cluster nobody cares about, I will either be able to merge immediately or be able to merge only after an approving review of a colleague. And so the overall repo protection is improved. This template engine and a package manager is constructing Kubernetes manifest from fragments of configuration. Code and value files. And understanding the impact of a change in any of those can be equally challenging and is equally important. It can follow the same methodology I have described earlier. You can read more in Costis Capilloni's blog. There are additional aspects covered in that article. To those who survived this far, understanding the impact of a configuration change is important. Configuration testing is just as important as code testing. Merge request pipelines for configuration are just as relevant as merge request pipelines for code. Choose the right tool for the job is important. The technology must work for us and not vice versa. Don't be afraid of making mistakes. I want to dedicate this slide to my kids. They attended my session when I gave this talk live on Q-Day. Live on Q-Day. They are 9 and 11 years old and they do not know what Kubernetes or GitOps is yet. So I added QR codes of jokes, riddles, and Dr. Seuss wisdom in some of the slides of my live session just for them. We are hiring so if you want to come and break more stuff with us, learn from that and then improve what we're doing, please contact me on LinkedIn or on CNCF Slag workspace. Thank you so much.


Hello everyone, welcome to theCUBE's coverage of High Performance 2023, covering everything in HPC, machine learning, AI, high performance analytics, quantum computing, part of ISC 2023 event coverage. This segment is Dell, NVIDIA, and the financial service. We've got three great guests here on the power panel, Andrew Lou, product manager with Dell, and probably Rob Morthy, who's the customer partner developer relationship at NVIDIA. Dell and NVIDIA power panel, Peter Neves, president and stack securities technology analysis center. Gentlemen, thanks for coming on. This is a super power panel. Thanks for joining me. Thanks John. So first question, how is HPC being used in financial services? Which use cases are you guys seeing most interesting right now in the for financial companies? Currently we see actually these areas being used for important applications at the credit valuation adjustment. So these areas are being used for risk pricing and discovery in capital markets. And that's what we call as HPC, multitative finance in parlance. And customers are doing large-scale simulations, Monte Carlo simulations, as well as doing risk valuations in real time. So they can get ahead of competition. And where we are seeing this action go is people want to do more of this combined with artificial intelligence and other technologies. For example, they want to use large language models and then use it for trading signals and do their own algos in quantitative finance. And we all know high-performance trading has always been the edge. Every millisecond, nanosecond counts. Now AI is coming into the mix. We're seeing a lot of action there. We're going to come back to that. Peter, what's stack? Tell us what the company you guys do. You're in the middle of all the action. What do you guys do? How was it started? Yeah, stack is main goal is to improve technology discovery and assessment for the finance industry. And we do that in two ways. We do that through dialogue, but we also do it through research. And our research is guided by the Stack Benchmark Council, which is over 50 of the leading vendors in technology, as well as 400-plus financial firms. And the way stack got started was it was really expensive for people to do technology evaluation. In fact, when stack was started, I was an early member. I was CTO at a trading firm. I had all of my high-priced engineers doing nothing but evaluating technology and doing these bake-offs. And the idea here is we all have similar workloads. We all have these workloads that are exactly what we do on a daily basis or a great proxy for that. So, stack came along and helped bring together the Benchmark Council to define those workloads so that we could do apples-to-apples comparisons of different technology stacks to see how they solve the problems, how quickly they do it, how much throughput they can get done, and how efficiently they do it. And now, 15 years later, that's what we do in a variety of areas, including HPC and AI. Real quick, while we have you segue there on the market, what are you seeing now? Obviously, the edge is so important in getting that technical edge and those benchmarks. How frequent is it being done? What are some of the current state-of-the-art criteria you're seeing out in the market right now with all this change? Yeah, I think that the most important thing that we're seeing right now is analytics, which has always been considered not real-time, not part of trading, but more back-testing, that what we see for HPC, right, historical research, back-testing, the things that's outside of the trade critical path is starting to merge with that trade critical path. People need to do more real-time HPC, more real-time analysis of data. And so, where we used to talk in nanoseconds for electronic trading, and we would talk milliseconds or seconds for analytics, those are merging together, and it's not because the real time's getting slower, it's because that analytics has to get faster and faster and faster. You nailed it. This is so exciting. That's why I love this power panel, because you got packets in the old days, well, old school, moving packets around as fast as possible. Now it's data and the right data mixing together. We're going to get into that. Andrew, Dell's been a player in financial services, and you could go to any servers. You see them all with Dell's. You guys have been in this game for a while. What are you guys doing now for financial companies? What are successes that you see right now? Yeah, we are. We are heavily investing in financial services industry. We've been working closely with NVIDIA and Stack and Peter and probably have been great partners to us. We've been listening to our customers, discovering what they want in their products, in their solutions. And we've been putting together NVIDIA products like the A100 GPU along with PowerEdge servers like the XE8545 that we launched last year in November called the Dell Validity Design for Risk Assessment. And that was, and Peter can explain more on this, that was designed to measure system performance against Monte Carlo simulations. So that system itself was designed by our engineers, put together, tested to help clients have a baseline understanding of where our systems function best.  If over the past five years, we've been doing theCUBE for 13 years, I can tell you I've seen all the stories, go back five years or so, all the marketing people, it's all about the solutions, not about the hardware, and don't talk speeds and fees, don't confuse anyone. I'll tell you right now, people want to know what the speeds and feeds are because it's impacting the solutions. All right, you're seeing more discussions around silicon chips, performance. Now they want to know, is it performance as a baseline? Right, this is now the new thing. Can, no one wants to run their app on slower or their workload on slower infrastructure. So you guys done a lot of benchmark testing right now with Dell. What's the current state of the art in terms of benchmark standards? What are people need to look at? What's your take on this? Yeah, yeah, I definitely agree with you on the, it is about speeds and feeds, it is about speed. I witnessed that over the last five years too, but I can tell you in capital markets and financial services, that never went away. Speed and throughput is the ante. It's table stakes for everything that's done in capital markets. And what we've seen, and it's been great to see Dell over the last couple of years and Nvidia as well, really step up and give financial firms the data they need to make decisions they need to make on the solutions they're looking to procure. In fact, the things that they care the most about what we see a lot of financial firms asking about is how fast can it crank through the data I have to do? How fast can it do these workloads? Because even though they're off real-time, even if they're offline, they still have to get done between close and open. And between close and open isn't overnight, it's an hour, you got an hour.  And they also want to know can I do it efficiently because the data center space I'm in is very limited and the power is very limited. And I think that's one of the things that Dell was pleased about with their last audit. Their last audit in Stack A2, which is our HPC benchmark, they had the highest space efficiency, they had the fastest cold times in the baseline Greek benchmarks, and they had the fastest cold time in the large Greek benchmarks. And just so that people don't get confused on what cold means, there's kind of two ways to do HPC in a cluster. One is you have all of your code already loaded and it's already primed and ready to go. That's hot. And that makes sense if you're only doing one task with your system. But if you're doing many tasks with your system and constantly changing out what task that is, you're starting from a place of cold. And so that's where Dell shined in their last benchmark, right, was with the cold times and with the space efficiency. I love the whole closing thing with, talk about the closing that time with Wall Street. I talked to a CIO and they said, every minute's worth millions of dollars, but I can get an extra minute. That's how important time is. This is really cool. Right. Talk about the NVIDIA part of this because you guys have a big presence obviously in the marketplace with the finance, the GPUs are hot, everyone wants NVIDIA, your software is booming. What are you guys doing right now with Dell on this? Talk about your solution when your relationship with Dell. Thank you, John. NVIDIA is in the top 10 in the US stock market by market cap, right? This is all due to AI leadership, right? And we have a close partnership with Dell where we are launching these products along with Dell. And we are essentially positioned for success, right? We play in three areas. Quantitative finance, as Peter mentioned, where we do all the risk calculations, right? The clients need real workloads and they vary. And the benchmarks provide a way where the clients can do actual workloads with the credit values in this adjustment CVS. In addition to that, we also have like an area called data movement, ETL, extract platform float, so where NVIDIA sets up the space along with Dell. And obviously, right? We live in this age of AI where everybody has heard about GPD for chat. So NVIDIA provides such solutions in the AI market.  And we have been reading the pack here, right? Where every client, every financial institution, hedge fund, bank is building the large language models. And these are essentially models for their own internal purposes where they're using this for building trading signals, right? So they're actually using it to generate trading signals at the speed of light, right? You need to know when Silicon Valley Bank is not doing well. First Republic Bank is not doing well, right? We have had like big companies like Credit Suisse hit with market volatility runs. And NVIDIA just provides that AI layer, right? With which you can do all the three solutions. You can do quantitative finance. You can do, move your data machine learning and you can do neural nets, right? Your own brain with chat GPD. And essentially, right? We kill all these three workloads where the client has the maximum bank for the buck. And right, who would not want such a hardware, right? Where Dell along with NVIDIA provides a solution which can be used for both algo trading, capital markets, and as well as large language models and generative AI. And that's where we find ourselves, right? We find ourselves at this intersection where clients are building all these solutions. They're mixing all these models and essentially building the platform of the future. This is great. This is the conversation we're getting to the next gen advancements in the industry, the financial industry. Peter, you mentioned this earlier about, you know, packets was the first real innovation speed of the packet, speed of light, closer to the transactional value on trading, high frequency trading, all that generation happened. Now we're in a new era where the data matters, right? This is where data processing, but having access to the data, so it's really a whole nother level on top of the frequency, the signals, these actionable points that you can take. He was just mentioning that. Like, this is the new thing. This is not necessarily new, but it's gettable now. It's mainstream. What's the impact to the financial industry as it's, I won't say democratization, everyone's using that word, but it is kind of being democratized, but the financial industry will capitalize on it first. What are they going to do with it? What's the trend? What are they going to do with this new advancement with HPC intersecting AI, which is compute, more power, more speed with AI? This is the real confluence. Yeah, it's really interesting, John, because the more HPC power you have, the more quantitative analytics you can do, the more research you can do, the sooner you can get through ideas. So there's never been a shortage of quantitative ideas on Wall Street or anywhere else. So the problem has been being able to validate those ideas and test them. So the faster people can validate and test them, the faster you can get through the 99 bad ideas and get the profitable one. Now, that doesn't mean that's going to be one profitable trade out there dominating. Speed is interesting, right? Because only one company can be the fastest. Only 10 can be the top 10. But the more and more quantitative strategies have in the market, the more companies can compete and also the more difference of liquidity in the marketplace, which actually makes for a safer, more stable market. So what we actually see is by diversification of strategy, which is being led by quantitative strategy, by the ability to do this HPC, the more that we're going to have interactions in the market for different reasons, which makes for a healthier market. And for everybody who's at home, who's maybe not in finance, who is trading on their own or worried about their 401k or something, it actually gives you better, tighter markets so that you're getting better prices when you get in and out of positions. Yeah, and I love that. I would like to add that. I mean, right, on top of it, right? We live in a world, right? Market data doesn't get you any advantage, right? Finance has always been behavioral, right? Motivated by psychology. And so what we get here is a new area called finance for alternative data, right? So people are, companies are able to generate signals at a consistent speed, right? Which is able to generate into signals, right? Data and they're able to trade based on the data. And so we live in this new world where you're able to generate new data at the speed of light, right? Which where we are working along with Dell and you are able to generate new signals that tell you where to invest, right? And these are being driven by such models and they are being used by downstream algorithmic models. And there you go, right? You can make alpha, which is what finance really wants. You want to make excess returns and you want beta, right? You want enhanced index returns. Yeah. And we are in that stage now where Dell has been a great partner. This is a great, Greg Segway. This is a really important point because as Peter was saying, you got more speed, more insights at scale. Now you have also the other end of the spectrum, misinformation. So back to Peter's point about the human factor, at the end of the day, it's the opportunity. It could be a day trade. It could be the next hedge fund. Who knows if they get the right signals? It's going to be a real interesting game and again, it's going to come back down to the scaling the intellectual capital of the financial professional or team, which brings us back full circle, the intelligence aspect of it. You get the speed of the network, speed of the compute, access to the right data or wrong data. That's got to be figured out. This is new, new concepts because it's at scale. Now AI has been done before. What's the big pivot here? What's the factor for these companies, if you guys could look at it as an industry and say, what's the most important thing people are paying attention to right now as they put their toe in the water or jump full in to this new world? Thank you, John. For us, it's basically large language models. Every client wants to build this platform. Either you are in business or you are out of business. For example, credit trees, which is a major bank, I mean, it had like market volatility then. So we have seen that clients who do not have such technologies are either going out of business, right? And it's not only for making profits, right? You want to make sure that you manage your downside risk here, right? You also want to make sure that you are able to capitalize on market opportunities and look out for your own areas. We call this like large language models NLP for early warning indicators. And this intersection of HPC and AI is helping us build that. Peter, what's your take on it? John, sorry, go ahead. Oh yeah, I see large language models is beating hot right now. And it is a talk across all industries, not just capital markets and finance. But there's another interesting trend that I'm seeing as well. And it's a little counter to large language models, not saying that those are going away. It's in parallel with. So large language models are massive training jobs, right? That just go on and on and it needs to be done and you need a lot of power to get through them. But the other trend that I'm seeing in finance is that when you're doing a smaller quantitative models and you're backtesting over years, you have to be able to, oftentimes they're training and retraining daily. So if you're backtesting over 10 years of data, you're retraining many, many models every day of that backtested history, which means you really need the ability to instead of go very deep training one model, you need the ability to scale broadly training hundreds in parallel over thousands of days of data and updating it with every new day. So we're seeing just this massive need for scale in the financial services industry in order to do their training because they're training so many models. And then once they've trained that model, they then need to test it on a vast amount of data to make sure that they're not just the anomaly of what was successful one day and then it blows out the next day, right? So it's massive training and then massive backtesting off of that training. And that's really driving a need for high performance compute. Andrew, you're in the business, you got to get access to the data, you got to tune it, operationalize it, then tune it and let it run. I mean, this is more compute. You're in the middle of it. 100%, I think what my colleagues have said is they're talking about the hot new trends, right? And of course, we're paying attention to that, but we also don't want to get distracted that there's always a consistent baseline of need for HBC within finance on the less, for lack of our term sexier workloads, right? Such as banks, insurance companies, right? How are people pricing loans, the risk of default, et cetera. And then I know probably mentioned behavioral, right? How does an AI machine or entity factor in, starting to factor in personality into decision making for these banks, right? Not just based on hard metrics and data, but how do you actually quantify some of the softer stuff that goes into that? And that's what we're doing with our hardware, right? We've got four-way, eight-way boxes coming out. They're in the works, et cetera. We've got other solutions that are gonna be launched later this year that are, again, thanks to NVIDIA and STAC. They've been, of course, Peter and STAC are an objective company, right? They have to maintain that objectivity. And the probably when I are working to figure out how do we maximize performance on everything, right? How do we get the best out of everything we do? And that's why we exist. Andrew, I gotta say you brought up a really good point. You gotta refresh the speeds and feeds and get that hardware, new hardware in there because here's what we just brought up in that last segment was is that everything's sexy now because look at the bank failures that was mentioned. Those small areas matter. They could have been identified with the signaling. Every workload now is part of that system. This is where I think it gets interesting because what was once not sexy, small little workload could be a huge impact when you start to think about the scale of the data. I mean, this is why the bank failed. The bank, some of the way the bank failed because they just weren't paying attention. They tried their best. They were trying hard. They just missed it. It was a complete miss. Completely flat. Well, it's funny how these things happen in life, right? I know we're not in the fashion industry but it's like 90s fashion coming back in vogue now. It's all a big cycle like Peter can tell you Well, it never went away in my book. I think always hardware always matters. Just went away with cloud. I think cloud abstracted away with DevOps what all that under the covers things but it still happened. There was still servers. Now you're looking at silicon to app, super computing, super cloud, super apps coming. We see this clearly. And I think the financial services is a tell sign always been an indicator. I mean, that's final thoughts is what's that indicator guys? What's the, if each of you can share your thoughts on what's the key tell sign right now that the financial services are telegraphing to the marketplace? What would it be? John, I can definitely add your rate. I'm a coin by background. I worked on all the use cases. I'm a CFA charter financial analyst, right? And I've never seen an area rate where both quant is in play, right? And you can identify crypto companies such as FTX failing, right? And you need to combine both the HPC coin part and the AI part, right? Otherwise, you're gonna see like an era where you will continue to see more banks not doing well, risk is not managed. So we are in this era, right? Where the intersection has already happened, right? It's not in the future. It's now, right? And firms that have been able to capitalize on it are doing well, right? And we look to continue this partnership with the NVIDIA Dell to take these solutions to clients. And we have been seeing pretty much demand from every client in the market. That's awesome, great, totally agree. Andrew, what's your take on the tell sign from the financial services world that the world is, the future is here? Well, you know, I'm gonna take a step back and answer that by saying that with, it's really, and it goes back to what you're saying, government regulations or lack thereof and how that's going to influence the wider ecosystem, right? Because what we're seeing with the bank failures is now the regulations are starting to kick back up, which will obviously then drive the demand for better HPC abilities, right? Because to Peter's point, the more data you can crunch, the better you can assess the risks. And therefore, the better you can either generate returns for your clients, manage any downsides that might occur, et cetera. So I think the indicators are gonna come where we see the regulations trending. Awesome, Peter, your thoughts, final word. Yeah, what we see is, you know, for a long time we talked about full stack developers. What I'm starting to see now is what I think of as full compute developers. You know, we live in a world of heterogeneous compute and that doesn't mean that we have a slew of different types of compute that we then take a job and run it homogeneously on one type of compute. It means that we break our algorithms up and we run them on whatever compute is best for that section of the algorithm and then get those answers together. And so we're seeing, and where I'm seeing that really being driven home is by financial institutions looking for developers that can program to a variety of compute. That's why I call them a full compute developer. And they're truly taking advantage of heterogeneous offerings, you know, and I think that's something that we see, you know, Dell and NVIDIA stepping into. I love that, love that line. Compute developers, they've got to be crunching the data. We're whole mother level. Next gen here. Prabhu, thank you. Andrew, Peter, thanks for coming on. Great segment on the financial services impact. Dell and NVIDIA and the financial services, this is theCUBE's coverage of ISC 2023. Thanks for watching. Thank you.


Hello everyone, welcome to this session of timers part 3. At the end of this session, you will be able to examine symbol and basic usage of off-delay timers which are available for the PLC programming. You will be also able to write ladder-run diagrams with off-delay timers for basic operations. So, let us understand more about these off-delay timers. In off-delay timers, output Q gets true immediately when input becomes true. But it will wait for a preset time PT after an input in a ladder logic becomes false before turning off the output. ET value resets on every next rising age of input. So here you will see the symbol of off-delay timer available in IAC 61131 part 3 standard having two inputs and two outputs. Let us learn more about these inputs and outputs. So input when it changes, timer starts. PT stands for preset time. The preset timer value indicates the desired delay which you want in your operation. The data type associated with input I in is Boolean means you are expecting here either zero or one whereas the data type associated with this PT preset time value is time means you have to mention your desired delay in seconds. Q here shows the output whether the time has elapsed or not. Time shows the time still remaining that is the time that has elapsed since the timer was activated. The data type associated with this Q is Boolean means you are expecting here either zero or one whereas the data type associated with this ET is time means you are expecting here the value in seconds. Let us learn the operation of off-delay timer with the help of the waveform shown on the slide. The main function of off-delay timer is to delay the falling age of output Q by predefined preset time. So here you will see that at T1 input change from 1 to 0 and after PT duration output changes from 1 to 0. So here a delay of PT value is added between the falling age of input and falling age of output. The output gets true immediately when the input condition at the timer is satisfied. So here at T0 you will see that when input changes from 0 to 1 output also changes from 0 to 1 at the same time. But timer is in still off state. At T1 instant you will see that when input changes from 1 to 0 timer starts and ET value starts incrementing towards PT value. So at T1 plus PT you will see that output changes from 1 to 0. So at T0 and T2 instant you will see that when the input changes from 0 to 1 output changes immediately from 0 to 1. Whereas at time instant T1 and T5 you will see that when input changes from 1 to 0 output will take PT duration to switch from 1 to 0. So at T1 plus PT and T5 plus PT output change from 1 to 0 respectively. After the input becomes false that is at instant T1, T3 and T5 timer starts and output will become false after preset time PT. But provided that the input has not gone true again in the meantime. So here you will see at time instant T3 input changes from 1 to 0 and at T4 input again changes from 0 to 1. So at T3 you will see that timer starts and ET value starts incrementing towards PT. But at T4 again the input value changes from 0 to 1 and ET resets to 0. So because of this this change in the input is not reflected on the output waveform. Means you have missed this change in the input on the output waveform. So let us see why this happened. So here if you compare the duration for which input is false are different. So for the output state to change from true to false the input must be false for at least as long as the preset time PT. So here you will see that between T1 and T2 the input is false for more than PT duration and because of this output is changed during this time period. Whereas during T4 and T3 you will see that the input is false for less than PT duration and this is not reflected on the output waveform. So for output state to change from true to false the input must be false for at least as long as PT duration. So this limits how you fast change your inputs. So this is the T of symbol available in open PLC editor under standard function block. Let us solve some examples with the help of off delay timer. So let us develop a ladder diagram for a system which turns on a motor when a switch is activated and keeps motor running for a predefined time let us say 5.5 seconds following the switch is deactivated. So let us see the solution for this particular example. So here you will see ladder rung is shown with input switch, output coil and off delay timer in between them and the waveform on the right hand side. Let us understand the operation of this solution with the help of waveform. So at P0 instant you will see that input 1 contact closes means input 1 changes from 0 to 1 immediately output 1 also changes from 0 to 1 means output 1 energizes. At T1 instant you will see that input 1 contact opens means here you will see that input 1 changes from 1 to 0 but here you will see output has not changed. So because of this timer starts counting with PT is equal to 5.5 seconds. So after 5.5 seconds at T2 instant you will see output 1 de-energizes. So here total 5.5 seconds delay is introduced between the falling age of input and the falling age of output. So this is one example which shows you how off delay timer can be used for controlling purpose. So let us see one example of simulation on this online PLC fiddle platform of a timer. Here you will see two rungs are there in this ladder diagram, two switches. One of the switch is timer dependent switch. Here you will see off delay timer and a motor on the right hand side. Here the preset value is set to 5.5. Let us see what happens when I change the status of switch B. So when I change the status of this switch B you will see that output motor started immediately. The accumulator value which tells you about the AT elapsed time is 0 here. Now here I will change the status of this switch B again. I will make it off. Here you will see the accumulator value that is ET value is incrementing towards 5.5.  So this simulation shows you how a T off that is off delay timer is used to add the delay in the operation. These are the references which you can go through for further reading. Thank you.


Welcome to NewsClick. Today we have with us the Lieutenant General Prakash Katoch who retired as commander of the Special Forces of the Indian Army. He is also a prolific writer and he has been writing not just on military affairs but also other issues that concern the country. Welcome to NewsClick sir. Thank you. I would like to begin by asking you something that Chief of the Army Staff, General Bipin Rawat said recently at one of the meetings where he talked about and introduced the subject matter of hybrid warfare. And I quote, hybrid warfare he said is a form of war fighting which is all-encompassing rather than focusing on destroying an enemy's military capability. It focuses on population and infra as the center of gravity. It covers the full spectrum of warfare by combining the vigor, lethality of conventional warfare with fanatical fervor of irregular warfare and untapped spectrum of technology for cyber and info warfare. It also entails creating acts of criminal disorder, law and order issues and public disorder in the target country during peace. It embodies the age-old saying all is fair in war, unquote. Sir, can you explain to a normal person what this means? When he says that the focus, the center of gravity is not just military capability of the enemy but it focuses on population and infra. I agree 100% with what General Rawat has said. In fact, we have been fighting hybrid war for the past 20 years, even in the case of Pakistan or with China. Conventional war is only one part of hybrid warfare whereas the understanding generally in India is that war means conventional war. So, any problems, whether we have it? So, is hybrid warfare what is also called counterinsurgency operations? Not exactly. You see, hybrid warfare. So, what distinguishes hybrid warfare from counterinsurgency? See, we have conventional warfare, we have sub-conventional warfare which is below the level of the conventional warfare. Counterinsurgency and counterterrorism will be part of the sub-conventional warfare in which you have also the employment of irregular forces, which Pakistan has been doing against us. Now, in order to fight that, it can't be if it is against the nation, the hybrid warfare is against the nation, not against the military. So, it is not the military alone that can fight hybrid warfare, it has to be at the national level. But, say, if there is a hybrid war in, say, domains of cyber warfare, electromagnetic and space, the military doesn't have any say here. It has to be done at the national level. So, when you say that the focus shifts to the center of gravity shifts to population in India and not the military capabilities. No, it includes both. The center of gravity will remain the population, but the application of military power also is as important. Because in present-day warfare, what is happening is that conventional wars are receding. They are not... One of the commentators on hybrid warfare and army's land warfare doctrine has pointed out that if we move towards counter ops, coin ops, which is the new term that is being used. I would like you to explain that later for our viewers. That it would mean giving the air force the lead role. Lead role that the army, because it would be a shift away from conventional to coin ops, counter insurgency operations. And where air power would play a critical role. And in other words, the lead role would have to be provided to the air force. Whereas the army chief is on record saying that the army would still be the leading force. How do you respond to this? Firstly, coin is not a new term. It's been an American term for a very, very long time. Coin. Coin. Yes. But so is hybrid warfare. Yeah. It is part of hybrid warfare. It's also an American... It's part of the sub-conventional segment. Okay. So what hybrid coin you're saying is all of it comes under the category of sub-conventional warfare? No. Other way round. Sub-conventional warfare and coin are part of hybrid warfare, as is conventional warfare. So hybrid is much more than... Yes. Hybrid warfare is at the highest level. It includes all this. But to say that in hybrid warfare, in say coin, you give the lead role to the air force, I think that's drastically wrong. You see, firstly in our own country, we are not using air force or artillery against insurgents. We are not the American army which is going to go and, you know, not bother about the collateral damage. That apart, wherever U.S. and NATO have been fighting, say, the Taliban or Al-Qaeda or the ISIS, has the air power been able to subdue these proxy forces? Not at all. So it's wrong to say that the lead role... You mean it's the ground forces that have always played the most critical role? Of course. Air force has a role. But it can't be the lead role can be given to and as far as we are concerned in our own country, we don't use the air force and the artillery, even the artillery. Not even attack helicopters. So where is the question of giving the lead role to... But we have done it in the past. Where? We have done it in Mizoram. Mizoram was different. Mizoram was one time when a brigade operation with air was utilized. And it was again because of the political hierarchy. Otherwise, that was not a class... Aizol was not a classical city which had to be bombed out. Let's return back to hybrid warfare. So you were explaining that hybrid warfare is... combines both the conventional and some conventional warfare. Plus, as the chief said, all forms of warfare. One of the important things he also refers to is the war of perception. Will you explain what war of perception means? It is the information warfare. It is that information warfare which you see today on social media which both the BJP and the Congress are playing. To attack your perception and lead your thinking in one particular manner. Unfortunately, we do this more in politics. But then he also said that the social media should be controlled. One would assume that if it's a perception warfare, then let there be battle of ideas. How can you control the social media? Has anybody been able to do it other than Chinese? So what does the army chief mean when he says that it should be controlled? I suppose he is... I suppose he is talking of the serving soldiers. That they should not get involved into this. But otherwise you cannot control it? So this is addressed to the troops themselves? Possibly. Possibly. He is talking of the troops under his command. That they should not get into this. But at the same time, when we are doing perception warfare or information warfare in J&K, from our side, even the military has to be involved. As I said, hybrid warfare is against a nation. One would say that the military always is involved because it comes out with its statements. It responds to any comment that is made against it. So it always intervenes and its word is carried by the media. So its version of the story is always carried by the media. So what more needs to be done beyond that? No, I haven't got a question. My point is that you said in the war of perception, its information, that is, you use the example of BJP Congress, each trying to dominate the social media coming out its version of the narrative to gain more ground and credence than the other persons. So it's a similar thing happening on the ground of, say, in J&K, as you said. Now the point I'm making is that the army has to ensure that its version of the events or its narrative comes out. That's already coming out. So what more needs to be done? No, it is you have to tackle, say in J&K, you have to tackle radicalization. So if you are... What do you mean by radicalization? Like the ideology, the Wahhabi ideology, the ISIS ideology. So there are hundreds of... Why should we be marking out J&K when we forget that in the rest of the country, let me complete. In the rest of the country, there is rabid radicalization by the Hindutva elements. And there is fear that this also probably permeates into the social media of the ex-service persons or the service persons and their families. And probably one of the reasons, one of the anxieties behind Armichi's statement is to guard against that. So given that when radicalization is taking place in the entire country, the Hindus are getting radicalized and you're saying Wahhabism is there. Why is that so special? It is as far as I'm concerned. Why should we ignore one and just focus on the... I am concentrating on areas where the army is involved in conflict. But Jammu is another... So you have hundreds of... Jammu-Kashmir is one state, right? Yes. So you have... Jammu has had the most unprecedented growth in Hindutva radicalization while J&K has been undergoing this trouble. Why is it that when the army talks about radicalization, it only talks about Wahhabism but never talks about Hindutva radicalization in Jammu and Kashmir, which is the area of your operation? Let me tell you a... You talked of Hindutva radicalization in Jammu. What about the Muslim radicalization in Jammu? No, I'm... No, no, I'm so... General Katoch... General Katoch, I... No, no, no, let me also talk. I'm not talking about that. No, the reason I'm pointing out is that we talk... There is enough talk about Islamic fundamentalism or radicalization in Jammu and Kashmir. There is... It's all over media. What one never gets to hear is what is happening in Jammu, which is very much a part of Jammu and Kashmir and its entire area is disturbed and armed forces, special powers and armies deployed there. Given that, why is it that it's always silent on Hindutva radicalization that has taken place? And we have witnessed it in the last few years as your own writings vouch for it. I have not said so. In fact, it is the other way around. I've even said that how is it that when Article 370 was there, 4,000 Rohingyas have been settled in Jammu. How is it that when the IB is saying that the Rohingyas were involved in the Sunjwan military camp attacked by Jem terrorists, how is it that no action has been taken against it? How is it that a chap like Yasin Malik... Why? Why? Why no action has been taken? That is politics. Don't ask me. No, but tell me, this is a disturbed area. Can we also give you more examples? No, I'm saying this is for 30 years this has been a disturbed area. An army has been called in. There is central paramilitary forces deployed there. It's now under precedence rule after having experienced six months of governance rule. Why, despite all that and complete control now by the center, why is it that none of these, any action is being taken? As you yourself point out that these are serious issues, why is it that nothing is being done?  Two reasons, now let me talk. First is that in all these areas, population is the center of gravity. That is one, which is not being tackled. It is not a military solution, it is a political solution, right? Second, most important, and I've written about this, do we have the political will at the center and the concerned states to finish this problem in JNK, in Nuxlite area, northeast? In my perception, no. Nobody is interested. Who are the people dying? Is the security forces dying? Who are the civilians getting killed? They are the riffraff. Is any politician or bureaucrat who is surrounded by layers of security being attacked? No. It's a game that is being played and I've written that number of times. Nobody has the will to finish the insurgents. When you say, I mean, I granted this point that there has been no political solution which is delayed, therefore, and prolong the stay of the army in these areas because these finally at the end of the day, as you yourself went out, it requires a political solution so that it ends and there is a closure brought to it. But there is something, the civilian riffraff that you talk about, what do you mean by that? Civilian riffraffs versus the soldiers dying. I am saying the mango people like me, I am getting killed. No bureaucrat politician is getting killed. They are surrounded by layers of security. You tell me in JNK today, if there is a problem of unemployment, there is a problem of unemployment in the rest of India also. Why is it that the government doesn't come out? Okay, this is the employment plan which we can create, provided you lay down the gun. Are we interested? Nobody has done that. Nobody has reached out to the people. And of course, when... Which is what has prolonged the stay of the involvement and the continuation of the... because there is no political closure to this. No, because there is no political will to close the case. There is no political will to close the case. Neither at the centre nor at the same as with Nuxlite areas. You know, Chidambaram said in 2010, in three years Nuxlite problem will be over. So did now Rajnath Singh. Rajnath Singh, right? His latest statement is that... And you read the article by RSN Singh in Indian Defence Review. The things actually gone worse. No, he himself admitted that it will take another two, three years. So that keeps on getting prolonged because there is... they don't see beyond military solution. They are probably banking on the military to crush it. Whereas, as you yourself point out that beyond the point, it has to be a political solution. Of course. The military can keep the level of violence to a particular level. Even Jail V.K. Singh, if you remember when he was the chief, he said so. He said the politicians are not optimised. What the military has done. And then Farooq Abdullah jumped the roof saying that... No, but that's a very interesting point you've made because this is precisely also what the Supreme Court in 2016 observed. That the prolonged use of military forces is actually a blot in our democracy. Because it means they said either the military is not able to bring about normalcy because and therefore it has failed. Or the civilian and the politicians therefore have failed not to make use of opportunities provided by the military through their operations. I would like to qualify it even further. You see, there's always talk of armed forces special powers act. Nobody, nobody talked of the disturbed area act. It is first the areas declared disturbed area and then the armies brought in. Army can't be functionally like the police. Otherwise, might as well use the police which is equal number today as the army. If you want the armed forces special powers act to be removed, you remove the disturbed area act, army will go back to the border. Can you manage that?  General Katoch, I think I'd like to end this interview because what you've said is the very important point you have drawn attention to that while we talk about AFSPA, unless the root cause is addressed, the disturbed area, if there was no disturbed area declared, army would not be called in, which is to say, if there was a political resolution of the problem before it gravitates to this or reaches this stage where the army has to be called in, before that it must be resolved. Or after the armies called in, they are able to restore order. That's the time the political opportunity should be seized and a solution worked out. Restoring order again is a very loose term. As I said, order can be restored by keeping the violence at a particular level. Army cannot restore the situation. No, bring about, yeah, political solution has to be brought about in the politicians. So have we made any progress towards removing the disturbed area act? No, we are very happy with the 240 terrorist killed and surgical strikes and whatnot.  No. What has, which government done anything about it? No, I think that's a very important point because if we see, and this is the note we'd like to end today, but we'll return and have General Katoch, left in General Katoch with us again to discuss because these issues are not going to go away very soon and we'll need someone like you to explain to our viewers from time to time what is happening on the ground and what is the armed forces own perception of the events. Thank you for watching NewsClick. If you have any feedback, do get back to us.


Stock option magic at Microsoft only reason that Bill Gates started the company was to be able to share ownership Share ownership because that's the secret to real growth Did some of the people leave you Bill? Yes. Does it make a fuck? No value of stock options from 1984 to 1997 the value of stock options in the US public companies has grown from 200 million dollars to over 1.2 billion or over 600% and from 97 to 2008 and it's grown to a hundred It's not just silicone valley because everybody thinks it's silicone valley Morgan Stanley 91% of their stock is given away to the public options Merrill Lynch 40% travelers insurance 39% Warner land with 35% Microsoft 32% JP Morgan 29% Lehman Brothers, which is gone now 28% US Airways, which is also gone 26% Sun Microsystems 25% Marriott 25% Bankers trust general mills MCI Those are big huge percentages of big huge companies that they were given away in stock options Why do they do it do they do it because it would be mr. Generous like you know mr.