Mae'r syfyllfa'i rhaid i mi allan yn ôl... Mae'r gwahyr be'r peth gynhwr llwyddiad hwnna, y lle'r Llywodraeth Cymru i'r surelylion Cymrae, mae'r ddau yn hyn o'r ddau. Pryddoch yn gwneud cael eu cyflwyn o'r dweud, a chynau'r dyfodol yn y gallan yn y panelodания. Mae'r ddau'r cyflwyn o'r ddau, a'r ddau'r cyflwyn o'r ddau'r ddau ar y cwymau. I'm delighted to be here and to be asked on behalf of the Institute to chair this session where our colleagues here, which I'll introduce in a moment, are going to launch a paper in relation to the crime terror nexus. This is a project involving King's College in London and Pantiray Research Limited, and our two speakers are going to be Professor Dr Peter Newman and Rajan Bajor. What the project is about is presenting an overview of the links between crime and terrorism. A difficult subject in the form of police officer, down through the years I would have known that that's a big challenge and a lot of commentators have different views on it. So I'm delighted to know that we have people who have done some research on it and open the debate so to speak in relation to it. There's always been a difficulty. Our first speaker will be on my right here, Dr Peter Newman. Peter is a Professor of Security Studies at the Department of War Studies, King's College in London, and has directed the International Centre for the Study of Radicalisation since its foundation in 2008. He has written a number of books, including Radicalised, No Jihadist and the Tread Against the West, and he also has produced a number of policy reports in this particular area. He currently serves as the OSCE Chairman, Chairman's Special Grant. Until the end of the year. It's gone. It's gone. It's gone. It's gone. It's gone. It's gone. It's gone. It's gone. He was a senior consultant for the US mission to the United Nations during the process of crafting during Security Council resolution on foreign terrorist fighters. I have no doubt that Peter is imminently so to speak to us on this topic and he also studied in Northern Ireland a number of years ago. Our second speaker is Mr Rajan Bajra, who is Head of Research and Terrorism for Cantoray Research Limited, which is a UK-based research organisation, and has specialised in counterterrorism financing and links between terror and crime among European jihadists. He walks closely with Professor Newman and he has become one of Europe's leading experts in the crime terror mixes. So what I'd further to do, I'm going to ask Peter to speak to us for 15 or 30 minutes or so, but I ask you to maybe use the podium if you want to deal with what you've been doing from here, if it suits you, whatever suits you. Because we're switching around. Okay, that's okay. Great. Excellent. Well, thank you very much and thank you for the opportunity to speak to you in Dublin. As my colleague has said, my name is Peter Newman, I'm director of ICSR and Rajan and I started looking at links between terrorism and crime two years ago now. And we were first focusing on jihadists in Europe and then gradually also becoming interested in other kinds of links between terrorism and crime, looking at what came to be called the crime terror nexus. And we were lucky enough to receive a grant that allows us now to investigate these links in all 28 member states of the European Union. And this is the first of 15 reports that will be published over the course of this year, focusing on the UK and Ireland. At the three areas where we've been able to identify links between terrorism and crime are the following. The first among paramilitary groups in Northern Ireland, which are deeply and remain deeply involved in organized crime in Northern Ireland, in the border area, and of course also in the Republic. Secondly, in British prisons where we have seen evidence of criminal and terrorist linkages and connections and cross fertilizing. Thirdly, amongst British jihadists who before radicalizing used to be in many cases criminals and whose skills that they acquired as criminals may in some cases be used for terrorist purposes. That brings me to one of the key points. Why does it matter? Why is it a problem when criminals and terrorists link up? Why is it a problem when crime and terrorism cross fertilizes? Very simply, because it makes both of these problems worse. It allows both of these phenomena to become more entrenched. In the case of jihadists, for example, it makes it possible for terrorists to use and utilize criminal skills, more easily get hold of weapons, more easily fund themselves and more easily acquire forged documents. From a government and law enforcement perspective, it is a good thing to watch out for these links, to prevent them from emerging and to counter them where they exist. I do not want to bore you with a lot of theory and conceptual stuff, and Rajan will give you a baseline assessment of organized crime and terrorism in a second. But just for a second, let me tell you that there are in the literature three types of links between crime and terrorism that have been identified. One is if you want an institutional nexus, which is about formalized collaboration between organized crime groups and terrorist groups. This is about groups collaborating. A second version of the crime terror nexus is what we call an organization nexus, which talks about organizations where it's not entirely clear whether they are purely criminal or whether they also have a political agenda. In fact, in many of these cases, it can shift from one to the other. The third nexus, if you want, the third type of nexus is of terrorist organizations who are recruiting from criminal miliws. This is particularly prominent in the case of ISIS recruitment in Europe. ISIS in Europe has attracted a lot of people with criminal pasts in some European countries. A majority of the recruits that ended up joining ISIS used to be involved in petty crime or in some cases organized crime before. In fact, if you look at these three types of crime terror nexus, we have found evidence of all three types across the islands of Ireland and Britain. Now over to Rajan for an overview of organized crime and terrorism in UK and Ireland. Just to give you a snapshot of the crime situation in UK and Ireland, the number is very quite a bit. The UK's NCA, they estimate there are around 6,000 organized crime groups operating in the UK with 40,000 individuals involved. Here in Ireland, the statistics from the garden are a little less publicly available, but the last open source statistic that was available is from a couple of years ago said that there was action against about 600 organized crime groups here in Ireland. But they didn't disclose the number of individuals involved. And when we look at the kind of activities, the behaviors and the locations that they're involved in, we see two things. The first is that they're really diverse, and I don't mean ethnically diverse. In fact, the majority of organized crime groups in the UK and Ireland are either Irish or British, although there are in some markets, so let's say in the opium trafficking in the UK, it's dominated by Pakistani crime groups. Famously, Albanian crime groups are musling in on the cocaine trade in the UK as well. I mean diverse in terms of their organizational arrangements, right? So you still have these hierarchical, traditional, top-down, family-orientated crime groups, but you increasingly have more loose networked crime groups as well, which afford more autonomy to their individual members. So this is why you would see crime groups cooperating with other crime groups, which you perhaps wouldn't have anticipated before. And as well as this, you even see criminals selling, or is essentially their business, to other criminals as well. So in the UK, we see this with drugs supply lines, the county lines, as they're called, from London to the surrounding areas. They're also diverse in terms of the activities they're involved in, right? So organized crime could be anything from human trafficking for labour or sexual exploitation, to weapons smuggling, to wholesale importation of drugs down to street-level dealing, to more traditional crime activities like extortion or more novel innovations like cybercrime and new aspects of fraud. We also see that they're very responsive, and they're responsive to law enforcement behaviour. It's a very true cliché that it's a cat-and-mouse game. They're constantly adapting and responding to how law enforcement cracks down on them. But they're also responsive to new market opportunities, right? They're essentially criminal entrepreneurs. If they see a gap in the market, they will look to fill that. And this is something that you can see in pharmaceutical trade. So we spoke with the UK's Medicine and Health Products Regulatory Agency. It sounds very, very boring, but they do some pretty interesting work. And they said in recent years it's been a significant rise in the illegal importation of unlicensed pharmaceuticals from countries such as India where they're produced on mass because it's just a low risk, high reward market. And if you look at it, it's really easy to see why, right? It's such a simple business operating model. It's not really a priority for law enforcement. And if you are arrested for this, it's unlikely to result in a custodial sentence, mostly just defined. There's not really violence involved in it. And so you can see why criminals would look to exploit this. So this is just a snapshot of the crime scene. If we look to the situation of terrorism, well, foremost on everyone's mind is of course the situation with jihadism. And in the UK, the threat is perhaps as severe as it's been since 2001.  You're looking at low-nacted plots, highly networked cells. You're looking at really rudimentary attacks with vehicle ramings, knife stabbings to coordinated bombings. You're looking at people that were born and raised in the UK, homegrown jihadists, as well as those that have travelled abroad, returned or in contact and directed with people overseas. It's also perhaps gone against some stereotypes about jihadism, so women have recently become involved in some plots. And this is all reflected in the attack scene in the UK last year, the number of plots and the arrests that have also seemed to be ever increasing year on year. So in Ireland, the situation is very different. I don't think the jihadist threat is anywhere near what it is like in the UK. It's a very different scenario. But where Ireland does obviously see the issue of terrorism is with paramilitarism. And just last month the British security service said that Northern Ireland is probably the most concentrated area of terrorist activity in all of Europe. And you can see this with a steady drumbre of terrorist attacks. So in the last five years there have been almost 350 shootings relating to Northern Ireland terrorism. As well as over 250 bombing incidents as well. And there's also a steady seizure of explosives, of firearms and of course the number of people arrested. And it's widely understood that all groups, whether they've actively been involved in decommissioning efforts, maintain access to weapons or arsenal of weapons as well. There's another terrorist situation that you see and that's with the far right. And of course in recent years there's been a resurgence of the general far right across Europe, especially in the UK with street movements as well. Occasionally from these subcultures lone actor terrorists emerge so you could think of Tommy Mer who murdered Joe Cox MP. But generally relative to the threat of jihadist and Northern Ireland related terrorism, the threat of far right terrorism is low. So that's the snapshot of the crime and terrorist situations in the UK and Ireland and we can now discuss where the actual nexus occurs. Right. So Rajan and I will now briefly touch upon the three areas where we've identified links. The first is that I want to highlight is on British prisons in particular. Prisons have always of course been places that were significant in relation to this particular phenomenon. That's because physically this is where criminals and terrorists are in the closest proximity. It's because when people enter prison they often ask themselves existential questions. They are open for new ideas and value systems. They experience a severe form of dislocation. They are cut off from their traditional social networks, from their families and friends. They sometimes if the prisoners bad need protection. So prisons matter. They are places where people network. We know that in Britain for example the number of jihadist prisoners has risen almost trebled over the past six or seven years. This is a big issue in British prisons specifically related to jihadists. Muslims make up 15% of prisoners across the UK in some high security prisons. The number can go up to 20% to 25%. The number of Muslim prisoners has been rising for all sorts of reasons over the past ten years. Very significantly it's said to have doubled. That's of course not a problem per se. Religion in fact, religious conversion can have a very positive influence on prisoners. But we're also seeing the rise of so-called Muslim prison gangs which are not terrorist groups. They are based on identity. They enforce strict rules. They are sometimes quite aggressive and coerce other prisoners for example to convert. This is a problematic phenomenon partly also because partly out of bravado I guess. These prison gangs are glorifying terrorism. They often declare themselves to be supporters of ISIS and Al Qaeda. It's not clear to what extent that actually continues to be the case after people leave prison. But because the prison authorities in Britain have downplayed this phenomenon for such a long time and have been quite intransparent about the existence of it, we do not really have concrete figures as to whether these are anecdotal stories or whether they really systematically exist across the system. A second problematic aspect is of course the increasing number of convicted terrorists which has risen sharply and we are documenting in our report one case where convicted terrorists actually started networking with each other in prison construing a plot that was then attempted to have been carried out outside of prison after their release. And this happens not only in relation to jihadist terrorism. It also happens in relation for example to sectarian groups in Scotland and Northern Ireland. This is one particularly prominent example Anton Duffy from Donegal who lived in Scotland and wanted to assassinate a couple of years ago two prominent loyalist leaders including Johnny Adair. Whilst he was in prison he recruited a member of his group and he also tried to acquire an AK-47 and a revolver from criminals who were part of the same prison. In other words it would have been more difficult for him to conceive of his plan and to actually carry it out to some extent without his presence in prison, prison to some extent facilitated at the plot. I know that very well because I served as an expert witness in the trial that finally got him convicted and Anton Duffy was not a former member of the real IRA or the new IRA. He was a committed Irish Republican but even these dissident groups considered him to be too crazy to be part of their groups and he wanted to carry out this attack on Johnny Adair in order to get himself noticed and get credibility with the leadership of these groups so they would eventually take him in. He also started talking about starting a sectarian war in Northern Ireland. So he was clearly out there on the scale somewhere and fortunately it was not possible for him to carry out his plot. At the same time it has to be said that without prison it would have been much more difficult for him to get the connections and to make it happen. So we also see this nexus amongst British jihadists and this is even born out by statistics. So a recent study of the UK's police national computer which logs everyone's convictions found that almost half of British converts travelled to Syria, had convictions on that system. As early as 2012, as of 2015, were analysing their subjects of interest, their nominals and they found that over 40% of those nominals had common convictions. So then they shifted their strategy to this so-called Al Capone approach. If you can't convict them for their extremist-related activities you would look to pick them up for petty frauds and common crimes, even some drug dealing, whatever you could do to disrupt their activities as extremists or even just as common criminals as well. What we find more generally is that often jihadists emerge from these subcultures where they're fluent with criminal networks as well. And this is reflected in the propaganda that these groups produce. So you can consider this group, they've got this fantastic logo right of Tauheed. They were a British jihadist group that were active in Syria and they would post so much propaganda online back when it was much more accessible. They would post it on Twitter, Instagram, on a Facebook page and so on. And often their message was targeting those from a criminal background. So two of their posters, for giving it the first drop of his blood and sometimes people with the worst past create the best futures, heavily playing on this idea of redemption or purification for previous sins. And this has a wide appeal.  So they would also play on other themes of not just redemption but power, status and respect. But it's a key example of how these groups, they basically reflect the environments in which they emerge from. And you can see this in various pockets to pockets, clusters and individuals but the case of the Woolwich boys in London, you had over a dozen guys all involved in street crime, petty crime. This is not organised as we would understand it to be. They all travelled to Syria and if they didn't die there they returned and were arrested. So this is something that we're seeing but it's important to distinguish this from the organised crime aspect. This is not what we're seeing. This is much more low level. This is social interactions, people that have grown up in certain environments and are connected to this criminal subculture. So finally Northern Ireland where we see arguably the densest and most institutionalised manifestation of this crime terror nexus in Europe. Many of the organisation that still exists, whether dissident Irish republicans or loyalists, could be classified today as what scholars call hybrid criminal terrorist organisation which means that it's not clear anymore whether they are principally motivated by a political agenda or whether crime has become an end in itself. And in fact in some cases I would argue it is difficult for these organisations themselves to say on what side they are. Paramilitary groups are part of the criminal ecosystem across the island but that's been true for many years, something that you know a lot more about than we do but there are two aspects that I want to highlight because I think they are relevant and have become even more relevant. The first issue is that of cross border smuggling. The fact that you have a border in Ireland, two jurisdictions creates opportunities for smuggling, that's been the case as long as the border has existed. And paramilitaries have been heavily involved in decades, that is also nothing new whether it is alcohol, pharmaceuticals, tobacco, other goods are being smuggled and profits are being made. It is estimated that this creates a loss to the Irish economy of 2.5 billion euros every year. To some extent it's a fact of life and there's only so much you can do about that. This has become more significant we believe because of Brexit. If the UK leaves the European Union, if regulatory divergence becomes more severe, if that is going to increase, that has the potential to increase incentives and therefore payoffs for organised crime. And it's really very simple, if you want it's the golden law of smuggling. If you have a product on one side of the border that is more expensive, more available, or is available on one side of the border and not on the other side of the border, there will eventually be someone who turns that into a profit. The greater the divergence is, the greater the profits, payoffs and incentives are. So if you have regulatory divergence in the case of pharmaceuticals for example, because the UK leaves the European Medicines Agency, then I can guarantee you today that there will be an increase in smuggling and this will benefit the main players in this game which are in the case of the Northern Irish border, dissident Irish Republican organisations. It's the most predictable thing that you can imagine and the answer to that is perhaps to close the border. But that's of course not politically desirable and as we know from our colleagues here in law enforcement and intelligence, it is also virtually impossible. So my personal view would be to stop Brexit. I don't know whether that is agreed in London but I do think that presents a problem and there is an organised crime aspect to that too and that needs to be watched out for. The second aspect are loyalist groups inside of Northern Ireland. And of course this is also true for some smaller Republican groups like the INLA and perhaps even smaller ones. But especially the loyalist groups have been central players in the drug trade in the North. That is no secret and that's been going on for a long time. There has been a process of that becoming almost institutionalised and entrenched and it's difficult to fight these organisations precisely because they are not just purely organised crime organisations but they also still within smaller and smaller constituency have a claim that is essentially political. They can still say in certain parts of West Belfast or East Belfast, ultimately we are your insurance policy, we are defenders of the community. You may not like the fact that we are dealing in drugs, but if you are being attacked by Republicans, we will come and defend you. And that makes it difficult for these groups to be eliminated because there is a small but still an existing level of community support for them. And this has become more problematic, I would argue, because some of these groups have also, what's still being organised or involved in organised crime, have successfully reinvented themselves as community groups, ex-prisoners groups who effectively get subsidies from the Northern Ireland Government. So you could actually argue that indirectly Northern Ireland Government is funding some of these groups and some of these activities. And that is a deeply problematic aspect. I do think that a blind eye has been turned for some time for the sake of not wanting to rock the boat, but I think in the long term the existence and the entrenchment of these actors in Northern Ireland is deeply problematic also to the peace process. So just to wrap up, we have a series of recommendations to start off, I suppose with the most boring academic one, which would be effective monitoring. So this is to standardise the definitions used and to publicise them so that accessible to all researchers. We also think that, not just in UK and Ireland, but all throughout Europe, radicalisation really needs to be rethought. And these old stereotypes of a religious fundamentalist that would not engage in crime or behaviours that contradict this sort of puritanical religious image just needs to be destroyed. Because many examples in Europe show that jihadists have engaged in crime, they've also taken, consumed drugs, which often goes against these sort of stereotypes. So we need to rethink what it means to be radicalised as well and understand that involvement in crime and criminal behaviour is not mutually exclusive from involvement in extremes. We also think we should have safer prisons, although this is easier to do, easier to say than do I suppose because in the UK, especially a programme of austerities, drastically cut funding for the prison cells. But this would involve training all staff, making sure that they're well equipped and not understaffed at all because we've generally found that prisons that are safer for criminals as a whole also do a better job against extremist groups. We also think that all financial streams need to be counted counter and terrorist financing is often being characterised as going against big international bank transfers and so on, whereas in reality you're often looking at small scale transactions and if it does involve crime it's on a very small petty level as well. We think that information should be shared and this is not just internationally between different agencies, we're talking about within government. Many examples of recent terrorist attacks with the benefit of hindsight is easy to say but often they could have been prevented or they could have been disrupted if different agencies within government had better shared information about the suspect. We also think that as Peter spoke about we should avoid regulatory divergence when it comes to issues of Brexit and relating to that to investigate the paramilitary crime that takes place in a much more systematic fashion. There was a great work of the independent monitoring commission and that ended six years ago and it would be great to see something like that on a much more systematic and continuing basis going forward. So the paper is available online at the moment, crimeteronexist.com and we also have established a Twitter account so please check that out at CrimeTerror. Thank you.


I think we are like a couple of minutes ahead of schedule, but I think we'll get it started and we get some extra minutes for questions if there are any. So the next talk is by Roland and Apoorva from AWS, and they will be talking to us about constructing Kuflo supernodes with Carpenter. Welcome them. All right, welcome everyone. How's everyone doing? How many people have had a Chicago pizza already? That's it. Luma Nadis, that's not real pizza. New York pizza is real pizza. No, I'm just kidding. Had to go there, sorry. My name's Roland Barcia. I'm the worldwide director in AWS and our specialist organization leading all things containers and serverless. I have the pleasure and the honor to be joined by Apoorva. He's gonna answer all the questions that you have. He's one of our lead specialists around the world helping customers build Kubernetes applications that run machine learning models with technology like Kuflo and other things like Ray. And so I'm gonna get us started a little bit talking about some concepts around Carpenter. Gonna hand it over to Apoorva. Talk a little bit about Kuflo and then he's gonna show some demo around running some of these machine learning models using Carpenter to scale. So how many people here are running machine learning models on Kubernetes? Let's see, how many plan to? All right, good. So a lot of challenges around this space, right? We went from a technology that was really started around running and building microservices and stateless applications in Kubernetes to now running a lot of stateful applications that deal with large volumes of data. And so when we're talking about scaling requirements around machine learning, it's not just CPU and memory, right? You have to deal with storage, the thing backing the data, how to make that resilient, how to make that scalable. You have to think about networking performance sometimes. The data's not resident or close. It needs to be streamed in. You need to run other streaming technology on top of the cluster, on the side of the cluster. And then those always available GPUs that are there at your disposal, right? Always accessible, but that's at a commodity these days where people are trying to figure out how to do things like slicing GPUs or sharing GPUs. And so you have to worry about these things. And you have a lot of different characteristics, right? I think it was in the room next door, the multi-tenancy con, right? Everyone wants to share clusters. Sometimes they do, sometimes they don't, but they definitely want to be able to have shared automation and shared resources. And so from our AWS customers using Kubernetes, about half of them really struggled with a lot of these scaling requirements when we're talking about cluster autoscaler. The reason they're struggling is because they are now getting very specific around requirements, right? They're thinking about things like, I want to switch instance types for these workloads. I want CPU intensive. For these, I want instances that have GPUs. There's all these new machine types that have machine learning characteristics or GenAI characteristics to run inference. There's also the notion of cost optimization. When the year started, everyone, before the GenAI moment, everyone was thinking about cost optimization. How do I use spare capacity, things like Spot in AWS or other clouds have their notion of spare capacity? How do I use specialized architectures like Graviton and ARM-based instances? So all of these things came into play and then we have the GenAI moment. We have tons of people now thinking about machine learning, running existing large language models, and how do I do that on Kubernetes? There was a lot of challenges around also multi-AZ availability, lots of different challenges. Hence, when we're talking about Kubernetes, we're really talking about pods that get scheduled on nodes. You have pending pods. And before you had something like Cluster Autoscaler working with something like a scaling group in AWS or whatever cloud component. And now what we want to do is consolidate orchestration with a single system around Carpenter. Just a note, the IG Autoscaling SIG has agreed to add Carpenter to the SIG. And AWS is in the process of donating Carpenter to the CNCF. So this is something that we're working on from a community perspective. So let's talk a little bit more about Carpenter. But really, Carpenter is trying to solve the problem around how do I get a node that when I run out of capacity with my existing nodes, how do I get a node up with the best instance type up as soon as possible, right? So Carpenter is really designed around getting that note up but with very dynamic requirements. And so for, and I'll give an example on the next slide, but the focus here is then, there's a thread on the foreground that's saying, I need, I have a bunch of pods, they're pending. I need them to be scheduled. I can't find a node to run them on. So I'm gonna look at a rule and get the best instance with the best cost performance as possible, right? That's a goal. Then there's another thread in the background that's doing optimization and doing things like node consolidation, looking at nodes to see if there's lots of different empty capacity and kind of optimizing things. So there's a secondary thread in the background doing optimization and I'll show some of that in a moment. And so hopefully you could see the text there, but when we're thinking about Carpenter to give time for a porva to talk about Kooflow and the demo, there's a lot of cool things that you can do. So this is what's called a provisioner. This is kind of the main construct that you create when you're using Carpenter. And you can do things like, hey, give me a provisioner that has a pool of different machine types, so you see C5, M5, R5, so different values. You could really go and say, hey, I want machine types that has GPUs, but they're not available using something else like Tranium or another type of machine type. You can get very dynamic with the behavior and kind of say, oh, I just want a machine with that type, it's very dynamic. You can have other types of rules like the secondary rule there is, hey, I want to exclude small instance types. So I don't want any nano micro small large. You can reverse that and get to smaller instance, but you can say launch me extra large and above. So that's kind of another thing. The other thing is capacity type. So we have reserved instances on demand instances, spot instances, which gives you great savings in a lot of scenarios, up to 90% savings, where you're using spare capacity in exchange maybe for some SLA. You might have some workloads like that and you're able to say, hey, something that's on demand or spot, it'll grab the best performance instance like spot. And if it's not available, it has cool things to handle spot disruption.  It has cool things like architecture. So if you have multi-architecture builds, AMD and ARM, for example, in AWS we have something called Graviton, which are specialized hardware, ARM based instances. It can detect that you have a multi-arch build and you can ask for the machine type that matches the image. So there's a lot of cool things that you can do from a provision or perspective. And so the other cool thing about that is that it's standard Kubernetes pod mechanisms, node selectors, node affinity, taints and tolerance, topology spread. It respects all those rules. So you can have workloads that run in certain AZs and target them. You can have certain types of processors like GPUs or Graviton. You can have certain architectures, but you're building a Kubernetes object that's kind of respecting the rules around Kubernetes. And it's using a CRD so it can be part of your GitOps process. Some of the talks you might have heard with Argo, with Pristina and Carlos over there, if you want to talk to them, really cool stuff there. And then the optimization thread is running in the background. This is a cool graphic that Raj over there created where you have here a bunch of different nodes with pods and you see some of the nodes are pretty empty. And so what you want to do is if you have this feature called consolidation equals true, you get this running in the background and you can kind of time it with seconds after empty to kind of go and consolidate the nodes to save money. And so this is something that a lot of our customers really like. It can also do things like, and you can do these things with node autoscaler with a lot more configuration, but it can do things like, if I'm on an extra large instance, I want to go down to large and it can have behaviors like that to kind of save money. You could scale down, you could scale up to run machine learning. And so Carpenter is kind of a key part of this architecture. We're really excited. It's going to be donated to the CNCF. And now I'm going to hand it over to Aporva to kind of talk a bit about Kubeflow and then show the demo. Thanks, Aporva. All right, so I think a lot of people might be familiar with this, but just in case you're not, this is what Kubeflow looks like at a higher level. Obviously, there's a lot more to it than what's depicted here. And this, by the way, comes from a medium block post by Mikhail Brees that did on Kubeflow. So it has notebooks. It has components that you can run for training your models. It has feature store now, hyperparameter tuning. You can run pipelines. Under the hood, it runs our workflows. You can track your experiments. So if you're a model using K-Serve, Selden was the session before this. That's another thing you can, K-Serve models with as well, and other things. So a really popular MLops platform for communities and because of, if you run Kubeflow, you can run it across clouds, on-prem, et cetera. And so for the demo, this is the architecture that I'll be talking about. So as a platform engineer, I wanna make sure that my data scientists are able to run their experiments using Jupyter Notebooks. But I also wanna optimize for costs, right? And so Carpenter provides that ability for me to be able to do that, which is chloramines and consolidation. So term things on like consolidation. Nothing should be running. If there's nothing running on the node, then I don't want to use that node anymore. Just terminate it. Get rid of it. I don't wanna pay for it. So when you're running Kubeflow and you want to enable Carpenter with it, so you're gonna need a couple of things along with Kubeflow. So number one is run the Carpenter, deploy the hunt chart, you get the controller running on the cluster. And then if you want GPU instance steps, you would need something like GPU operator. Other, you know, when each GPU vendor has their own kind of something demons at running to recognize the accelerated hardware and register that with the Kubernetes control plane. And so you need something like NVIDIA GPU operator. And data scientists come in, they log into the Jupyter Notebooks. They don't necessarily need to know what's running under the hood, right? All of this on the right side of the picture is kind of what the platform operators have to deploy on the cluster. And we'll talk about data on EKS a little bit later, but basically you can write Terraform Blueprints or CloudFormation or whatever your infrastructure's code tooling choices and deploy that. And then you would have these Carpenter CRDs, provisioner and node templates, which basically match to the type of instances that these CRDs would be responsible for provisioning. All right? So I jumped the demo here real quick. So here I have a Kubernetes... Go ahead. No, no. So here I have a Kubernetes cluster with EKS, and I have two nodes in the cluster. And right now, these two nodes are used to run, basically, it has Kubeflow running on it, as well as, you know, Kubeflow with Kubeflow you have STO. These are all the different components that are deployed when you deploy one in a Kubeflow. And then I also have the NVIDIA GP operator on it, so the GP operator has even sets that basically registers the nodes that have GPUs. And then I also have Carpenter with the Carpenter controllers running on it. All right? If we look at the config map for the GP operator in the config map, so when you deploy a GP operator, you can pass in an optional config map which tells the GP operator how these GPUs should be shared if you want them to be shared. So here I have a configuration for NVIDIA A100Gs to use a MIG, which is a multi-instance GPU sharing enabled. For this demo, I won't be going into details of that, but just wanted you guys to see that out. And then I have a default configuration for NVIDIA A100Gs. And basically, I don't want to share my GPUs if it's NVIDIA A100Gs. That's what the config says there. And then another one for NVIDIA A100Gs to use time slicing. So in this scenario, if I have a node label, a specific node label which I'll show later, I want the GP operator to be able to create four replicas for those GPUs, for each GPU. And so for every single GPU, my pod would be able to have, my pods can use four GPUs, essentially, out of that single GPU instance. And so how that maps to carpenter is what we'll show next. Yeah. So here I have three different provisioners. One is a default provisioner, so if there is nothing configured as far as constraints go, then I want carpenter to use the default provisioner, which is basically get a node from any of these instance categories. It can be spot or on demand. It can be an AMD architecture, AMD64 architecture. So it's basically my default provisioner, which I want carpenter to fall back to, if none of the constraints map to a particular provisioner. Next is a GPU default provisioner. So in this, it looks similar to the default one that I have, which is use spot or on demand. But here I'm only passing in the instance category G and the instance types of G5s. So this is in AWS, it maps to NVIDIA 8 and Gs. And next is a provisioner for NVIDIA time slice. So here, again, it matches similar to what we saw before with the GPU, so same as before, spot and on demand. Use the instance category of G with the G5 instance types. But here I have an additional constraint. And this is the key. So here I have a constraint that says that any time a pod has a spec which maps to a node label or a node affinity node selector, which is nvidia.com slash device plug and config with the value of nvidia 8 and G dash ts, map that to the time slicing configuration that I have provided to the GPU operator. So this is how Carpenter knows that I need to use this provisioner to be able to do the time slicing. And so what Carpenter will do, whenever this provision is triggered and the node gets triggered, is to apply this label on that node and with that value nvidia 8 and G. And that is what the GPU operator sees to apply that configuration for GPU sharing. So let's see how that works. So here I have a Kubeflow UI. I'm going to spin up a notebook. And basically, this is something that a common use case for data scientists to be able to spin up a notebook. This particular notebook is going to be using four GPUs. And so selecting an image, which has a CUDA driver built in, use four GPUs of nvidia type, and launch it. So as soon as I launch it, Carpenter is going to see that the notebook pod is going to be in pending state because there is no GPUs available in my cluster right now. So it's going to launch an instance with a requirement that has four GPUs. And so it's going to, if you look at the details of the node, it's created. So you already see the GPU operator running on it. It's going to bring up a node with four GPUs, schedule that pod on that particular instance. And so as a data scientist, when I come in and I check how many GPUs do I have, run the nvidia-smi command, I'm going to see that four GPUs. So in this scenario, the configuration that I've applied is the GPU default, which is no time slicing, no make nothing. OK. Next, I'm going to create a pod with one GPU, same as before, pick up image with CUDA in it, one GPU, nvidia-vendor. But in this scenario, I'm going to use an advanced configuration for an affinity config. And I'll go into the details a little bit later, but basically this is where I'm telling this pod to be able to use time slicing. So again, Carpenter sees there's a pod pending with a required one GPU. We check the Carpenter logs. Here we can see that it has provisioned a G5 2x large, which is a node with one GPU. It's figured out that it has capacity and spot available, so let's use that. And now you can see that because we have passed an affinity config for the time slicing configuration, it has triggered the node creation using the GPU TS provisioner, which was the one that we are using for time slicing. And then one GPU. So now if we see what needs to happen in a pod spec for the notebook, where how is it that you can configure a pod, a notebook pod to be able to use that affinity config? This is where we kind of get into the weeds a little bit. If you are not familiar with how you can configure that, I encourage you to check out Kubeflow documentation. But in a nutshell, you basically pass a config map with the required pod specification for your notebook pods. And this is where I'm passing in an affinity config, a node affinity where I want that label to be applied. So nvidia.com, do as plugin, and use that NVIDIA 810 GTS value there. And this is how Carpenter knows that there's a pod coming in with this particular constraint. So now that I have the instance with one GPU enabled, time slicing enabled, we should be able to see that NVIDIA device operator has recognized that I need to apply the time slicing configuration, which means that I need to have four replicas of the single GPU available for my pods to be able to consume that. This is what the GPU operator does in the background. It's going to apply those replicas on that instance. And now I can schedule four pods on this single GPU instance or requiring a single GPU. So all of these GPU 1, 2, 3, 4, each is a notebook pod with one GPU required each. And because of time slicing being enabled, all of those four GPU pods are being launched on an instance with single GPU required. Next, I'm going to show a demo of how something like this can apply to a pipeline as well. So if you're running Qt4 pipelines in your cluster, standard pip install of the KFP, I have two tasks to basically contain the components. One is a GPU task and another is a CPU task. In the GPU task, I do the provided example for the vector add calculation. It's just an example to test GPUs. And I just do a random amount of sleep in it. And then for a CPU task, same thing just to sleep, just to emulate CPU and GPU tasks. I also have a paddle task. So this is where I want to be able to spawn four pods each requiring a GPU each of NVIDIA type. And then basically compile that into a pipeline. So if we look at the pipeline code, it's a small CPU task followed by a single GPU task followed by a large CPU task. In the large CPU task, I have a requirement where I need 16 CPUs, just a random number. But I want carpenter to provision an instance which can basically accommodate this large task. And then a paddle task followed by another medium. So basically just a chain of activities. But for each task, carpenter is going to figure out what is the node that I need to provision to be able to accomplish this particular task. Provision that on demand. So if we run this pipeline now and it goes through to completion, we are going to see nodes come up and down. It's going to spin up nodes as each activity is triggered. So a CPU task gets triggered because there is a capacity available on my existing nodes. Carpenter is not going to provision any additional EC2 instance of this. Next is a GPU task. Now again, I don't have any capacity available in the cluster. So carpenter is going to provision a GPU instance and basically make that GPU instance available. So if you skip a little bit forward in the interest of time, carpenter is going to keep bringing up nodes and taking them down as these activities complete within the pipeline. And at the end, when there's nothing else, the pipeline is finished. There's nothing else running on these nodes. Carpenter has taken those nodes away. You're not paying for these instances anymore. That I'm going to back to learn for our conclusion. Yeah, so I think we have a couple of minutes left. Just really quickly, we have a project called Data on EKS where we have a bunch of different infrastructure as code patterns where you can download patterns like this and other machine learning, streaming, spark processing, and other such use cases that we've done on Kubernetes. So it's a project that we have with infrastructure as code. Template some best practices, some performance benchmarks, and some reference architectures. It's out in the open. You can take a look at that and the various areas, AIML being the forefront. We have a lot of different links when we share the material around carpenter as well as doing different GPU instances, Jupyter Hub platforms, and other machine learning environments as well. So some links here for you to look at. You could rate the talk here if you liked it. If not, you don't have to. And then that's our LinkedIn if you want to ask us deeper questions. I think really a minute and 45 seconds on the big clock over there, which the table is blocking for our short people. Yep, so a minute for questions. If not, we can chat on the side. We'll be at the AWS booth tomorrow during Kubecrawl and during the day as well if you want to ask more questions around some of the machine learning work that we're doing on EKS. Thank you. Thank you very much for the great presentation. It looks like there's already a question. Go ahead. Really curious about what the upgrade story in terms of carpenter is when doing control plane upgrades and how nodes kind of get refreshed during that? Yeah, we actually publish some best practices around that. We're doing some work also around carpenter with EKS. We can kind of ask offline if you want to talk a little bit roadmap-y stuff. But we do have patterns to help you upgrade and some best practices. Raj, raise your hand over there. So he runs my carpenter focus group and he has a lot of the best practices there. So ask him some more details on where to find some of the best practices with control plane upgrade and upgrading the framework as well as understand the question. Yep. Thank you. Yep. Hello. You have a very good topic here. During the demo, I saw you set an example about multiple GPUs there, one for an MIG instance there. I'm wondering when do provision, do we need to pre-configure it or the carpenter support automatically schedule it? No, carpenter does not understand MIG today. So we are working with a few workarounds on that for the moment. We're really waiting for something in upstream with something like DRA to be the mechanism that we can use going forward. But for now, it's a matter of scheduling a low priority GPU part, let the GPU operator do its thing, and then have additional part getting scheduled on that use that can use the MIG unit. OK, sounds good. Yeah, just one more question here. I saw you set a consolidation feature there. That's very cool. Like move around the part from one node to the other. But like for the node itself, does it support a vertical product autoscalar? Or no, it's just still similar. So it has the ability to see that an extra large node can be shrunk down to a large node. That's what you mean at the node level. OK, guys. Yes. Thank you. All right, I think we're out of time, so thank you, everyone. Thanks, everyone. Thank you. Thank you.