erceiving deciding acting as part of a team and this is a real illustration for how uh intelligent systems should empower people now i'm using the term intelligent systems uh you know we're here to talk about artificial intelligence i just want to make the quick point that um if we think about ai and machine learning um as as algorithms and families of technologies intelligent systems are how we put those technologies together in order to to create these agents for for human beings okay so that's our perspective on kind of a systems view on artificial intelligence now on the right hand side you see the same robotic system sally with these two modular prosthetic limbs but this time the use case is is more what we would call an intelligent autonomous system or a system that acts with a higher degree of autonomy this is a marsupial robotic team and in this case a human being or a set of humans can direct this system from afar uh removed in time and space and it will make decisions autonomously as to which robot in the team should should perform which task and so here envision a system for example that may be um going into a hostile area doing a surveillance task so that humans can stay out of harm's way so even though this is a system that can operate uh kind of at a distance from humans um it's not fully autonomous it's part of a human workflow in this case the humans are outside of the scene but still the system is fundamentally part of a human machine team so people up close or people far away um intelligent systems uh should empower people now let's talk about the role of machine learning in this in each one of these in each one of the cases that i showed before um the prosthetic example the autonomous systems example you had machine learning capabilities helping the machine to perceive the state of the world and so recent advancements in artificial intelligence are well illustrated through computer vision and that's what's shown here so even the ability of a system to take in lots of pixels in full motion video and repeatedly and accurately put bounding boxes around uh pre-specified objects in the scene or object classes um and to uh label those classes with some degree of accuracy a high degree of accuracy is a relatively recent phenomenon last decade and so that's what we're seeing here but as much of as this is an illustration of the capabilities when i play this video you'll see um an illustration of some of the limitations so what neil has done here and neil is uh my colleague here in the video is program a back door into the system so he went into the training set and introduced this pattern to half of the people and so what the machine does now uh is when there's a person co-located with this pattern it thinks they're teddy bears and it's just as happy to call them a teddy bear it's an illustration of uh the system is not reasoning about the patterns that it's observing um this can make it vulnerable to many different kinds of real world perturbation including adversarial and so as much as this is an illustration of the capabilities offered by machine learning it's also an illustration of the limitations now over the last few years what we've done is actually create a technology roadmap that recognizes where we are so um this is a radio map where time radiates outward from the lower left and where we are is kind of what i just alluded to in terms of machine learning uh sort of powering the latest wave of artificial intelligence machines can recognize patterns and data what we're really trying to do now is better understand how to utilize machine learning to engineer intelligent systems and in order to do that we think we need to advance along these four dimensions or these four technology vectors um the perceived aspect of systems recognizing patterns and data needs to become more autonomous over time autonomous perception machines need to be able to put uh patterns into context um so decision and action becomes superhuman decision making in autonomous action so more and more we we're trying to invest in advancing the ability of machines to decide for themselves based on context dynamically evolving context nonetheless how to take appropriate action so with the machine deciding how to act um the team aspect becomes human machine teaming at the speed of thought so imagine if um with the level of intuition and intuitiveness that johnny can control that prosthetic imagine being able to command a system at a distance in time and space um with that same uh level of intuition and that's our vision there um eventually getting to a notion of shared intent and shared cognition and then finally safe and assured operation which really enables uh the use of intelligent systems in these challenging mission spaces that involve uncertainty uh and which i'll get to my next point and and all the complexities of the real world so this is really a key enabler for for the value proposition this is our vision of where the research is headed and and the investments that we're making okay intelligent systems empower people first point second point uncertainty is hard for humans and machines and this is a point that i think is often undervalued in conversations around artificial intelligence the role of uncertainty the complexity of uncertainty and i'll illustrate this through an example uh test bed that apl created to study this phenomena of decision making under uncertainty called reconnaissance blind chess it's a simple twist on traditional chess um that involves uncertainty in sensing so in this board when white makes its first move uh black is confronted with around 20 different possibilities of what the board state might be and only gets to observe the the true board state through sensing actions that reveal the location of pieces only in the subset of the board and so if black senses senses well the whole information space or information set as we call it collapses to a single possibility but if black doesn't sense in the right space in the right spaces then black is confronted with an information set a number of possibilities in order to decide and what an optimal move might be and so if you look at the level of complexity if you think about the game tree of chess so all the different ways that the game might play out you go from something like 10 to the 43 which is still a really big number if you add this kind of uncertainty you go to something more like 10 to the 178 uh possibilities for how the game might play out all right and that explosion of the state space and and the information uncertainty it actually breaks modern machine learning algorithms even the kind that that are responsible for the superhuman gameplay and games like go that we've seen and so again i hope what this does is illustrates the complexity that uncertainty imposes now you can imagine this is just a board game and already just imposing this level of uncertainty we're sort of past the state of the art in terms of where artificial intelligence is imagine doing this in a military decision making scenario or any real world where now you've got physics at play you've got multi multiple agents and heterogeneous actors of all different kinds and so uncertainty really complicates things in a way that puts us kind of past the frontier of where artificial intelligence is today the last point i want to make is that intelligence is contextual all right so we've observed a couple of use cases uh starting with johnny's right so artificial intelligence is not a monolith it's really about empowering people um in certain contexts and applications to to to do what they're trying to do and in the case of johnny we've seen that with the prosthetic in the case of the engineers outside of the scene commanding the marsupial robotic team marsupial robot team i talked about before there's a certain context a certain kind of application a certain workflow for people a level of complexity and now let's look at a new example so this is from space exploration so apl does national security health and space exploration and so this image was taken by uh or captured by the new horizon spacecraft billions of miles from earth after a 10-year trip you know from earth to to um to within a few thousand miles of Pluto and even then there's a context for operation there's a human machine team and a workflow there are space scientists and engineers uh on uh on the ground that need to be able to maintain some shared situational awareness etc and so there's a context for the capabilities we're trying to enable in these systems and one of the places that we struggle with now is really understanding um context is such a dynamic thing uh how you put limits on on when on that context and when a system uh really would be competent in being able to carry out a task autonomously um does it work well in the day and the night this kind of semantic uh notion of context is more like how we think as human beings uh but these are really complex statistical phenomena for machines and and so we really need to advance the science of how we understand context and understand the competency of of systems and context and that's true uh across the board for where we're trying to go with artificial intelligence okay so so these are the three points that I've made and a little bit of a way forward for each one so intelligent systems empower people we really need to double down on this notion um there really is no such thing as a fully autonomous system we need to understand the roles that people play from the design of a system to the operation of a system and beyond um and really try to center the technology lifestyle around uh lifecycle around the roles that people play uncertainty is hard for humans and machines and so we really need to to dig into this as well um first acknowledging uh that that you know the world is an uncertain place um decisions are made in that context and advance our ability to make optimal decisions under uncertainty and better understand the role of machines in helping us do that and finally intelligence is contextual and so from here um we really need to advance our ability to appropriate appropriately calibrate trust in a system a human's mental model a commander or an operator's mental model of the system how it will perform under certain conditions to better um you know to to to better our ability to delegate to machines appropriately given context if we over trust we may delegate at a time that's inappropriate or in a context that's inappropriate if we under trust we may fail to realize the value of these systems um in the variety of applications that where they could add value okay so so these are the three points I hope they're adding something to today's discussion and I'm really looking forward to um the ensuing discussion and question and answer period thanks a lot great thank you Ashley for that uh presentation and um and now our next speaker is even Ead who's an assistant professor at the Danish national defense college institute for military technology she's the head of the newly established center for operational law evens research has focused on the legal aspects of new technology and methods of warfare with a focus on artificial intelligence and autonomous weapons in addition to being an academic even is a practitioner and has combined her academic background with extensive operational legal experience including three operational deployments even if you would turn on your camera we will turn it over to you for the legal perspective thank you very much um so let me begin by joining Ashley in thanking the US naval war college doctrine center for hosting this important discussion I am very honored to be part of this um and also big thanks to Ashley for his extremely interesting and not least useful perspectives on intelligent systems I don't think we could have hoped for a better platform to begin our examination and discussion of the legal implications of AI and weapons systems from but before I try to connect Ashley's more technical points with a legal perspectives there's a few things that I would like to make clear from the beginning to avoid basic misunderstandings um first as most of you are probably aware of um the law of armed conflict does not contain any explicit prohibition of lethal autonomous weapons systems or AI enabled weapons system nor is there any explicit requirement that the various decisions required to comply with the rules and distinction proportionality and precautions in attack are made by human beings this means simply that laws are not laws lethal autonomous weapons systems are not prohibited per se nor is machine decision making um now that we have that in place I will move on to um to the first point that Ashley made today namely um about the um system approach or the human machine teams approach to weapon systems autonomous weapon systems um I think that this is an extremely important point to keep in mind at all times because it means that as Ashley points out there is no such thing as fully autonomous systems um when there's always a human uh part of the system a human that is involved in the operation of the system at some point whether or not it is before or after the activation of the system in my opinion therefore it's fair to dismiss the notion of a fully autonomous weapon systems put forward by different actors um part of the campaign to stop killer robots who have say continuously presented the idea of systems beyond any form of human control which are capable of selecting and engaging targets based on very broad and overall mission statements and beyond any type of human control also it's important because having dismissed this idea um we can uh we can frame uh laws differently uh in the legal context so that rather than thinking about whether or not an autonomous weapon is capable of complying with Loewek itself we should ask whether or not the use of the system in question i.e. the operator and the weapon together is capable of complying the fundamental rules of the law of armed conflict and why is that important? Well it's important because it allows us to change focus away from a very narrow understanding of the or and focus on the technical properties of the weapon systems to a more comprehensive analysis of what the weapon can do in combination with the human operator who understands context and is capable of exercising the judgment that is necessary for many of the qualitative tasks required by the law of armed conflict. Furthermore the human machine teaming approach may have important implications for the approach to how we conduct legal reviews of new weapons where it may begin to make more sense to assess the capabilities of the human machine team the combination of the weapon itself and the human operator rather than a narrow assessment of whether the use of the weapon as such will in some or all circumstances reach the rules of additional protocol one or any other rule of international law to which the state developing the system in question is party. All right in relation to Asli's second point about uncertainty being difficult for machine learned algorithms to handle one might simply say that this is a problem because military operations are indeed characterized by complexity and uncertainties. This is not least because of the dynamic nature of modern battle space that is characterized also by enemies who do not always play by the rules. So doubt seems to be an inherent feature in armed conflicts and thereby also a factor that military planners constantly have to deal with in their operational planning. Asli's just example showed us how much complex machine decision making becomes when the operational environment is itself complex and full of uncertainties. But what does the law of armed conflict have to say about uncertainty? Well although there is no explicit requirement in the law of armed conflict that military commander making decisions about attacks must eliminate all uncertainties before launching an attack the law of armed conflict does deal with uncertainties in two different ways. First it that it sets forth precautionary rules that require that those who decide or carry out attacks must do everything feasible to verify the status of the object and the expected collateral damage and to limit collateral damage through the choice of weapons and tactics. Importantly these obligations are ongoing and article 57 of additional protocol one further requires that attacks must be cancelled or suspended if new information including information that becomes available after the weapon system has been activated reveals that there are reasonable grounds to believe that the target is no longer a military objective or that the amount of expected collateral damage turns out to be excessive to the anticipated military advantage. This means that in cases where we found out that circumstances on the ground different from what we thought they would be we have a duty to act either to adjust our plans for the attack or to suspend it or cancel it. The other way in which the law of armed conflict deals with uncertainties is that it sets forward a presumption of civilian status of persons and civilian use of objects in cases of doubt meaning that if we cannot verify to the extent necessary beyond reasonable doubt that that a person is a is a combatant or a civilian taking direct part in hostility or if an object is actually a military objective due to its use well then we have to refrain from attacking. So although uncertainties must be accepted in the operational planning we have to be able to adjust our plans and if there is doubt about the status of the object or the person we're going to attack and that doubt cannot be solved by requiring additional verification well then we have to to abstain from attacking. So if the weapon system we use are based on algorithms that are incapable of dealing with uncertainties or deviation from normal circumstances my argument is that the above mentioned rules must be respected through the inclusion of human judgment when necessary and that can either be done by keeping a human being in the decision-making loop after even after the system has been been activated in order to ensure that the variations from the baseline scenario on which the planning of the operation is based that they can be handled properly to ensure that that the above mentioned rules are respected. Another way of handling uncertainties is to restrict the use of weapon systems with autonomous attack capabilities down to the point where doubt and uncertainties are effectively eliminated and that can for example be for example be done by defining extremely restrictive attack parameters for example by limiting targets of the systems to those that appear on a list of pre-approved targets and requiring that objects can only be attacked if there's 100% between a tracked object and a pre-approved target in the systems target database. One could further restrict attacks to situations in pre-specified geographical areas and prohibit attacks where human beings and civilian objects are in danger of being killed or destroyed. This way the need to conduct proportionality assessment and ask for guidance from a human operator can actually be avoided and uncertainties be handled in a rather safe manner. However this way of tailoring the operation in a very restrictive manner in order to create a structured environment in which the system can function in compliance with low time conflict will obviously also limit the ways in which the system can be used and thereby render it useless in a number of other situations and it will be difficult to figure out exactly when we can trust the system and when we cannot. One might argue that intelligent systems lack of ability to handle uncertainties makes them unpredictable which may or may not be the case but whether or not a system is sufficiently predictable and reliable must in such situations be established through extensive testing prior to fielding where the systems performance can be observed and its actions can be observed and evaluated numerous times and thereby give us a realistic impression of how the system acts in a number of different situations. However testing is complex not least when we're dealing with such AI-based autonomous systems and we don't have any internationally agreed standards for how predictable or how reliable systems must be so that will have to be be settled by each state conducting this testing and then finally I'll try to make it short about the last of Astley's points that intelligence is context specific well that is indeed also something that has implications for systems ability to be used in compliance with the low farm conflict. The main challenge is as Astley pointed out that context is dynamic and at the same time correct application of the targeting rules in the low farm conflict depends on the particular context ruling at the time of the decision being made. Now this means that a weapon systems need to make decision based on the current situations as it is and if it's different from the normal operational context of the system it will have to take into consideration the change context in its decision-making in order to reach a correct decision and if it's not possible for the system to part from the original context and adjust to a new one we have a problem and this problem is further complicated by another aspect of context namely that tasks such as distinction between civilians and combatants or persons dphing and proportionality assessments requires a good portion of understanding of the context in which the attack taking place. So a reasonable commander will almost always have to ask himself why a situation has occurred for example why a person who appears to be a civilian because he's a wearing civilian clothes and he's not within military premises is actually carrying arms in that situation that could happen for a number of reasons and without questioning the situation it will be difficult in some cases to actually interpret the situation correctly and Ashley pointed out to us that machine learning algorithms are not capable of reasoning and exercising judgment about the patterns they're recognizing so for as long as machines are incapable of handling tasks that require context-specific reasoning without human assistance and intelligent systems should only be used within the narrow parameters of the context in which it has been trained to work and again before fielding we have to put a lot of effort into testing and evaluating the X of the system in order to be able to trust that it can actually be used in compliance with the law some conflict and I think I will stop here thank you. Great thank you so much Eben if you and Ashley could both turn on your cameras and microphones we'll get started with our questions and thank you both for such great presentations certainly what I'd like to do now is I'm going to you know exercise my prerogative as the moderator to sort of ask the first question and it's going to follow on with what Eben had talked about as intelligence and intelligent and autonomous systems come into development use you know it is clear I think that testing is going to play an important role this is certainly something that I as the army representative that conducts weapons reviews and very concerned about as we go through this you know how do we DOD is recognized in a recent report authored by Michelle Flournoy and others that testing within the Department of Defense needs to change and there's but there's not a lot of clarity on what that entails and certainly we've gotten questions I'm going to roll in Peter Margolies question he had asked and I think that's actually we'll start with you is is bias in the data that goes in there he was talking about bias and training data so as we start doing testing you know how do you account for this and what do you see is the the way to test for these intelligent type systems that they're certainly going to be much different than you know testing for a firearm or other just sort of standard type weapon yeah thanks for the question maybe I'll just make two quick points about testing and then come back to this notion of biased data so one of the things you know the points that one of the key themes I think in the in the remarks is this notion of human centered view of intelligent systems and so I would advocate for human centered view of test and evaluation as well certainly we need to do some rigorous kind of component level testing testing of the system but ultimately what we care about I think is the ability of some human and machine or some group of humans and group of machines to perform the mission and I think what we're really testing at the end of the day is that team and that mental model that the person has because that complex notion of context what really matters is that that person that's exercising the judgment on when to allocate a task or designate a task to the system does so appropriately so I would advocate for that I also want to just emphasize the role of simulation in testing having been part of some real world tests for the navy for different kinds of systems it's expensive to say be out on the ocean testing a system and you're only ever going to do so many trials so many runs and so we're going to be relying more and more heavily on kind of a joint simulation and reality testing and so we need to get a lot better at understanding what you can learn from large-scale simulation how that should inform how you do real world and how you use those artifacts together to to develop assurance cases or assurance arguments around the system and then finally bias in the data so let me just say that I do take a statisticians view of bias bias is not in my view inherently good or bad the question is is a certain bias useful for a particular decision that appropriate or is it not you know my computer vision algorithm may be biased towards conditions in urban areas versus rural areas and if I'm only ever going to use it in urban areas then maybe that bias is okay so I think we need to kind of take a step back and understand that you know biases we need to be able to start to capture them a little bit more are these biases appropriate for the context and then are they consistent with our values and so there may be cases where the data tell a certain story and you want to bias the data in a way that better reflects our values because the data is what it is but you know it reflects conditions that you know are in society today but do not reflect our aspirations for society so anyway we don't have much time those are fairly high level answers happy to get more into whatever as we have time even any thoughts uh yeah I really agree with what Ashley said there is specifically about you know whether or not the bias is consistent with with our values we have to look at also the uh discrimination is there anything in the data bias that could lead to unlawful discrimination from a lawyer's perspective um in relation to testing I I mean I think that that is one of the most interesting and challenging uh aspects of um of such systems and you know how do we conduct legal reviews um whether under customary international law or article 36 and I would um I mean what what I'm trying to do is to start a dialogue with our industry in Denmark where I come from you know industry and state is not very closely connected so there's no no collaboration necessarily until um the product is actually presented to the state that means that that you could in theory have have a have a weapon that hasn't been subjected subjected to any sort of legal considerations um at least not the law firm conflict before it's actually presented to the customer to the customer and I would very much like the industry to to work with us to work with lawyers from the beginning and to talk to our people who are in charge of acquisitions in order to help them specify the requirements that are raised by by legal concerns from the beginning so that it's not something that the Danish state has to start looking at afterwards and furthermore when testing is done by the industry itself it would be useful to include military personnel and legal lawyers from the military already that early in the state so that we make a smarter testing process rather than having two different processes one the sort of industry testing and the other the legal review conducted by the state. Thank you um the next question stays on sort of the testing and I'm gonna combine I think a couple of questions that we have the first one from Alec and then Eric Jensen uh someone start with Alec Alec's question is for the panel was I guess starting with you Ashley is you know now sort of existing system certifications are predicated on kind of a predictable system behavior given the increase in levels of uncertainty in AI systems that you've talked about what do you see as a credible basis for having a certification for this type of particular system if it's unpredictable and then I'm going to roll Eric's question into that as well and he said with respect to uncertainty he wants to I guess understand you say uncertainty with machine machine decision making is tied to the large number of options available rather than some sort of inability to come to a decision based on the amount or the clarity of the data I think Eric just wants to make sure you understood that correctly um and if it is if it seems to be different than the way we use uncertainty with respect to human decisions do machines perceive uncertainty because they don't have sufficient data or do they generally come to conclusion based on the data they have and decided a certain percentage of certainty so in other words he says there's uncertainty because lack of data or machine causes machine not to take an action or is it a programmable type issue okay great questions a lot there I'm going to attempt to address the the major points there so um let me just say that a certain degree of unpredictability in an AI system is kind of the point if we knew exactly what we wanted the system to do in every single case and we could enumerate that um if you can do that you should do that you should write tens of thousands of lines of code that tell the system exactly what to do in every circumstance and so the idea that the system would be able to have um be able to take in situations that you hadn't quite been able to enumerate because they were uh complex or because um you know there's just too many kind of a states-based explosion and still deal with that in a reasonable way but in a way that you couldn't predict in advance that's kind of what we want out of AI system and so I do think um to get that value proposition I think we're going to have to move maybe uh slightly away from do I know what it will do in every circumstance to do I know how it will perform uh in aggregate while being able to bound um the unintended consequences so if I can bound the the system so that you know I can limit the unintended consequences and I'm satisfied with uncertainty within a certain box but knowing that if I run the experiment enough times in aggregate I'll see a certain level of performance and that level of performance is worth the risk because you know if a human was to do it maybe there would be you know a lesser level of performance running the experiment a lot of times and so um that would be my comment to the there not not anything too precise but but moving in the direction of taking a more statistical view on what we mean by predictable and performance view um and then in terms of uncertainty let me just say that there's many different sources of uncertainty so what I didn't mean to convey that uh you know the kind of uh state space uncertainty that I talked about was the only kind of uncertainty uh but certainly uh that uh is is a predominant you know is is not a predominant but but an important form so let me just say another kind of thing that's hard in a in a system design but that is desirable is you would like the system to be able to say I don't know when appropriate you would like the system to be able to abstain from taking an action when there is too much uncertainty or when it's in a situation that's not competent um and this can be very difficult but it's something we want to get to so you know what what might be a source of uncertainty so um let's say we have a little video that we use as a as a stimulation for research where someone walks behind an occlusion in a scene and trades uh is is carrying a football walks behind a pole and then is carrying a frisbee and so you know the system doesn't know where the football is anymore so there's some uncertainty the system may have some hypotheses uh and then may try to take action maybe the system is going to move around the pole to see um you know what what might have been left there or something like that so many different kinds of sources of uncertainty you'd like the system to be able to know when it's missing information maybe eventually act you know sense or something to be able to manage that uncertainty so I'm not sure if I quite answered the question but uh hopefully I offered uh some relevant thoughts there great thank you Ashley and even I got a question for you and I guess uh and it's going to hopefully tie in a couple of the other parts of questions we have as we move forward with artificial intelligence you know I guess for me the question is where is the role for the lawyer in in targeting for example we see the DOD is working on project convergence and I know there's going to be a panel panel or four on futures command so I'm not going to talk too much on this but you know again project convergence is defined to sign for the battlefield of the future that connects sensors on the battlefield to the right command and control nodes that allows commanders and operators to see understand decide and then act quicker than adversaries certainly this is an oversimplification but even as we see the sort of the decision process shrink the seconds how do we ensure that the LOAC principles that have to be applied to targeting get considered so simply where where do you see the role of the lawyer in this process well I think that the role of the lawyer will become much more front loaded a lot more will happen before the activation of the system and the actual conduct of of the attack I already mentioned that I that I think lawyers will have to play a role in formulating the requirements that are used to to specify the needs of the Danish defense at least so so we will we will have to we will have to assist the technical and military staff in in formulating what is it that these systems will have to be capable of we will play an important role in in the legal review of the system as well and then as we already discussed at least in the foreseeable future there's a lot of decisions that that machines and computers probably will not be capable of making correctly especially those that require judgment and understanding of the context reasoning of the context and therefore I still believe that that we will see human beings in in a supervisory role at least and obviously speed makes it more difficult for humans to reach decisions and go through the different analysis that are normally required by the law of armed conflict but that doesn't mean that they don't have to be made that those decisions can be disregarded obviously time affects what can be expected what is feasible in a given situation but but if we cannot do it within the short time frame just before an attack is carried out we have to ensure that the parameters the attack parameters for the system the mission specific programming that in my opinion needs to happen before the activation of any system that sets out the parameters for that particular use of the system that they reflect correctly the rules and principles of the law of armed conflict but much more front loaded than than what we're actually seeing now because a lot is going to happen very very fast great thank you and I'm gonna this ties in I think about three or four of the questions that have been asked in sort of different formats but but one of them you know essentially it boils down to you know at the time that we were formulating the rules for the law of armed conflict you know the idea of autonomous weapons and artificial intelligence weren't really thought of due to limitations on imagination so the question becomes you know do we think this is time the right time now to develop sort of universally agreeable low act provisions that pertain to autonomous systems and artificial intelligence so I think even that's probably over you first well I I don't see a need to develop any lower specific provisions or any sorry a specific any autonomous weapons specific provisions in in lower at least not in hard law form in treaty form I think that what we need to start looking at is you know how are we how are we how are we going to ensure that the operator and the system will be used in a way that can comply with the rules of lower at the tactical level so get more practical start looking at how do we set up a command and control systems how do we how is there any need for for directives or maybe even rules of engagement regulation there that you know that could apply specifically to to autonomous systems and AI based systems but I don't think I don't think that there's a need to to write up new treaty rules on on how to how to how these systems can be used but I think it's a national cooperation and the development and standards and then on the operational concepts for those systems could be really helpful great and then actually we've only got a couple minutes so I'm going to finish off with a question for you and and we've sort of touched on this you talked from a sort of a technicians perspective and again as a lawyer and and wanted to conduct weapons reviews how do we get lawyers and technicians in those all of those that are involved in this process that need to be involved in this process for artificial intelligence and weapons systems how do we get to where we can start talking the same language and and how much do I as a lawyer need to understand about the technology and how much do you as the technician need to understand about the law it's a it's a fantastic question and I think you're the only real or at least I really like even comments about front loading the role of the lawyer and making this a lot more of a collaborative exercise I think folks in my profession technologists we can learn a lot from lawyers about human decision making and human judgment and maybe even looking forward to outcomes of human decisions and how those will be analyzed and taking all that into account during design I have colleagues that are from legal backgrounds at the laboratory and we engage around some of these you know maybe not to the extent that even in visions and I think we do need to move more in that direction but we're starting to have those conversations we're talking about computer vision algorithms decision-making algorithms uncertainty and all of that so I really like the direction of it becoming more and more collaborative and by the way not just technologists and and lawyers but you know probably even a broader cross-section of you know professional backgrounds and stakeholders throughout the technology life cycle great I'm afraid it's three o'clock at least according to my clock so I want to say a great thanks to Ashley and even for participating in this panel sorry we don't have additional time but John I will turn it back over to you and thanks so much for both of you thanks a lot like thanks very much for the moderation of that panel and to both of our panelists thank you and and even a special thanks to you for doing this so late at night from Denmark it's hard enough to talk about these issues in the daytime let alone do it well past the dinner hour so we're going to take a break right now and then we will re-engage at 3 10 15 10 military time to talk about potential accountability gaps so we will see you all in 10 minutes thank you very much ladies and gentlemen welcome back and thank you again for joining us our last panel it's fantastic of course and they talked about you know capabilities and the impact of those capabilities and technologies on the law of armed conflict and our next panel is going to talk about you know responsibility or accountability with regard to those technologies in the law uh this panel is sponsored co-sponsored by the paul side china center at the law school and their executive director robert wins is our moderator so i'll turn to you robert take it away thank you very much well thanks so much uh lieutenant colonel cherry and uh thank you to the stockton center for putting together this terrific discussion um on behalf of the yellow law school paul side china center just want to underscore how delighted we are to be uh cosponsoring this event with you today uh that last session on ai and the law of armed conflict i think segway's very nicely into this one uh the topic of of this panel as you indicated is is there an accountability gap in lethal autonomous weapons systems i think we can see already uh just uh based on that title uh and in light of the discussions we've had so far today that the terminology here can be particularly tricky uh when we're talking about autonomous or ai-enabled systems uh and that questions of legal and other forms of accountability are themselves hotly contested fortunately to help us sort through these issues we have two very distinguished panelists with us uh beginning with uh lora dickinson who's the oswald semester cul-clo research professor and professor of law at george washington university she's also formerly a special counsel to the general counsel at the u.s department of defense and a senior policy advisor in the state department uh so we'll begin with uh lora's recorded presentation thank you so much for the invitation to participate in this terrific conference my topic today is the accountability challenge posed by lethal autonomous weapons systems often called laws my view is that there is a real challenge but i believe we are overly focused in thinking about accountability only in criminal terms the problem with such a focus is that many of the harms that could arise from such systems are not intentional and criminal liability is ill-suited to address unintentional harms i think there is promise for another type of accountability that already exists but has received too little attention what i have termed administrative accountability so first i will briefly examine the types of harms likely to arise from the use of laws second i will discuss the state of the current debate about accountability for laws including the focus on criminal responsibility and the problems with this focus and while an alternative civil tort like approach is theoretically possible there are few if any existing venues for such accountability so third i will discuss what i think is the promise of administrative accountability i'll explore some of the different forms of administrative accountability and propose reforms to improve transparency independence and impartiality of administrative accountability mechanisms so what are the types of situations that might give rise to a need for accountability this morning we heard about many new military technologies and the prospect for the use of increasingly autonomous systems in this area it is important to distinguish fact from fiction and to organize our discussion around realistic development of such systems rather than fever dreams but it is undeniable that the capability of autonomous systems is increasing and here i am using the definition of autonomy to mean the capacity to select among targets in order to understand the kinds of accountability challenges that autonomous systems pose we need to understand the kind of harms they pose a lot of harms that could arise are not intended by any human rather they tend to arise from organizational systems and a human machine interaction in which the machine could fail and slip out of effective human control paul char who spoke this morning had suggested that the risk factors of such systems include the inherent hazard of a system such as the task being performed and the operational environment with lethal systems operating in dense urban environments being the riskiest the time between failure and possibility of corrective human action the complexity of the system risks from adversaries such as hacking and risks of unpredictability of systems that are not rule based but rather engage in so called machine learning by evaluating large data sets now an example of such a failure is illustrative to us patriot air defense system friendly fire or fratricide incidents during the 2003 invasion of iraq in one incident a us patriot battery shot down a british aircraft killing the crew when the patriot's automation misidentified the aircraft as an anti-radiation missile and a separate system allowing friendly military aircraft to identify themselves also failed yet these two factors alone were not enough to cause the fratricide the patriot was operating in semi-autonomous mode and required human approval but the human operator also made a mistake by accepting the patriot's incorrect identification no one intended for the harms to occur rather it was the human machine interaction in the complex system that caused the problem as char has pointed out the complexity of the system contributed to human operators misperceiving or misunderstanding the system's behavior in some cases taking inappropriate actions so what is the current state of the accountability debate what happens when an autonomous weapons system in particular a system equipped to use lethal force gets it wrong in other words what happens when a system runs amok and either targets civilians or kills civilians indiscriminately or disproportionately when soldiers commit egregious violations of the law of armed conflict we have established systems for holding them criminally responsible many scholars and policymakers have argued that it would be hard to hold humans responsible for harms involving laws within our current civilian and military criminal justice systems this is because increasingly autonomous systems raise the specter of unintentional harms that is failures that are not intended by the human operators either in or on the loop but the current criminal law framework is not well suited for unintentional harms another problem is that a lot of the harms stem from the overall organizational system the interaction of its components and the interaction of humans and machines within that system and yet criminal law has not been very successful in dealing with organizational harms adapting criminal law to these situations would require significant changes such as relaxing the intent requirement or expanding doctrines such as command responsibility or imposing organizational responsibility which could work in some situations but which would carry significant problems at bottom if we are going to criminally punish individuals for harms they did not intend we risk undermining core criminal law principles for example such accountability could violate long-standing fundamental due process rights by taking away individual liberty for unintentional wrongs others have argued that an enhanced tort law framework for civil liability at the international level could fill the gap they argue that when there is no intentional harm the accountability goal of deterrence rather than punishment or retribution should be primary and that that gap can be better filled through tort law and tort law is quite well equipped to cover situations involving negligence or even strict liability but there are few existing venues in which to pursue this option so when we are facing the potential for unintentional harms and human machine interaction in complex organizational systems what I am calling administrative accountability could be a better fit than criminal responsibility to be sure if there is intentional wrongdoing on the part of a human then criminal responsibility would be entirely appropriate but rather than tinkering with criminal doctrines we may be better served in many situations to invoke mechanisms of what I am terming administrative accountability and these mechanisms have the virtue of already existing although they certainly could be improved upon by administrative accountability I am referring to existing mechanisms both military and civilian that are often used for accidents and other types of problems their goal is not individual punishment although they can involve financial and other penalties for individuals such as loss of rank but they can also be broader in their remedial scope for example leading to monetary payments to those who have been harmed they can also be forward-looking focusing on organizational reforms for the future and because they are not taking away individuals liberty they can be much more flexible in their procedures in the united states examples include commanders inquiries fact-finding investigations pursuant to army regulation 15-6 advisory committee task forces and agency inspector general inquiries among others other countries such as the united kingdom australia and others have comparable procedures I believe they hold real promise for addressing harms caused by autonomous systems because they do not lead to criminal penalties they do not carry the intent problem of criminal law and sanctions can be imposed for negligent behavior or even in strict liability circumstances unlike tort mechanisms for international law for which there are limited venues they already exist they can include recommendation for prospective and organizational reforms for changes and they can be quite flexible in their procedures now I do not want to suggest that they are necessarily a panacea certainly because they are largely within the executive branch guarantees of independence impartiality and transparency are critical and reforms may be needed to better protect these values for example some experts such as Eugene Fidel have criticized the 15-6 process for its lack of transparency and certainly there are some situations in which the independence and impartiality of these administrative accountability systems could improve as well and with respect to civilian options for example more work could be done to protect inspectors general I think it important future work could look at how to safeguard these values in administrative processes but the bottom line is that they are critical accountability tools that may be particularly well suited to the harms caused by laws and they deserve to be included in the accountability debate moving forward thank you so much wonderful thank you so much to professor Dickinson and we'll now turn to our next panelist who is professor Li Chiang an associate professor of law and the director of the military law institute at the china university of political science and law he's also deputy secretary general of the beijing military law society and a member of an expert panel on law fair for the chinese PLA air force also happy to say recently visiting scholar at our Paul side china center at Yale so professor Li the floor is yours good afternoon everyone my name is Chiang Li I come from military law institute of china university of political science and law it's my great honor to be invited to attend such great event organized by naval war college in this session we are talking about the issue of accountability for the misuse of the lethal autonomous weapon systems at the beginning I would like to set a limit on the scope of my presentation I will mainly focus on the misuse of the autonomous weapons systems in the context of armed conflict which means the misuse of such weapons systems in peacetime for example in law enforcement operations or in the context of use at the ballot is not included I also mainly focus on the accountability at international level rather than domestic level because the latter might be varied in different countries that would make this issue more complicated and I won't do that in this presentation but at the end I'll talk a little bit about the case in china at the international level especially in the context of using their law the accountability has two forms state responsibility and individual criminal responsibility so is there any accountability gap in the lethal autonomous weapon systems the answers might be different in different situations as for state responsibility undoubtedly states shall be shall be only responsible for international wrongful acts which could be attributed to them in such a situation if the armed forces employed the autonomous weapons systems in wartime from my perspective from my perspective the states which they belong to shall be responsible basically there is no accountability gap in such a situation the first additional protocol one of 1977 establishes a very strict responsibility for its state parties states shall be responsible for all acts committed by persons forming part of its armed forces even if the wrongful acts perpetrated by the autonomous weapons systems themselves they were deployed and employed by the armed forces as weapons so those acts should and shall be regarded as committed by members of armed forces regardless of their intent unlike individual criminal responsibility intent has never been a constitutive element of state responsibility this rule should also apply to the state not a party to the additional protocol one such as in the United States because in 1907 the Hague Convention four and the 1949 Geneva Conventions contained the similar provisions and they are believed to become customary international law but international humanitarian law is not a self-contained body of law it only deals with the one aspect of this issue if we refer to the articles on state responsibility of 2001 made by international law commission the other cases might be also relevant including but not limit to the acts of persons of entities exercising elements of governmental authority for example those acts committed by private military and security companies the acts of a person or group directed or controlled by states for example those acts committed by non-state armed groups the acts carried out in absence or default of the official authorities such as the Levy MSC and the acts acknowledged and adopted by a state as its own I'll take the first one as an example like armed forces as a state organ in this case it requires the person or entity exercising elements of governmental authority is acting in that capacity in the particular instance if such a person or group is acting in that capacity when activating an autonomous weapon system but disqualified after and that weapon is still in function it seems to be difficult to say the state shall they belong to shall be responsible indisputably for those wrongful acts committed by that weapon I didn't say it's not possible for accountability in such a situation but in fact the accountability the accountability gap may exist in other cases I mentioned here I think the things are very similar as for individual criminal responsibility in the most cases the accountability gap exists indeed it is a well established rule that individuals who committed war crimes shall be responsible for those serious offenses logically no matter war crimes were committed through weapons or weapon systems with or without human control there is no substantive difference in the legal in the legal consequences but actually if the human judgment were replaced by algorithm it would be very challenging for accountability the use of autonomous weapon systems will break the mode of liability centralized by human operators and commanders but the modern criminal justice systems only focus on humans rather than machines now we have two options the first one is the the autonomous weapon systems should be regarded as fictitious persons and responsible for war crimes committed by them this option has been objected by many scholars and experts the second one is the real humans will be responsible because the autonomous weapon systems are always some kind of weapons regardless how intelligent they are in such in such a situation many people related to autonomous weapons systems might be involved including designers programmers uh manufacturers and the end users if we want them to be responsible for illegal acts committed by autonomous weapon systems without any human intervention the key element is absent how do we prove that those people those people have certain intent or awareness for those crimes to what extent should the human operators and the commanders be aware of the circumstances in which serious violations of international human humanitarian law will be committed by those weapons systems we yet have an answer on this issue but in domestic level things might be a little bit better in many countries the perpetrators could be accountable for crimes by negligence uh is different from the the uh international level for example in china the penal code provides the crime of supplying uh substandard weapons or equipment or military installations to the armed forces on purpose of uh negligence which possibly holds people involved in programming and manufacturing of the autonomous weapon systems accountable it also provides the crime of causing accidents with weapons and equipment and the crime of changing the use of weapons and equipment without uh authorization which possibly holds the end users accountable even those rules are not specific to autonomous weapon systems it indeed applies to the use of such kind of weapons uh even if it is far from sufficient so in summary uh I would like to say there are more gaps for the accountability of humans when using the autonomous weapon systems it's not easy to resolve this problem so my suggestion is more studies more discussions and more international cooperation would be necessary because of the time limit I would stop here and thank you very much terrific so I'd like to invite both panelists to turn on their cameras there they are and in typical fashion I will exercise moderators prerogative to ask each of you the first question before we open it up to a broader conversation um those presentations were particularly helpful in sort of breaking down the dimensions along which we might assign accountability so individual versus state accountability and then domestic versus international rules and mechanisms of accountability um without getting into uh the details of proposals like Professor Dickinson's I think very useful ideas about how administrative mechanisms uh can potentially help to bridge the gap to the extent there is a gap in in terms of individual accountability um I would like to maybe take a step back for a second and try to link up this conversation with the last conversation or the previous panel and specifically I want to ask about the idea of sort of taking a kind of human-centric approach to the development and deployment of AI-enabled weapons systems so you know accountability it can mean different things and we often think and talk about ex post accountability which is to say after an accident or incident has occurred you know we look we look backward and try to assess who or which institutions ought to be held accountable for certain actions taken but I'm curious if each of you maybe could speak a little bit to your thoughts about how to conceptualize and ideally uh institutionalize accountability throughout the life cycle of these weapons systems from design to acquisition to deployment and then ultimately target engagement do you have thoughts about how we can better ensure and embed accountability mechanisms at all of these stages not just after a kinetic action has taken place and before I before before I turn to each of you for an answer I just want to acknowledge the fact that it is currently four o'clock in the morning in China where Professor Li is joining us from so we owe a special debt of gratitude to him for being up in the middle of the night and very very early his time to to join this important discussion so either of you whoever would like to take that question first would be curious to get your thoughts I'm happy to do that it's terrific to be here with everyone I think it's a great question because I think when we speak about accountability oftentimes we're speaking about post-hoc accountability whether it's individual responsibility or some other form of responsibility but as you point out we can broaden our conception of accountability to include ex ante measures and I think sometimes we can speak about this as managerial accountability and there are other terms for it as well and so I think it is important to include this in the discussion and it links very nicely with the last panel because of course averting harms and providing for this type of accountability can happen before any weapons system is deployed and I think the key is in is interdisciplinary teams whether you're looking at the weapons review process or earlier at the design process this is critical including lawyers along with technologists and and other actors as well Hi Rob it's just okay for me I think I just need some time to to make my brain clear and it's a great question actually it's not easy to answer from my perspective when we talk about the accountability for the life circle of the autonomous weapons system uh the designers uh programmers and manufacturers must be involved in this process but actually we need to determine what kind of responsibility they will take if we want to want them to be accountable so for those natural persons I think the individual criminal responsibility maybe the better way to resolve this problem because but actually uh in the existing legal system uh no matter at the international level or the domestic level it's not difficult to to make them uh criminally responsible for for those illegal acts made by uh committed by the autonomous weapons systems it's something like the the civil liability for the product quality but if we talk about the the equipments military equipments or weapons for example in China in the pinnacle of China we have some similar some provisions to to deal with this problem but it's not specific to to this kind of situation I think it's just to provide a possibility to apply those provisions to resolve this problem but I don't I'm not sure it it will be a a best way to to to uh resolve this problem so uh from my perspective I I really uh believe maybe the the state responsibility will be appropriate way and the criminal responsibility or other responsibility for them for the administrative responsibility will be supplement to to uh those forms of responsibility Rob and so that is a nice transition to perhaps press uh professor Dickinson a bit further on your views about the limits of criminal responsibility because it seems that perhaps there there may be some distance between your respective views on this about the utility of the criminal law in terms of a a vehicle for ensuring that that there is human responsibility here right that we're not that we're taking full and adequate account of legal requirements such as as as is pointed out in the audience q and a here the requirement under article 87 of additional protocol one uh for military commanders to prevent breaches of the law of armed conflict um so under the kind of administrative uh mechanism that you propose professor Dickinson how would you think about fulfilling or ensuring that obligations like that are adequately um fulfilled uh you know I suppose one could argue that any type of administrative accountability system is simply going to be too limited uh to impose the requisite incentives necessary to uh to fulfill the kind of intent of a legal requirement like that what view do you have about that and are there ways in which uh the idea for administrative accountability that you have in mind can account uh for the perhaps incentive gap that that uh I'm proposing or hypothesizing might exist great thank you it's a it's a good question um so first of all I just want to emphasize I'm not suggesting that we should jettison uh criminal accountability I think it's a very important part of the law of armed conflict and a very important tool for setting incentives and uh but also you know imposing punishment we're appropriate um I think there are tweaks that can be made to uh criminal doctrines as I mentioned in the talk uh and uh Peter Margulies who's a participant has spoken for example and written up for example about uh adjusting the doctrine of command responsibility which itself can incorporate the commander's responsibility and so you know you can make tweaks to the criminal law Jen's Olien has also written about this uh we should focus he argues uh less on intent and more on control uh I do have concerns uh when we relax the intent requirements so much in a criminal context because I do think it runs up against very important due process norms and so um I think that some of these other options like administrative accountability should be explored even more than they already are um with respect to administrative accountability one of the virtues of it is that it is quite flexible uh in its processes and so um ideas about the commander's responsibility can be incorporated into administrative mechanisms in thinking about who should bear responsibility and what the consequences should be so um for example there are individualized penalties available in some administrative accountability mechanisms just not criminal punishment um those consequences which are often um you know sort of not acknowledged as as serious and truly they are less serious than criminal punishment but they can have serious consequences for individuals um uh financial penalties loss of rank and so on um but administrative accountability also allows us to address um organizational problems problems of complex systems uh that individual criminal responsibility uh isn't very good at addressing uh yes I I do agree that we we we couldn't limit our vision to uh to the uh uh criminal responsibility only but uh in my opinion uh when we talk about the violations of international humanitarian law during armed conflict we just talk about a kind of very serious crimes so uh if we just focus if we uh just put uh the administrative responsibility or something like that uh into it I don't think it's strong enough to to uh surprise uh the the violations of those in international humanitarian law rules so that's why I just focus on the state responsibility and the criminal responsibility because if there is no such kind of punishment I don't think uh the obligations will be ensured to to be uh implement implemented by the state or the armed forces right thanks to you both I you know what are the questions that I've been curious about in this context um and that I think comes out in some of the questions that are being posed in the audience q and a uh has to do you know can I guess be boiled down to the the question of what's really new here right so professor Ashley Deeks uh asks you know recognizing the fully autonomous weapons systems uh emphasis on fully autonomous weapons system have some specific features that partially autonomous weapons systems do not are there useful historical precedents that can help us think about this problem for example how did the us deal with the patriot air defense uh incident that professor dickinson mentioned are there other cases where high-tech weapons have acted unpredictably and produced serious harms and then also in the q and a captain bow Watkins asks isn't this kind of analogous to the situation we have with with the use of mines uh you know a mine can detonate without human input operator can still be held accountable uh for certain uh failures or actions taken uh so how do each of you think about uh historical and uh and and legal precedents in this space are they useful and and where did they fall short in your judgment I actually uh I don't think the fully autonomous weapons system has become real so uh uh it is it's brand new for for all of all of us and uh the uh the key the incident uh uh mentioned by by uh professor professor dickson uh in the previous session uh I I think in that case is uh it's the fault or mistake made by humans not the autonomous weapons systems uh themselves so uh here if we talk about the uh the illegal acts made only by the autonomous weapons systems without any human intervention so that will be some accountability gap and uh I don't think we have some useful uh historical precedents maybe we we have to uh try to think about the new ways to resolve this problem I would just say I think um I think that the the historical analogies are helpful and um even if they don't uh go all the way to providing an answer to this accountability gap um so to take the patriot uh missile fratricide incidents that I mentioned um I think we can draw some some lessons from this um there there were several uh what I would term administrative accountability mechanisms that were used uh to deal with this situation uh U.S. Central Command uh conducted a a fact-finding investigation um and in that case the principal investigator general david edgington of the air force uh concluded that um there were some problems with how the system worked but that no human no individual human acted criminally negligently or recklessly um and therefore did not recommend uh individual discipline um and us and com accepted the recommendations um there was also a separate defense science board task force that found problems in how the system operated and recommended significant changes uh to how the system work worked and uh another analogy not from uh autonomous weapons systems includes the um the strike in 2015 the mistaken strike uh on on the medecincent frontier hospital in conduz by the U.S. Air Force um and after which there was an army 156 investigation which again concluded that that there were actually in this case were individuals who violated the law if not criminally um uh but the but the biggest issue was the cascade of failures within the complex organizational system and there individuals were disciplined but not criminally um and in addition uh there were significant organizational or systems reforms that were made and so i think this is a helpful analogy it may not be a perfect system um but where you do not have um individual intentional harm or even negligent or reckless harm uh there are options for other forms of penalties and accountability that being said many people criticized uh these inquiries for failing to be sufficiently transparent uh and sufficiently independent and i think one could look at how to make such inquiries more independent and transparent um as well so i think these analogies are are are helpful that's great thanks so much um we're getting close to the end of our uh session so i want to perhaps combine a couple of questions um ibn ide from the danish national defense college asks to both of you about your thoughts on professor rebecca krutoff's suggestion about introducing the concept of war torts to ensure victims of the harms caused by the use of ai-enabled systems are properly compensated um and you know professor dickinson you alluded to this in your remarks and you indicated that yes perhaps that could be a useful avenue but for the fact uh at least as i understood what you were saying correct me uh if if any of that is wrong but uh but for the fact that uh those venues to pursue those claims currently are lacking uh or or at least as a practical matter are quite limited uh in terms of accessibility and that sort of leads me to a question that i want to put to both of you which is you know both specifically on this question of of torts uh as a means of accountability but more generally the idea of whether there may be new institutional mechanisms you know beyond some of the innovations to administrative uh systems that professor dickinson has outlined are there international institutions that need to be established here to to uh or new uh mechanisms either domestically or multilaterally uh that we ought to be thinking about in this context to help sort of uh resolve some of the tensions that you've identified or is it simply a matter of uh better utilizing uh updating and uh you know ensuring access to institutions and mechanisms that already exist uh do you have thoughts on either of those questions uh well i'll just say really quickly i i mean i think tort law is is interesting because of course it's designed around um uh dealing with in many cases uh non-intentional or negligent harms and and even in some cases uh strict liability situations um it and i think professor crutoff's recommendation is interesting i i do as you said think that the one of the big challenges on the international level is the lack of venues um on the domestic level in the united states we have significant immunities that we've placed a variety of good reasons some would say those immunities should be reduced but we have immunities on on tort uh tort claims uh involving military action so i i think it's not uh super practical in the near or medium term um i think uh i think it's a little bit more practical to focus on existing mechanisms but at the same time i think it's worth thinking about uh the the horizon the the distant future um as others have pointed out uh creating new systems is time consuming and hard uh but that doesn't mean we shouldn't necessarily uh think about it but i think in the near and medium term it's more effective to focus on the existing institutions that we have great uh leigh you get the last word here before we conclude uh i would like to to clarify the my point and i i did mean uh there is uh actually in as a domestic level there are really many uh forms of uh responsibility we can uh invoke to to hold some somebody out to to be accountable but if we look at the international level uh actually uh we only have the uh two options the state responsibility and uh criminal responsibility but uh even if we talk about the the composition the world thought we have to uh establish this this responsibility before that and then we talk about the conversation that's my point great i think that's a terrific point to end on and i think it's wonderful that we're having these conversations i hope that this is uh just a starting point it's certainly uh you know not our not the finish line but uh it's wonderful to connect on these questions and hopefully this can uh stimulate some further uh constructive thinking and collaboration going forward so i thank both of you for approaching the conversation in that spirit and for your very insightful points raised throughout the hour uh lieutenant colonel sherry back to you thank you very much robert and thank you to the laura and leechung for your presentations and and leechung you had to get up in the three o'clock hour to do this which is very united states marine corps of you we're very proud of you and thank you but thank you to all three of you for your participation um and uh we will turn to the future uh in our next panel in 10 minutes so we'll see everybody back here at four o'clock or 16 uh i'm sorry for 10 or 16 10 thank you very much oh welcome back for our last session of the day thank you all for joining us and hanging in there so far um i will say that if we were located in newport uh this panel which is cosponsored by the labor institute at the united states military academy west point would be standing between you and a lovely evening in newport uh but since this is on zoom uh and also it's going to be like 26 degrees tonight here on so um we don't have to worry about that uh our moderator for futures command and technology on the battlefield is sasha raiden and sasha is a director of research and publications for the labor institute and i turn it over to you sasha thank you so much thanks lieutenant coronal cherry and thanks also to major tinkler and professor kreska and the whole stockton center team for organizing this it's great um and hi everybody who's watching i hope you're still staying with us in this last panel of the day and it's my pleasure to introduce the panel on futures command and technology on the battlefield with colonel stefanie ahern and professor chris janks and i'm especially excited about this panel because it marks the beginning of labor's partnership with army futures command and so i want to take just a few minutes before introducing the panel itself to talk about this partnership for those of you don't know the labor institute for law and land warfare is situated at west point in the law department it's an american centric think tank and we look at the law of arm conflict that's our thing in some related areas as well and we were founded a few years ago in 2016 with the intention of bringing together military expertise and academic expertise on the study of the law of arm conflict because we feel really strongly that in order for the law to keep its relevance as we move forward into future battles future conflicts we have to have the practitioners in the room and we also have to have those academics with deep expertise in time to think about these things in the room as well and so we see ourselves as a bridge between these communities and also others uh as we'll talk about in order to have a place for them to come and discuss their approaches and inform each other on their approaches and this is one of the reasons why we're so excited to work with army futures command because we feel this is exactly what we'll be doing with with futures command as as your as futures command is thinking through what warfare will look like what the army will need in 2035 and beyond and i'll leave that to colonel a hern to talk more about but as as they're doing that we at the lever institute will be helping think through the legal implications with our internal experts our senior fellows and board members and our wider network of experts and we are especially we think and as has been mentioned throughout the day today we especially think it's important that these issues are thought about in the early stages at the onset it's really important to integrate the law as well as other things into the planning discussions etc and so today we're thrilled to have two of the people who will be heavily involved in this effort here with us so we have colonel Stephanie a hern who is director of concepts at future and concept center at army futures command a bit of a mouthful hope i got it right and professor chris janks who's an associate professor at the deadman school of law smu and important for our purposes here he's also one of leavers senior fellows who will be heavily involved in this effort and they both have long bios a lot of expertise in their field i encourage you to look online at the events page website for their full bios but for now in the interest of time i'll turn it over to the panelists where this panel will as the title says focus on futures army futures command so it'll be a bit different from earlier panels today and colonel a hern will first give an overview of what army futures command does what it is a brief overview and then she'll turn to what we might some of the things we might expect in future warfare and after that professor janks will raise some legal issues or perhaps legal questions we should be thinking about and then we'll have time for audience questions and we really encourage you to submit your questions throughout you have two really great experts here and it's such an opportunity to ask things so without further ado i'll turn it over to you colonel a hern thank you sorry can you hear me okay perfect um so i just i first of all i want to thank you so much sasha and professor janks is such an honor to be able to to share this this discussion with you virtually um it really is a critical partnership for us at the futures and kaza center and army futures command to be a part of this partnership with the lever institute as well so just very quickly starting about army futures command and then the task that general murray the commanding general of army futures command has given those within my organization um so very quickly on army futures command it is the first four star level command that the army has stood up since 1973 it was established in 2018 in austin texas so that's where i'm currently at now um there are four main missions that it's really looking abroad that how do we consolidate all efforts within the modernization efforts for the army together looking at how we fight what we fight with and who we are and so the four main missions that the tasks that that a ofc has the first is describing the future operational environment so we have a threat section of organization uh mr mournston that looks at uh what are the defense and other trends from an intelligence perspective that we need to be designing the future to be able to address the second core focus is to develop and deliver future concepts so as we're thinking about the ideas of how the army could operate in the future from the operational level to to smaller um how could how should the army be operating in this future environment the third main effort that army futures command has is to develop and deliver future force designs so we don't just stay at that broad level of the ideas once we get the concept then we do experimentation modeling simulations wargame analysis to be able to refine it to a much more precise level so that when we pass it off to other parts of the army that they know specifically what what we need that future force to be able to do and then finally to support the delivery of modernization solutions so part of the reason why army futures commands exist is to make sure that when we have an idea about something that the warfighter needs how do we much more quickly get it from that idea to an actual solution in the warfighter sands and so making sure from that modernization from the acquisition side that we are really helping streamline and bring those parts together um so it's much more than just the tech material but that material aspect is absolutely critical in what we do and so I'm the director of concepts we call it doc so doc and what general Murray has asked my team to do working with many different parts of the army futures command is to develop the army's next operational concept for 2035 and describing if we were to have a battle with a near peer adversary and 2035 how could we operate or fight how could we be equipped and how could we be organized and so we still firmly believe that war is going to be a clash of wills and that if countries decide to fight uh that their its death and destruction is probably likely um how could we in that uh that high intensity conflict be able to prosecute things in a way to be able to win on the battlefield um but even though that the nature of warfare is is most likely going to stay constant uh it's still governed by rules and it's not that hobesian uh chaos and that's why from the law of run conflict whether it's you 100 years today or in the future those those rules about how we're prosecuting warfare absolutely really matter to us um so our g2 the intelligence part of the army futures command recently developed a future operational environment 2035 to 50 many of the things of of those of you that have any political science background um you know that unipolder moment is probably gone and so what this document describes is alternative future is of what could be so very similar of the global trends 2035 the joint operational environment 2040 um and it helps explain as we progress into the future how it changes in the international power um but then also from a technology perspective and I think really from this discussion today is how could potential revolutionary technologies change what that warfare the future battlefield could be and so from the responsibility that we had in dock is is really based off of that future operational environment that we have that we expect um what does that mean for those of us that are focused on the land power and how the army could fight what we could fight with and how we could organize and one again one of the the main efforts that we within the army futures command are are approaching us is we call the teaming night's approach but it's really from the very beginning how do we get those of us that do concepts working with the science and technology experts working with the threat experts and thankfully with this partnership also bringing in from the legal perspective the think tanks the academia how do we get as many of those ideas from the front so that we're not writing science fiction um and that the scientists they're able to see what could be are able to bring together some of these solutions from the beginning and so we just we really appreciate being able to be a part of this discussion as far as just as we take on this this concept right now sorry Sasha I can't hear you sorry uh thanks a lot and now maybe you uh you can talk about some of the things we might expect even though we don't know what will be but yep absolutely um and so so thank you there is much that we don't know about the future warfare there's maybe six things and this isn't you know they're not in any order but some of the the uh based off the experimentation that we've done and based off of the outreach with s and t with think tanks with academia that that we are expecting so the first being the ubiquitous sensors so we do not expect to have ubiquitous information or ubiquitous understanding but you know the the phone that i'm talking on right now is itself a sensor it can see it can tell you where you're at in the world some of them to be able to see how fast is the wind are there uh electromagnetic signals going through so a sensor being able to to understand to give you some kind of information we're not going to have perfect awareness but from the military perspective our ability to hide and our ability to surprise is is critical to many of the things we're going to do and so this is going to be for us and also those adversaries what does that mean for the for the future um the second is the increasing role of information and so information part of that is the digits the data that's moving across the battlefield but it's also the role of ideas and the disinformation the misinformation the social media the deep fakes um i think peter singer and his uh book on like war is showing not only what's able to be done from an information warfare perspective but when you actually are having warfare on the battlefield how that role of information is really complicating things we don't think it's going to get easier the third as far as we have these the sensors in the role of the information is that our weapons and equipment are going to need to have greater what we call range or the ability to distance um to be able to go faster to have greater awareness and to have better lethality um and in most cases precision um we're going to be on a battlefield where in the current counterinsurgency in the counterterrorism we are often operating within and among the people if we're going to be facing a great power adversary that may not necessarily be the case um and so how do we make sure that our systems are designed for for the threats that we'll be facing the fourth is that in addition to the technologies that we have in our developing today the role of new disruptive technology so uh artificial intelligence and machine learning autonomy and robotics we fundamentally believe that especially if those are able to work together they could change what's possible on the modern battlefield and so we need to make sure that we are understanding where the science is where the technology could be sooner and and being able to use those when and where it makes sense and then the fifth uh section is is that if it's dull dangerous or dirty um we should be using machines when possible and so much of that work by 2035 is really going to be in that dull category so what is your administrative your training your logistics if we could just do predictive maintenance using AI if from a training perspective you have a soldier that shows up at basic training and for the rest of their time they're able to have that training go with them there's a lot of really kind of fundamental uninteresting things that that AI would be able to help with um but when you look also from the dangerous perspective a first principle is that first contact with an adversary should probably be with a non-human and what does that mean um I think the other thing though is that as we're trying to figure out the AI autonomy and robotics it's not just what humans are able to do right now so if any of you has been watching uh in Star Wars or the Mandalorian where you have vehicles that are able to hover off a cliff and not crash how could that possibly change the way we're able to operate and we're you know we're trying to make sure again this isn't something that's fine as fiction but we're we're putting technology in 2035 and then the last thing um and then uh appreciate the questions that have been feedback um is that it's just increasing on what humans must do and so we in America we we fiercely believe that humans absolutely must be uh on the loop to be able to uh when we're prosecuting lethal force um in addition to having an accountability we are friendly believe that human and machine teaming um benefits and soldiers creativity they are able to understand the context they're able to adapt to the unexpected and frankly they're unpredictable and when you're trying to face an adversary that's something that actually is a benefit to us um however there are things as far as swarms and missile garages when you're in a protection side um and I think in previous discussions with Professor Tanks our previous systems when you're having incoming missiles some of those things may need to be able to have you know automatic use of autonomy um but just like we have phones that are protecting us against spam and car pistons that are firing we need to be deliberate we need to think about this upfront what makes sense to make sure that humans are helping inform these decisions um and so I just wanted to finish with PI 2035 is not going to bring terminators um there's also real risk for any country that's wanting to defer these decisions to machines and really the the role of of low active role of labor institutes to make sure that we're thinking through what are these legal implications for those large scale conflicts that could happen in the future and something that we haven't seen for probably 30 years you know what are the laws that we need to develop versus race symbols versus we actually just need to clarify law versus policy um we are still appreciative of this partnership to be able to make sure that we're thinking from the very beginning about what those those implications are thank you very much um Professor Jinx over to you thanks Sasha uh and thank you uh Colonel LaHern and also thanks to major Tinkler Lieutenant Colonel Cherry and the entire Stockton Center for organizing this event and including a panel on the Army's future command though to be fair it's perhaps fitting as in the very near future Army will commandingly dominate Navy uh in football so I appreciate that Lieutenant Colonel Cherry um I want to briefly highlight some law of arm conflict either issues or questions based on Colonel LaHern's remarks but before doing so I want to clarify the framework that I believe Army Futures Command is operating under and by which those legal issues arise as Paul Shar mentioned in the opening keynote the terms artificial intelligence or AI and autonomy are subject to varied understandings I certainly am not venturing into those definitional black holes but just want to suggest that perhaps a more helpful way to think of them is as technological descriptors and as we just heard from Colonel LaHern Army's technology transformation is an evolving process and that serves as a reminder that we shouldn't think of technology in either static or binary terms that a system or machine is either AI or autonomous or it's not rather I think the Defense Science Board's explanation of technology is reflecting a quote capability of the larger system enabled by the integration of human and machine ability is more useful and that entails assessing what humans and machines are capable of doing along with preferences as to which entity performs which tasks and I think that's reflected in what Colonel LaHern just mentioned about dull dirty and dangerous tasks being performed by machines when possible while preserving a role for human judgment in the application of force a lot of attention is paid to technology and the use of force but much even a majority of the applications as we just learned really are going to be in administrative training and logistics functions but those non use of force applications are still subject to the law of armed conflict in several ways but I just want to focus on one the constant care obligation from article 57 of additional protocol one to the 1949 Geneva conventions the constant care obligation refers to the requirement that quote in the conduct of military operations constant care shall be taken to spare the civilian population civilians and civilian objects particularly in the area excuse me particularly in the area of driverless vehicles whether ground or air it will be interesting to see how understandings of the constant care obligation evolve earlier today we heard speakers discussing how the law of armed conflict does not prohibit the use of autonomy within the next 10 to 20 years which is the time horizon for most vehicles in the U.S. anyway to be driverless will driverless technology become so ubiquitous that the constant care obligation will require first world countries to use driverless vehicles in and around the battlefield now in terms of the use of force I think it's important to remember the law of war poses obligation on obligations on persons and not weapons machines years lack agency don't have legal personality and cannot assume legal obligations I also want to stress the danger of conflating weapons which may be per se illegal versus the unlawful use of weapons we've heard about some technological challenges involving machines perceiving context maybe the technology will overcome those challenges and maybe it will not but even where there are challenges or limitations say on the ability to distinguish between military objectives and civilian objects that doesn't mean the weapon is per se illegal rather where and how that system is used would need to be considered and I would just commend Mike Schmidt and Jeff Therner's uh Harvard national security law journal on out of the loop for anyone who's interested in more on that discussion as a result what I think that what we should expect here in the U.S. is the development of AI enabled systems inversely proportional to the possibility of an untoward event involving civilians that means the navy developing subsurface systems and the air force high altitude systems before the army develops anything other than defensive systems colonel a her and in our discussions leading up to today's event you were quite clear that neither your office future concepts nor army futures command are the proverbial good idea fairy you know utterly divorced from reality and you stress the feasibility that feasibility is an important uh criteria and to that end you have resisted my attempts at promoting turning the 60 ton M1 Abrams tank into a hover tank but I want to I want to end by just noting that an earlier panel suggested that the role of at least some lawyers may need to shift to be more front loaded either in the acquisition fielding or use of weapons and systems so given the need for future concepts to be feasible are you will you in futures command be considering where and when do the lawyers civilian and military play a role and I'll turn it back over to Sasha or colonel a her yeah go ahead colonel a her I think uh the reason why general Murray was so supportive of having army features come in and be partnered with a liberate institute was when he went and visited west point this opportunity came up and he's done it and so I think this is as we've from the the director of concepts started to work on this kind of the what the future concept should be some of our initial deep dives have been with lawyers and ethicists but I think this formalized partnership that we have with we were instituted is completely addictive of why the previous approach that we had whether it was you know the rigidly sequential of let's come up with an idea and then eventually we'll walk through the widget we can't afford to do that so having this partnership with the legal community a lot of on conflict those that are it's the core of of who we are as Americans and what's expected of us thank you and that that opens up our time for questions and it relates to a question posed by professor Jensen and professor Jensen if you I don't know if that answered your full question but he had asked how do you incorporate legal advice in your work and do you think there's sufficient legal input into your command to ensure that weapons development complies with a law of armed conflict so I I think that relates to what professor Jenks just asked you I don't know Colonel Aaron if you want to add anything to it absolutely Colonel Jose Corat is the staff judge advocate for General Murray and is one of General Murray's key adjusted agents but I think we are trying to make sure that it's not those of us that are within our mutual commitment how do we make sure that after thinking across these systems and the thinking across in time we really are doubling a couple of months and in so simple that we're getting this that's the front end of our efforts so I would say that we are welcoming the partnerships and and are eager to make sure that this is not just something that we come back a year from now and remember this this good discussion that we had and wonder what happened in the meantime thank you we've focused a lot today on other panels and here on autonomy robotics and AI but you're at that's not all you're doing as you said earlier and so we have a question here that says to what extent do you think biotechnology could play a role in future warfare over I think that it's in the both in the very productive how are you making sure that our soldiers are more resilient that if something happens to them that they can recover much more quickly the women need the health but I think what what we're seeing today and that we've gone through for the past nine months we're also showing the tremendous implications of what biological threats can have so it's it's not just the great powers that are paying attention to this and so there is a a lot of work that's that's being done across the army teachers coming in but also working very closely with with academic and and some of our lab partners can make sure that that we better understand what are those threats but also what are the opportunities for for the resilience and the maximizing the human performance again in an American way but you know if we can make our soldiers able to do better to be healthier to be more resilient that that is a goodness that we're trying to explore as well thank you and on that idea of what others um what others might be doing one of the questions that um professor sassoli is asking is do you also think about how the enemy both state and non-state actors will develop and that just I think he means that generally not just in biotechnology no thank you and I I think this was one of the things that when we were scrutinizing what humans must do um that I I forget to mention and he said even though that we are bound by law of our conflict and the principles that were codified in some of our funding documents we are fully aware that some of our adversaries are not and that doesn't mean that we're going to follow their example for instance just because others still want to live for property doesn't mean that we should however in order to make sure that we are much more resilient for how others might be operating in this space we want is we appreciate any of the support and understanding coming from the legal community about what that could mean but this is something that from a concept perspective we're often trying to make sure that we can make ourselves more resilient against those that may not apply the same standards that that we're willing to prosecute thanks and we have another question that says the problem with a legal analysis of any problem is the lawyer is effectively constrained by the facts presented and the intended use how do you work with your legal team to get a legal opinion on these future weapons and are you actually embedding them like jags in the war games and then asking for an opinion similar to the previous questions I think this is part of why kernel core is is part of general money's immediate staff and that's replicated throughout the the different parts of the organization is that you can't wait you have the solution and then say what do you think it is then that that's not making them part of the solution and making sure that you know the letter in the spirit is brought in from the beginning but again I think that there are some opportunities that we have in part because we've been focused on being counterterrorism for 19 years so we've grown entire generations of officers that are used to focusing on a different problem side and so making sure that we understand where where is the law and as we're thinking through some of these new problems that technology opposing that we're also bringing in experts that can help work through these problems from the legal side as well I mean the thank you the going back to how the U.S. has fought over the last really now 20 years there have been a number of me and fighting largely non-state actors and encounter insurgency there have been a number of policy choices that have been made regarding civilian casualties I also think that we because of the nature of the warfare we've engaged in over the last 20 years I think there's been some forgetfulness about the idea of sensor-based targeting as you know as a very real thing that a lot of former military will identify with you know growing up on the idea that you would target you would target based on a radar radar emission so I think I think there is some reeducation both of the military and maybe even also of the public whether it's just because of the nature of the warfare of the last 20 years or if some of the policy choices the policy choices that were made that while perhaps appropriate for a counterinsurgency may not be as appropriate or as applicable in a high intensity conflict with a near peer adversary we have a question from a panelist from a panel earlier this afternoon and forgive me I don't know how to pronounce your name even either she has she's directing this to both of you what are your thoughts on how to handle distributed decision making and future conflicts in order to ensure that the appropriate type and level of human machines are made are you considering new command control concepts and the need for more clear description of roles and responsibilities I don't know who wants to take that first I'm happy to jump in um so yeah the the answer is absolutely um part of the challenge with having the ability to have centers in so many places and you look at specifically how the Russians have been prosecuted in their warfare for a while is that when you have large formations together they are able to actually quickly annihilate them and they don't care what's in the way and so one of the things that we're we're looking for the future of how we could fight we're not reading with the command and control the how the organization but how could we operate the equipment that we would need and then the organization to be able to to not only be able to maneuver but to be able to have that command and control so we aren't starting with the questions on how we're going to command and control across the battlefield but that will be quickly part of this this future concept that we're looking at no I think that's a that's an interesting question and we've been so focused on technology and the select and engage aspects of the use of force but at some point and I think this goes back to Colonel Aaron's remarks at the outset you know wanting to understand where technology is making decisions based on sensor inputs that get into or that cross the line into what we think of as a command and control so where where is it that we're comfortable with command and control or command and control like decisions or assessments being made by machines or algorithms versus where do we need the role of the you know of the human yeah that makes me think of the program you had a few weeks ago Colonel Ahern somebody said what if a what if a commander has command but not control what that might mean um we have another question that is directed to you Colonel Ahern has army futures command beyond thinking about the role of the tactical commander within the future within the future this future operational concept and what are the skills and traits that you'd want from future company commanders for example will he or she be expected to have more technical expertise these technologies as these technologies develop um yeah this uh Rob he says he's sure you're thinking about what will be asked of future army captains and nco leaders as well as the responsibilities we'll have so uh there's two part answer on that too one that we're starting at the operational level and so how would we prosecute that war um and so the much more of the the thorough analysis will be done later um but that said the army futures command uh in direct support of what General McConnell's priority of people and the Secretary's priority on people is that we we know now that we need to have leaders who are able to be agile and responsive and ethical but also have a better appreciation of the technology that they're working and so there are initiatives that army futures command has already launched there's some of them a partnership with Carnegie Mellon University to make sure that we are developing leaders that have technical the data experts that we will need in addition there's another initiative that will start right at the beginning of the year called the software factory and so helping more people across the army just become more familiar more comfortable with data data management data engineering so that we aren't having to rely on people outside of the army to be able to move forward but yes and much more to come thanks uh we have a comment here by Pete Pedrozo there have been a number of tests that have pitted man versus machine and in every case the machine wins if the US continues to assist insist that there be a person in or on the loop but an adversary does not won't that place us at a significant disadvantage on the battlefield like bringing a knife to a gunfight over i think that's well i don't know if either of you have a comment on that i would say that to go is probably one of the examples that comes up and the machine didn't win every time and in fact when you pair machines with human machine teaming usually the human team the human machine team will win and so i think part of this is understanding you know when you had the um the go project is part of it was how did they define success right and if they won by one there's still one um and so i think the one of just uh of who we are as americans but we actually think that you know being very deliberate upfront about what are those things that humans are just naturally better at um and i don't think the evidence has yet said that uh that having machines always beat humans so i we're definitely not pursuing this step past because it's better Sasha you know on that and i guess i should clarify that i'm speaking in my individual capacity and my comments don't reflect either army's future's command uh the lever institute or or the army uh i mean as to Pete's question uh i think one of the reasons why the US has been steadfast in resisting this push to adopt the term control in terms of the role of the human on the application of uh use of force so the ai principles use the the term governable um and we talk in the united nations context about exercising you know exercising judgment in the application of force i think that is an implicit recognition that machines as you are performing in many ways in ways that we we we're kidding ourselves if we think of or use the term control just look at what the patriot system is doing uh at any given time so we need to have humans with a role such that they can turn off basically reboot a system if there's untoward untoward events but i think the US has been us and a number of countries have been wise to resist the push towards uh accepting the terminology for a role uh and rather using the term judgment and talk to one more thing on that dude that the colonel john void had come up with the udaloop so observe orient decide act that often within the army we'll talk about see understand decide act and so when you're trying to think through how are decisions made and actions done um being able to see and understand much more quickly is helpful to be able to take a massive amount of information but that decision has to also be based on context and you can train algorithms but you can't train them for the unexpected and so having those ability to be able to sort through massive amounts of information coming from you know hundreds if not thousands if not millions of sensors be able to make sense of it to make decisions especially a lethal force uh it's something that that we just we feel very committed to um and that there are tremendous implications if you're just allowing a machine to try to sort through what information that it's it's not taking into context colonel a her and if i if i could um i think you know you and i talked about this before you know i started out as an infantry officer in the port bending they talked about your span of control that the you know the infantry squad leader can appropriately adequately supervise you know x number five six seven uh soldiers and you can say that they've got more than that but realistically they're not effectively supervising or managing those and i just wonder with all these sensors and all of this great input that we're currently going to have given to war fighters down at a very low level is there any concern or is that anything that futures command is looking at is you know sensor overload in terms of how much is an infantry squad leader able to process at any given time in terms of multitasking all this input absolutely but it that's not a 20 20 challenge either you know when we went from the type set to the typewriter to the computer this is something that we've had to how do you separate uh the the noise from from the actual real information so i think part of this is why the training aspect of this you don't just show up one day and they issue you the ai and you go out and fight a war and part of this is developing that trust over time of what is this machine what is this autonomous robot able to do how are we able to train it and to train ourselves to be able to understand how these partners together um and so it is absolutely a concern of having so much information that it's just it turns into noise but that said you know that being able to have a system to work through this to be able to if you have a an operational approach and you can test it millions of times as opposed to in five war games you'll have a better understanding of what's in the realm of possible but but it's it is absolutely a concern because at the end of the day is that human we are still limited on what we're able to to understand and to the context we're putting it in thanks and we have a question from colonel reeves actually a first question to both of you so first could you please discuss how you see the proliferation of groups like the russians wagon like the russian wagner group how they may impact future combat operations and also could professor janks discuss the existing law and whether it is adequate to address this growing threat i don't know who wants to take that first thanks you want to go first i mean just in terms of uh jane i'm not sure if you're coming it just non either the the mercenary aspect or the private security private security contractor aspect of the the wagoner group i mean that's obviously not a not a new phenomenon um uh so i mean i think and as we've seen but you know non-state actors terrorist groups are all increasingly using uh increasingly using technology that we may have thought was reserve for the state so i don't see a particularly unique challenge uh vis-a-vis the fact that countries are private actors or state-sponsored large somewhat state-sponsored uh private actors seemingly private actors are operating in the battle space of another country i mean i think that gets into almost even a you said bellum uh and an international legal wrongs kind of doctrine more than a specific low-act doctrine unless you uh want to get into the discussion about mercenaries never would add as far as um the speech operational environment the document that we had that focused on the 2035 to 2050 um statistically speaking you uh it is rare to have large scale conflicts and in fact one of the efforts that we want to do is we want to deter great powers uh from starting a large-scale conflict because of the immense suffering that goes along with it um but that doesn't mean that conflict is going to end and so having some of these talks to see what others what to see what they're able to get away with uh is probably not going away and so what's the role of information and what's the role of being able to get to places quickly what's the role of being able to strengthen our allies and partners so that when they have these challenges one we can very quickly understand what's happening that how can we reinforce them to make them more capable to also uh be able to handle some of these challenges that are most likely going to get worse if not better thanks we have a another question uh from mr. Eric Rischel from army futures command it's to both of you he says with the preliminary news regarding china's development of quantum computing do either of you have thoughts on how that quantum leap in computing power would affect cyber warfare information and similar systems you know we have some others frankly that are participating or in the audience that I know are much more qualified to comment on that than I am the the main thing that I would say is that we are very thankful that we have amazing scientists and amazing technology experts within army futures command families that are watching us very closely and helping inform us about what could be in the future and then on the threat side that are paying attention to what china is doing as well and related to professor jenny about sensory overload does the future technological landscape potentially change the nature of command will it be possible for command and all the responsibilities and the authorities we think about going with it to reside in just one person or will command necessarily have to be dispersed and professor jenks what implications does this have for legal principles such as command responsibility and obedience to orders so that last part was not what I was expecting but I would say that I there are many layers of command and so it's not one commander and then everybody else is waiting for one person to make the order so depending on the type of operation depending on the severity of of that operation would depend on what level that we would have so we have many different echelons in part because we're really spread out across a wide area um the one thing that I say and I turn to professor jenks is that we at the individual level but you are absolutely responsible for the action that you commit um so if it's legal escort moral then if it's a command you should do it if it's not you are personally responsible and I think you know we have safely uh the law has shown that that's that's an individual obligation in addition to uh to what uh we are doing at the unit level but professor jenks let me turn it over to you um I mean part of the challenge is we're we're talking about notional not just you know what if uh kind of systems that may or frankly may not be developed at some point um in the future I would just I would just stress that all countries have a built-in incentive to have reliable predictive systems systems that are designed and perform as uh as designed so it's it's always surprising to me at some of these international uh discussions that we've engaged in in in Geneva where the premise of the the premise of the discussion is the idea that a military would have developed a system that doesn't perform reliably and does things uh unpredictably no military in in the world wants that system and would pay money and develop and field uh such a system so I mean I think part of the challenge in years we're just going to need to wait and see the kinds of systems and the ideas that kernel a herning futures command the army and the rest of us military come up with recognizing that I think one of the strengths of the law of arm conflict and one of the reasons why it's still as relevant and applicable today in 2020 is that it's predicated the answer to so many law of arm conflict questions is contextual reasonableness so I think in the end that will be the answer to a lot of questions in terms of uh compliance or uh functionality under the law of arm conflict with some of these future systems we're almost out of time and there's still a number of really interesting questions I think given the time I'll just end it with one that might be a quick one for quick answer maybe for both of you uh Charlie Dunlap says is there any prospect that AI and machine learning could somehow supplement or even replace the human legal advisor do we have an AI jag I don't I I don't see that I think the challenge I mean we had Mike Meyer uh was moderating a panel earlier I think a more near term issue is how do you operationally test an AI enabled weapon system a system and it's one thing you've got a new rifle round a new artillery round you can fire that however many thousand times and have an under have an idea how it's going to perform that's really just a function of ballistics and physics um but with an AI system that's designed to you know expect the unexpected in react to frankly limitless uh scenarios and environments how you could how you could test such a system and whether or not simulations uh what role simulations might play in the uh in the testing so I think that to me the the real issue is is there a way where we could do legal reviews of AI enabled weapons but that's a question I think that's some still some ways off I don't think any commander would be supportive of having a machine providing him a hard legal advice um so helping inform their legal advice maybe but but not as a substitute no thanks and I'm sorry for the other questions that we don't have time to answer I want to thank our panelists very much for for all your input and your information and now I'll turn it over to Kiran for his closing remarks thanks thanks Sasha and thanks also to Colonel Ahern and Professor Jenks for giving us a glimpse into the future there and how the US Army is um looking to deal with some of the challenge that's that's likely to pose uh I'd like to thank again all the panelists and speakers today for what's been an excellent day and to all the attendees for joining us we will reconvene at 1100 hours tomorrow morning uh where we're fortunate to have a keynote presentation by um the Deputy Attorney General for International Law at the Israeli Ministry of Justice and then to follow three panels on cyberspace and international law so thanks again and we hope you can join us tomorrow