and we're live. Hi everyone. Welcome to the sixth annual MIT IAP Computational Law Workshop course. My name is Tammy Rogier and I'll be one of your three TAs for this course and I speak for myself and all the other TAs and instructors when I say that we are so excited for you to join us this year for this special workshop. We are in unprecedented times so this year we'll be splitting the workshop over the course of four one-hour sessions. Over the course of these four weeks we're providing you with a conceptual overview of computational law and this lecture series will include everything from seminar style lectures to classroom discussions to supplemental readings to key challenges posed by our invited speakers. In true MIT fashion this course will also include an optional experiential learning opportunity which will be through a class project that you'll hear more about later. Okay so make sure that it's on the right slide. Okay to go over some housekeeping rules first I just want to reiterate that everything this year from class sessions to our activities will be virtual and available through links that will be provided to you shortly before each class and we ask that you please reserve an hour before class each week to review the class materials in advance of the discussions that we'll have for the week and to keep things simple we'll be meeting at the same time every week from 12 p.m. to 1 p.m. eastern standard time every Friday for the next four weeks including this one and I know that I was a lot of information so if you want to refresh any of it as you saw it's on the slides four and five of the slide deck for this course and now I'd like to introduce you to your main instructors and TAs who will be here with you throughout the course of this throughout the duration of this course starting with Dazza Greenwood your instructor. Thank you so much TMA. Hi I am Dazza I'm a lecturer and a scientist at MIT in the media lab and also in connection science which is in our school of engineering and I basically am the I run the computational law research initiatives primarily out of the media lab and this class is very much part of those initiatives this is our more open engagement interface so we're very focused on expressing the law and legal instruments and legal processes as as code and as services and we've got a terrific cast of characters for you this this semester some amazing speakers opportunities to do projects that you'll hear more about from Ryan later this this in the hour and and also we'll be doing some open office hours I know from emails many of you would like more opportunities to talk and to have discussion so but one toward the end of the class today we'll have an opportunity for you to sign up for that as well in the meantime if it wasn't clear from what TMA said I should just also mention we're going to be using the chat to pose questions to the speaker as they're going and also other general questions and we've got as you're about to learn a terrific cast of instructors and teachers assistants who will be monitoring the chat so please do use that to to pose your questions or or comments so with that I'll hand it forward to our co-instructor Brian Wilson hey everybody my name is Brian I am a fellow in the MIT connection science research group where I serve as the editor-in-chief for the MIT computational law report and one of the kind of unique things about the computational law report is we don't want it to just be a standard sort of legal publication we want it to be a place where we can also you know in addition to producing content we can also hold conversations with people in the space who are doing interesting things and we can also convene and we also can convene and learn and do these interesting sorts of kind of experimental learning challenges like we're doing with this IAP course so I'm really happy to be here with you all and extending that vision a little bit and I will hand it over to Andrew. Thank you Brian hello everyone my name is Andrew Dunzowski I'm the research editor of the MIT computational law report and a practicing attorney by trade very excited to serve as a TA of this course we have an excellent list of guest speakers and so we're looking forward to their contributions and the participation of all the students here so with that welcome and I will hand it off to Megan. Hi hi everyone good afternoon my name is Megan I am also one of the TAs I am a PhD candidate and a lecturer at SengSpo as well and I'm so happy to see all of you here as I mentioned in this welcome video that I was formerly a participant of this course and it really opened my eyes and I really hope that with the material with the exciting lineup that you all share that same experience that I had so with that I'll pass that over back to TMA. Thanks Megan hi guys I'm TMA and I'm also an advisor at the MIT computational law report my backgrounds in finance and blockchain and for the last several years I've also been a part of the MIT computational law community this is my third time TAing for the course um yeah I'm just really excited to be here again and to be learning with you guys so with that I'm going to get started with like the core part of our lecture today today we're going to be hearing from professor David Restrepo who is a part of the faculty at HGC Paris where he's a director of research on legal metrics and smart law which stands for scientific mathematical algorithmic risk and technology driven law today he'll be discussing opportunities for using computational law to audit data supply chains and if you like Dasa said if you have any questions come up during his lecture please feel free to submit them via zoom and we're going to get to them during the Q&A section of professor Restrepo's talk okay professor the floor is yours yeah Dasa oh sorry I was just going to say you might want to just show the slide with the David's background yes oh wait why is it not showing there we go oh thank you thank you TMA thank you Daniel thank you Dasa um so let me um so can I get uh can the screen share now yeah I'll stop sharing professor thank you very much so um hello everyone very happy to see some familiar faces I hope you're you're seeing my screen now um so very glad to be very glad to be here today and to share some thoughts after the great talk that Dasa shared with us at HEC a few days ago uh so I thought that the first thing I would do uh was to connect with the computational law concept and as TMA said we are basically working with the smart law concept which is actually not that far from computational or actually very similar um and it's what we use smart law as an acronym as you realize for scientific mathematical risk and technology driven law um so it is sort of slightly different from the idea of code is law there is some sort of alienation of law to code uh and actually um it is closer to computational law in some sense we see computational law so for some somehow was a starting point and probably the core of the smart law understanding um but it adds some others other concepts that I'm going to try to show you right now and because the sort of main thesis is that there is some sort of hybridization of law in the process right so it is not that we have sort of a computational application for law but that actually legal rules are becoming hybrid to some extent and that at some point um the nature of rules uh become partly sort of what we know as traditionally legal and partly computational and the fact that they become hybrid entails some consequences for the way we understand law which basically are related to the risk aspects uh and the mathematics and the scientific part are sort of support elements so let me give you an example there is an article that we have I have written with Gregory Lekovich on this uh and that you can actually have a look at I think Megan helped on that article so uh it was it was really great to have Megan also with us um so let me give you a very simple example how where you can see this a good example is the cycle of financial law today and financial law has an interesting element is that in finance is one of the areas where there is more innovation in business um and uh what happened in in finance is that in the fintech different uh actors started using a computer program this is usually what we know as fintech computer programs to support or enable banking applications um as a consequence of the use of these uh technologies um some obligations were put into uh corporations or banking or banks to implement regulation and this is where we shifted towards uh red tech so different sort of startups and companies are starting developing tools i'm going to show this in a more concrete example sorry developing tools uh to ensure that in the use of technology by the banking and financial services uh legal rules were being respected so we sort of shifted towards the computational aspect right of of um of the law um the thing is that in order for financial supervisors to be able now to monitor the activities of uh banks and financial services implementing technology and implementing technology to implement legal rules they also needed to carry out this process through technology so what you have is a loop what we call fintech red tech subtech where the entire cycle of financial regulation goes through technology there is just a very recent paper by the european securities and market authority december 2020 is a very recent on algorithmic trading and what is interesting here you see very well the relationship is that in different regulations in the european union which are basically the market abuse regulation and the miffin 2 to the directive um there is an obligation uh overall there is an obligation for uh actors operating in a trading venue and to um investors or um any person professionally carrying an activity in a trading venue to ensure that their algorithms the algorithms that are deployed in the trading venue uh are actually will not cause any uh disorder trading or will not uh behave badly under stress conditions right um so this is an obligation as you see in the different uh articles that i'm citing that are basically in many cases on the bottom of the um person engaging in the um in the in the in the trading venue and um when you see article 17 so on the right hand on the bottom you see that an investment firm that engages in algorithmic trading uh shall have in place effective systems in control and these include on top uh ensuring that the algorithms have been tested okay so how this is actually performed what you have here very simple is a fintech red tech subtech uh context on top is a very very famous case of um of um spoofing uh and in hyper frequency trading where algorithms were used to buy and sell in very short time um uh shares i think these were some sort of derivatives so basically what you had as a consequence of this regulation is a startup uh on many others actually developed this one specifically was called algo guard who was actually certifying there's a private company certifying the algorithms that companies were using to deploy it in the stock market and in turn the regulator actually this is nasdaq that you might be familiar with nasdaq services for regulators have developed services to verify the algorithms that are certifying the algorithms that are actually being deployed in the trading venue right so what you have is a whole cycle of regulation totally performed through algorithms so this is and what of course how this is organized is very much on sort of a risk approach uh and that's why this was an example and sort of mathematical models that go behind so this was just to show sort of the additional legs that we have put to the computational but are of course totally included in the computational aspect um and this breaks me to a small anecdote before going into the core um i don't know if you're familiar with this what i'm showing right now uh this is a story of the accountant um and why i'm doing that is because of course we are at an MIT course and at the very bottom of the um of my list i have lawyers and computer scientists um and this story for those who are not familiar this was a huge controversy uh between um the um abasis that were used in the abacus that you have on the right and the algorithms that were used in numbers so um this happened uh the you know the start of using algebra and and numbers uh arab numbers started in the eighth century in europe coming from of course um uh from um northern africa and and and then um in the 13th century accountants were actually a protected profession i don't know if this sounds familiar to lawyers but they were a protected profession and this protected profession had the monopoly of accountancy and the tool they were using for accountancy was an abacus uh and these other guys arrived saying well you know what we can do the same thing with numbers and algebra uh and and some equations and what happened was that um in many states including for instance in floren's 14th century the use of um um numbers were was forbidden for accountancy accountancy was a protected profession and the uh accountants had to transport all their material in the kingdom to be able to do the accounting where there were a market or to keep the accountants of the merchants or the king so they had an equipment in horses transporting these throughout the kingdom and then these other guys could do that just with some paper um and and of course there were accused of using some sort of magic and and being able to cheat by using the numbers and all that so what i wanted to show in this is one example with the accountants that they want to use these with lawyers but you have a history of lawyers interacting with other disciplines with theologicians of course with economists engineers managers and today with computer scientists now the story says that um as you might know or i hope you do i hope you don't go to an accountant that uses an abacus today most of you might use an accountant that uses numbers um so the story says basically that most of the accountants at the time became algorithms uh and uh well i'm not saying that all of you have to become or lawyers have to become computer scientists but just the path that we were in other disciplines going through similar interactions before so these haven't been said um let me um let me say some of the implications of this and maybe i'm just going to focus on one is that um in some of these systems like it happened like i just showed in the financial cycle you have some interesting changes that are happening um and one of them i just put some of there for you is the fact that we are moving towards a system where um we are kind of heading towards hundred percent compliance i don't know if this is something we want but this is what is happening for instance in the fintech cycle where you have the entire system being actually recorded and the insurance compliance hundred percent and uh and this is interesting because back in in in in vays um Roscoe Pound said that the life of the law lights on its enforcement and it seems that sort of these achieve towards some sort of computational dimension at least have to get us thinking about whether the system that is hundred percent compliant is something we want but it seems that it's something that could be possible to some extent um now the last preliminary remark is that this is shifting us this is just uh i'm like i'm glad i got this first session because i'm able to i can't just sort of bring these sort of crazy ideas together for all of you is that we sort of see a change in the fact that we are having a course on computational law or law with the MIT and me from HSC Paris both of us not having a law school um shows something here is that we it looks like these elements reflect a shift also of the legal science from sort of field science and empirical science would Aristotle would call it to a lab science and a lab science in which we sort of check and program and and and and and change parameters of algorithms to do regulation rather than you know regulating the physical work so sort of a shift towards a lab science which is very compatible with exactly these course we are seeing today what you have on the top of the right now i'm moving to the content top right it's one of the work we have been doing it at HSC Paris a dozen is familiar with it which is identifying networks of lawyers how do you choose a lawyer for your case and here what where i'm showing is some of the work we've been doing identifying networks of lawyers you have these of course with several legal analytics um and what you have is here the dots red or green red or yellow represents the net win or loss rate of this lawyer so if there is a red dot means the lawyer has won more lost more cases than one if you have a yellow one has won more cases than lost and then the the width of the line represents the amount of common cases now this is some stuff that has become a common place in in in in many researchers and one of the things i wanted to highlight to you because many of you come maybe from a common law background is that there is different techniques to identify different issues and one of the things we've focused recently is on how difficult a court case is okay and and for that we implemented different methods and we came up with a very sort of basic one but this is just the execution which was um um clustering cases by the common articles cited like legal articles cited in the decision just very basic and we started doing that where basically with the number of cases um and and and where we have five articles in common four articles in common or five articles in common sorry you see it below five k ages uh actually these cases represented groups of cases relating to the same issue so you see here on top of b on green where holiday is not given by the train company and b in red related to rb and b renting and we thought can these tell us something about how difficult a case is and we started looking at for instance if you see the green community app with the green dots those are green dots means that claimant won so it seems that in these sort of cases clay it was an easy case for claimant i mean of course we haven't read a single case this was all done automatically so we haven't read a single one but it seems like these were all easy cases for claimant and these other ones were easy cases for um difficult cases for claimant i'm going to skip these we're also working on the automation of the cycle of the supreme court in france uh sort of attributing these directly to a chamber and working in collaboration with the court which is the supreme court on arcade but i'm going to jump this go to the to the framing of our problem so here is the question data supply chain this is the problem statement so um in most of the literature and discussions even in practice about data supply chain um there is a lot of focus on the b to c relationship business to consumer relationship right how the business or the data process or collecting the data is actually using the data and whether it is in compliance with privacy regulations now in our research the research we've been conducting we realize that actually there is something that we decided to call the data supply chain and then this term has started to gain some traction which is how the data flows in the back end of the processor so if you think about i don't know um walmart collecting data from their consumers actually it is very likely that walmart is not doing much of the work walmart is probably transferring this data to i don't know microsoft or some other teams doing business intelligence and these teams in themselves are transferring this data to other companies so the question is how can we ensure that the data protection regulation is being sort of respected throughout the chain that was so that's the problem statement right knowing that uh for instance for walmart it would be very difficult to get to know what a company is doing in europe or in india or in brazil uh with the data they have transferred to them and our starting point was article 28 of the european general data protection regulation which establishes that basically the data processor the first data processor walmart is responsible for ensuring data protection in let's call it the supply chain so we started working from there right that was our problem i told you that there is a whole chain uh so when you collect data you might have a subcontractor that is storing the data other subcontractor that is doing the business analytics although that are doing visualization ai and in in turn one of your subcontractors may have subcontractors so you have chains of not only of contracts but you have chains of through which the data flows okay so you have at the same time documents and data flowing um and we are sort of um started looking at the types of documents and tools that are used in the data supply chain which are in most cases common to chains of contracts which we are familiar with and we have contract reviewing technologies for that increasingly and some of the things we realized is first of all that there is a heterogeneity of contractual artifacts there is a lot of different documents that you have on your left these are legal documents that are used we have different formats and we have different ways of follow-up execution sort of giving orders or cascading instructions right so this is the whole sort of type uh devices that are used throughout the entire cycle um the other element that was important for us to look at was that supply chains are increasingly transnational they are not like only in the US or in Europe right so you might have a someone doing ai that is in Brazil or in Argentina or in Morocco while you and this company itself might transfer the data to a different place so uh ensuring the um compliance in the data supply chain is somehow enforcing some sort of global standard through a chain right um and this is increasingly important we know the supply chain issues in other industries uh we had that with goods like government or cars or and the whole supply chain control has been a lot of uh there has been a lot of work in supply chain in general now the question turns now to how to do it in in data supply chain where there is increasing increasing um amount of data flowing and some of the challenges more at the contract management level that we found for instance is centralization of data right uh just to give you an example imagine that you have um the original contract between the consumer and and and walmart let's say let's say says that um the data has to be stored for will be deleted after two years okay then walmart transfers this data to microsoft that microsoft to another company you have five or six or seven companies that have been using this data right how do you actually ensure that this data has it will be deleted after two years in all this company right and and this is some of the things that the problem is that sometimes the data is not centralized the storage is not centralized so you need to make sure it is deleted in every place where it has been stored um so we came down to three main pillars of compliance in chains of contracts or in the data supply chain first there is the coherence through contracts meaning if the first contract says that uh the data has to be stored for two years you need to make sure that the other contract down the supply chain also says that the data has to be deleted after two years so this is ensuring some sort of coherence in the contracts through the supply chain now the second element of compliance in the supply chain is to ensure that the documents or the contractual documents legal documents are in compliance with the regulation with regulation where the main processor is and probably also necessary that they are in compliance with the national regulation where the data is being processed and the third element of compliance is actually the effective verification of the obligation meaning that the data is effectively deleted after two years right so this is the auditing and verification that with the actual data so we have review i'm not going to go into this because of course it will be too long but we have reviewed the different technologies that exist in a paper that is available in on the internet and basically we've come with these i'm just going to give you the overview to final to finalize the talk the overview of the sort of technologies we've been exploring so coherence through chain of contracts we've been working here with the smart contracts and our main topic here has been first of all to identify what can and cannot be converted into smart contracts so when you go and look at these different documents that are clauses that are very difficult at this stage to be translated into smart contracts to choose a specification language we have been working with a symbol ale with some of our colleagues University of Ottawa that have developed a symbol ale as a specification language for smart contracts probably in our review of smart of languages for specific specification languages is one of the best we've found so far so what we've done on on this level is to try to formalize the contracts into smart contracts using symbol ale okay in order to be able to verify for instance that all the contracts through the chain include a six month or a two year period for a deletion of data now these works in principle fine if we manage to do that we have faced different difficulties in the formalization let's say it works fine the problem is that some of these companies have already thousands if not millions of existing documents already and no one will sit down to sort of try contracts right from the natural language into smart contracts so we are also have a research line on using nlp to retrieve the information from the existing contracts and populate the ontology populate the smart contracts categories or ontology that we have developed through the formalization process so this is the first part we've been you're going to have access to the power point so you're going to be able to see that what kind of elements we've been formalizing the second element is compliance assessment with regulations and here we are working on two different levels we started working with some of the existing databases opp for instance and we started very interesting we started using mostly us actually so we are in europe and we should have been using gdpr but we started using some of existing us data sets for these which were available to identify what is sort of a high level category of a data processing activity okay so what kind of data processing activities this for instance first party collection or data retention data storage so as a general category and for that we've been doing that through a machine learning i'm going to show in a few minutes but i'm not sure i have time to go into the detail but we've been retraining this and then formalizing the rules of gdpr to verify compliance of those passages and the documents with gdpr now the again the first level is to identify what is the paragraph about or where is the section about what type of data a processing activity it relates to and and then we have been matching if you see on the right hand side because again the data set we used to train our algorithms it's not gdpr we started matching sort of data practices categories that were identified by the usable privacy policy project for articles so we can match them at the verification stage last point so i guess to get into the time i'm think i'm kind of almost fine auditing and just since you brought it up a couple times we do we definitely have time and the questions that we're tracking are trending very some people are very interested in the gdpr part so okay feel free to to um go at your natural speed okay perfect i was i was about to finish here but then i'm going to get back to these just for a second so what was the challenge here is that the there is an annotated corpus of privacy policies in which we trained our algorithms now these privacy policies recognize some data practices that you have on on on the right hand side or on the left hand side as well first party data collection third party data collection data security user access and these data practices are further specifying with attributes now the challenge we had is that there is almost no available data set to train this algorithm to train our algorithms on this data and this that is available is actually it annotates privacy policies in the prior to gdpr so it's basically if you take this it's all non-compliant with gdpr so our challenge was to match if you want the data practices and attributes with gdpr articles in order to be able to use this matching when we would be doing the verification um i can't um i can do that maybe in the questions but we do i could provide you with a with a we have already a a software doing this it's not commercialized but we have it's a sort of research software that does in part this verification um now the last stage is the auditing and verification of effective compliance so this is this is the um this is the treasure right this is has the company actually deleted the data because you know lawyers in many cases are interested in for instance whether there is uh if there is already a breach in the contract but the document is already contrary to a previous agreement okay it's one thing but the other question is or let's put it like that with an example let's say that the first contract says you can keep the data for two years and some contract down in the chain says that the company storing the data will store the data for three years which is already contradictory to the first agreement right so there's already a problem there now the question that comes next is has the company deleted the data after two years or three years because if it is after or before two years well there is not actual violation right the data was indeed deleted within the time period that was initially established um and here uh which is just the greatest part we have not worked yet we have not worked yet because actually there is a whole set of recent papers and solutions that are being explored one more difficult than the other so what we have been doing is actually doing the literature review of this um but i can tell you already how we plan to work here to have an idea so basically um there are different protocols most of them using blockchain these are the the the five papers that are sort of key in some there are others but that we have been key for our literature review and basically the idea comes as how do you generate right how do you generate a um notification how do you include a notification in the smart contract to say that the data has been deleted okay so you can have of course a or or the or the data has been collected so one of the first ideas we have is of course to use iot for elements that are can be verified for iot but others are a bit more difficult and we need some software additional software development so the fact that for instance we need to verify the deletion of the data uh you might we might need some logs or we need when some techniques to verify the data has been deleted in the database but then again you need to make sure that you have access to all the databases or that you can verify in all the databases uh that where the data was stored and that's where some of the blockchain technologies that are being used now to trace data as it flows it's interesting um now the other possibility is just to have manual entries uh such as to say the data has someone very someone confirms the data has been deleted but of course it doesn't mean the data has been deleted and once but let's say that someone manually say the data has been deleted through the smart contract technology we can go all the way up the chain to the first consumer and business to let them know that everyone in the chain or at least someone in the chain has deleted the data so you still have this sort of challenge of automating this process but the advantage of the smart contract is that it could allow sort of the instant update through the contractual chain that the obligation has been fulfilled so that's very much what I have to share with you of course I'm happy to go more into the detail of the different elements and I'm not of course alone in this as it would be impossible for anyone to work on these different things alone so there's a great team I think some of them are connected and and some of the external collaborations we've had most of our team is composed of of lawyers and data scientists that work together so thank you very much again thank you for the invitation and I'm happy to answer some questions outstanding thank you so much David that was that was tremendous and we have elicited a number of very good questions and and we have added some of them that seemed common or particularly on point to one of the speaker slides and so TMA if you don't mind screen sharing again and advancing to the questions yeah problem um and then if you could perhaps uh retake the gavel at that point as our toast master and and pose and just pose the questions to David so that um so that he can just sit back and enjoy the show you got it okay it's not in the notes maybe I'll just know I have a question here I have a question here that I'm reading and I'm happy to start with that one if that's fine yeah yeah go go right ahead okay so there's a question about the open nature of rules uh and of course I mean this is one of the biggest challenges um one of the biggest challenges we have and um and that's why uh let's say from a more theoretical perspective we I really think that one of the issues is the hybridization of rules so we have to take the hybridization of rules seriously um rather than sort of think that when we are implementing for instance a smart contract we are translating a contract from natural language into code I totally disagree with that I don't think that's true I think we are just somehow rewriting the contract into some extent you can see it like that um there are some people that argue that you have like that you can see it as a contract stack to use sort of an engineering word uh is like a stack of different documents including the smart contract um but just to give a simple example at some experience I have my own uh when you want to do something like um the reasonable standard of a reasonable person right and and I like to give a very basic example with that that that hopefully makes things clear is that you rent your house to someone um and you say that the person has to take care of um the house or the apartment uh the reasonable person would do let's imagine all this is coded and in smart contracts and connected iot and stuff so basically um your garden um you the contract foresees that you have to take care of your garden as a reasonable person let's imagine this means watering the garden twice a week or we don't even know what does what does take care of the garden a reasonable manner means but when you are going to code these you cannot put it in a reasonable manner you have to sort of change reasonable into frequency so you have to say for instance that you have to water the garden twice a week if it doesn't rain that's what the code says and this is not the same thing as reasonable but there is no way of coding reasonable I mean you could do this in a more complex manner but in other ways you are going from a standard standard of reasonableness to rule and that's a definitely legal change because these are two different artifacts in law so may I ask David uh following up on that of so one of the approaches we take in the media lab to artificial intelligence for example is a we like to focus on what we call extended cognition so uh not so much replacing people with software and human processes with software completely but augmenting what people are doing and the way we're trying to interpret that or at least explore it in the computational law context might be at certain key junctures to invoke a human decision point for example like uh to ask uh whether the parties agree on whether something is or isn't reasonable or to if necessary go to arbitration or some other decision body and all that of course could in theory be set up in an objective way where you trigger a request for a response and some parameters you get back and answer and you continue executing the code but the judgment the human judgment perhaps for whether something is or isn't reasonable could still uh be situated with people um would that be one potential design pattern to address this yeah so definitely definitely that's and that's what we think from the design perspective and I can only but agree with you if we have the chance to start from the design perspective the the issue is that in many cases we have already systems in place like the high frequency trading that has to implement a definition on spoofing which was initially defined for a person or market manipulation right market manipulation it requires the intention for instance and and and once you need to change these into an automated system verification of algorithms performing certain uh uh exchanges of of of stocks for instance um it is very difficult uh or it's it's not very difficult but i mean depends the speed at which you are doing it but uh i i guess you can have nowadays more than 15 transactions of buy and sell in a second uh for uh high frequency trading so the question is how much human judgment you can include there uh so again i mean so i i agree but that's why the finance example is a bit probably dystopian to some extent for probably extreme but of course for the garden example we can totally look at from the at the design level that the ideal thing would be to do something like we just mentioned sort of from the augmented perspective deep so we i i see that we're we've only got about a little under 15 minutes left but we have so many questions we want to see if we could squeeze one more in um tm a would you do the honors yeah let's let's go with gareth's question so gareth asked you david whether you differentiate data and documents as a pragmatic matter or like more like an artificial differentiation data and documents yeah oh that's very interesting that's such a difficult question actually um so i would say it depends for what because i am a pragmatist so i for me the interest of a concept is related to its use um so let's say uh documents are for instance data when we are going to use it for a for a for training an algorithm right so they are part of the same unstructured data but when i probably in the context i was using it what i meant is that there are documents which are mostly contracts governing the relationship between two persons and then there is the data related to the contract let's say the data that has been collected of your consumer behavior and usually the contract is governing the data or the transfer of data from one company to the other so what i mean is there is some need what i meant probably is that the differentiation was just meant from the classic textual analysis between contracts let's say or documents and the actual flow and what's happening with the data that's sort of the way i'm using it i hope that was a an answer for the question that was posed well it's a good start um and and that's completely appropriate for the first session of of our class and hey forever it's worth i completely agree with you it's a deceptively simple question uh that when you pull on that string it raises all kinds of fundamental issues so david i just want to on behalf of the the whole team here and i i think i speak for the participants as well thank you so much for that presentation and for taking the time and engaging with some of the questions uh as well we have many more questions than we had time for at this point um and so um you know perhaps we'll we'll make those available to you so that if you if you were able to you know shed any more light i'm sure we'd welcome it but no pressure thank you desa thanks again for the invitation and to everyone i'm happy to follow up on questions and and comments with everyone and again i mentioned we had some already developments in software that i didn't share because i guess the time was limited but very happy to follow up on this as well with everyone so thanks again for the invitation and your questions outstanding thank you thank you okay so yeah timmy i'm sorry you've got the gavel okay i was going to say now we have some time for brian to go over supplemental readings and the group project which i think will be an exciting thing for everyone to participate in and actually i'll do the readings and then pass it to brian for the project um so these are readings that we suggest that you do uh and um and think about uh and we are going to be offering office hours for discussion um we think that these readings are going to um catalyze a lot of questions and we we hope some interesting ideas uh and i'll just say right now we will follow up with an email with a zoom link for those of you who are able to um to participate but the we're we're planning to have the office hours uh this coming Tuesday January 12th at 1 p.m eastern uh and we'll block out an hour of time for that and um we'll and you'll be off mute so we'll be um having an opportunity to actually actually talk um one quick note on the these readings is uh well the first one we think is seminal from sandy pentland um the second and third you may notice the names are also teachers assistants and uh or excuse me the second and third both come from people what when they were students in the class it was this was uh this began with their student projects and so when brian tells you about student projects um you may think about doing a paper or or the pitch or the other opportunities and who knows uh it may end up becoming publishable and may even become part of a curriculum one day um and uh that last one by um gabby on democratizing the law with open data i have to say is i understand it's somewhat us-centric but nonetheless i really commend uh everyone to at least glance that it will help set some of the foundation for understanding what we even mean by the law as data and and what those workflows look like from a more practical uh perspective okay so with that i'd like to hand the gavel to brian who is going to um tell you about optional projects that you can do all right so if anybody is interested in doing a course project uh you know we heavily encourage that um we're coming at this from you know kind of this foundational standpoint that law has always been an algorithm and i think david's presentation did a really good job of highlighting the different ways that this is developed historically um through the convergence of law and other disciplines and so i think that was a a really key point that can kind of serve as the foundation for how how to think about projects as we move throughout the rest of this course and in true mit fashion we're really focused on building and creating as a means to explore these kind of unique tensions between law data computation and design um so to that end we're offering students and participants in this course the opportunity to present either a paper a project or a prototype and to start out we've come up with this google form that will be available to everybody to kind of sign up to and we'll use that as a way to communicate with you so that we can help guide you in the direction that you want to go with creating something hopefully with the end goal being something that is you know either publishable something that creates a business venture or something that you can actually demonstrate and prototype um so obviously to participate just sign up for the google form and we'll follow up with more information and then we'll include instructions later on about how to use github as a means to submit your projects to this course so that this legacy kind of endures far beyond just this four weeks and really lasts into the uh into the coming uh months and years because we think you know the the insights here are going to be ones that are truly groundbreaking and can really serve as a great sort of reference point for people who are interested in this sort of work in the future so i'd encourage everybody to sign up and if you have any questions um please feel free to submit those to the the google form that we have for questions as well and we'll be sure to get those answered and with that you know i'll hand it back over to tma and she can kind of tie this all up and get us uh get us ready to go thanks brian yeah i'm noticing that a lot of you in the chat want to create a sort of telegram group so Courtney if you want to send me the link at the end of this class we're going to be sending an email with um david's presentation and other resources so maybe we can include that in the email as well so everyone can access it if it's a class link i should just say we we need if it's a telegram or something like that we we've had them generated by other people in the past and it frequently doesn't end well when they it can be difficult to administer or maybe they use their account so if people want a telegram group we can do a telegram group we've done it in years past we'll send you a link to uh to go to come into the telegram group and we can do that very quickly so um whoever suggested that good i good suggestion uh and if you've created one already please just delete it so we don't have like you know multiple telegram groups which is another thing that we've experienced in the past and that doesn't go well either uh could we advance the slide please great there we go okay so yeah we'll include our telegram group in the email that you guys will all get um after this class so thanks again david for your amazing presentation and this concludes our very first lecture for this year's MIT computational law course thank you guys so much for being such wonderful students and if you have any questions come up before our next class there's a form that we've linked on i believe slide 11 can you just double check ryan if that's there yeah oh no that's the project slide so we'll also include the slide to the question form as well this way you guys can submit any questions and we'll get back to you before next class so thanks again everyone and we'll see you back here next week goodbye we've got four minutes uh as well so uh if could i just have could you please advance the slide to the next steps okay so just to just to be explicit about it um read the readings office hours Tuesday the 12th at 1 p.m we'll send an email with that and also with the telegram uh group uh sign up and also with the link to be able to pose questions uh ongoing uh sign up for the project that ryan just told you about it's optional but if you have time and you're interested it is a great way to learn to actually get your hands on a project and you'll learn github along the way uh david's classes in github our classes in github understanding how github works is is one of the skill sets that's helpful for actually doing computational law and uh a teaser for next week uh we're going to be getting into standards and so some of what we heard about today from with uh data supply chains and um auditing them will go more deeply in a couple of um uh domains most importantly perhaps being supply chains uh so we're going to look at uh edifak and x12 and some of the other transaction and supply chain standards and how those how all those little transaction codes like uh you know like give me a quote um do an invoice um a purchase order and acknowledgement and all of that stuff how that all rolls into the trading partner agreements and can allow for um significant auditability and also raise some of those interesting issues and questions that david um david um highlighted to us so actually we think looking at the standards is one great way to get your your head it's almost the same as like reading the statute when it comes to transactions that have been expressed through code um okay so i just wanted to sort of say that out loud so to make sure that you heard it um quickly there was one question about uh the due dates for the projects and you know there doesn't have to be a due date for the projects if you want to keep working on it after class but we would uh like people to submit a project before the last class and so we'll have this kind of initial form as a way to sign up and express interest and then we'll have a subsequent form where you actually include the links to all of your final presentations as pull requests in our github repository it's fine okay i guess now if you don't mind doing it again tia ma you like bring now is the official end to our very first lecture so yeah we're excited to see all of your projects and hopefully you guys can check out professor streffo's research and his amazing presentation which i'll definitely be looking at there are so many good diagrams of yeah anyway okay we'll see you guys later bye thanks bye bye


Well, people will be peddling their way through Lake Bemidji State Park's trail system Saturday as part of the fourth annual Fat Bike Rally. For one day only, the trails at the State Park are turned over to those who fat bike. Throughout the week, the park crews have been out grooming the trails for the big event. An estimated 70 fat bike riders are expected to participate with mild weather and slushy conditions. Participants are expected to have a great ride with some fast times. We are going to have actually really good trail conditions, knock on wood. It is above freezing during the day, but they are grooming it all week, so that allows a time to set up, and then every night it's freezing. So it should be a fast course throughout the day. By the end of the day it might get a little bit slushy, but that's alright. Proceeds from the event go to support the Truck North mountain bike team in Bemidji. The race begins at 9.30 with the race beginning at 11. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


Say, everybody, look, Tuesday, what's Tom saying? It's Wednesday, I forgot we were all far Monday. I thought yesterday was Monday, so welcome back on this Wednesday. Good to see you all, Wayne. Yes, I'll tell you, what a market we're having. We had so many good stocks that did so well. There's VVOS for crying out loud. Unbelievable stock, great little move. Looks like everybody can hear me on YouTube, so which is great, that's a good thing. So that means we're probably gonna have a good day. But so far, everything looks awesome. Let me tell you, yesterday, I was talking to Josh yesterday, and I know you guys are in the cyber group room, but we gotta talk a little bit about the stock Jasper. I haven't seen, I don't think we ever see a stock got halted so many times. Go check the stock out really quick. So let's check out the stock Jasper right here. This was ridiculous stock yesterday. Now, pre-market was nothing great. We'd have a great morning in pre-market, but then all of a sudden, we're watching the stock, and it was on the watch list. And all of a sudden, you could see around nine o'clock once the meeting started, and then we were around here. Boom, we were already up about, we went from, I don't know, 80 cents to a dollar 20. Now listen, please, I don't wanna hear all the bashing. People like to trade penny stocks. It doesn't matter. We don't trade penny stocks in Nasdaq national stock, but money's money at the end of the day because this thing literally ran, here's 9.30. It got halted right out of the gate within 20 seconds. You could see how it ran from like $1.40 all the way to about 2.40. It's a dollar for crying out loud. Then it backed off, came down hard. Going back to a dollar, it was so volatile. And then all of a sudden, right around that 10.30, boom, boom, boom, boom, boom, all the way to $3.80. And then it did it again this morning in pre-market when program trading kicked in right around eight o'clock in the morning, the thing ran from 2.20 all the way to 3.40. Now the reason why this stock is extremely volatile, we were talking about it in the cyber group room. Obviously some of you here in YouTube and Facebook didn't, you know, we're not part of the trading. You always could join us if you want. But one thing that we noticed the reason why there's a lot of halting going on is because there was not that many outstanding shares on the stock. So it really makes the stock extremely volatile. So that's what happened with that. But it was just a really cool situation to see how unbelievably, you know, hold after hold. And you know what, there weren't even like five minute holds. There were 10 minute holds on that stock. But it was a great, a great runner up. I know a lot of you guys did very well on it. But you know, listen, there was a couple of the stocks out there I also want to bring out. There was another one, ANATNF, that moved pretty nicely right here. You could see how that stock had a really nice move at 9.30, ran from like 3.80 all the way up to about five bucks. Then it kind of, right around here, around one o'clock it finally broke all-time highs. And you could see how the thing ran to 5.50. That one also was pretty good. Now it's good as Jasper. And then the RSLS was also very awesome too. That at 9.30 ran from like $9 all the way up to about $19 a share. That's $10 of a run up. So if you came here for the first time, that's what you missed. If you finally came back to your vacation, you're like, all right, let me start at 2000, you know, all over again. This is basically out. One other stock I also got out of is that I did take a loss on Tesla. And thank God I did, I just want to show you here. But Tesla, it started trending down, obviously news came out. I got out right around here. I didn't like it. I got out around 116. I own it around 118, 119. And good thing I got out of it, because I didn't like it. And remember, what did we trade off? Level four. And you could see it, that it ran from all the way down to a price of 106. So that's a pretty substantial crush right there on that stock. It did come back a little bit. I was going to buy it back. I'm not fully sold yet, okay? I always like, people always ask me like, when's there a good time to buy a specific stock? And I'll tell you when. When you get the worst of the worst of the worst news and it doesn't want to go down any lower, that's when you buy. And it's like the same thing they say about real estate. When's the best time to buy real estate, okay? When nobody wants it, all right? That's the best time. Same thing when it applies to a stock. Now, regarding about what's going on this morning, listen, got a couple of really nice moves. VVOS, you could tell that thing literally had a really, really nice pop right here. VVOS, I know some of you guys were trading at pre-market, but what a great run up on that stock right there. I mean, literally at eight o'clock, I had that nice 730 program trading move. And you could see how it ran right there. And then obviously it was backed off a little bit. And then from a dollar into a dollar 50, huge, huge winner there in pre-market. Couple of the stocks I like to keep an eye on. There was this stock up 52%, not really crazy about the trend of the stock. It did trade 800,000 shares though. So that is alarming to me. And that I'm definitely gonna put on there. GERN, another great little pre-market trade for some of you here. Unbelievable move right there for everyone here. Look at that beautiful 7AM run. You could tell I think just literally ran right up there. Great, great pop right there. The stock is up really, really nicely. It's up about a dollar right now, but GERN is going back on our watches. We've done a lot of you remember we traded the stock in the past. Long time ago. GERN, I know it's got good tier sizes, it's tradable, everything else. Plum is, I don't know what's going on with this stock right here. INM, you know what? It's been a really good swing trade. You could see how it kind of built a little momentum here. I don't know about the day trading aspect of it, but the stock was around $5, went to $17. I remember that. We traded back in November, ran from $2 all the way to five. So we'll put it on our watch list. We'll put it on the B list. INM, we always have different lists. Look at that, we're already at 13%. You see how things are kind of more or less dropping down there. That pretty much does it for the NASDAQ. Here's a stock getting destroyed. Now, is it a pharmacy? Okay, typical. All right, pharmaceutical stock, we know they don't come back. So don't try to focus on trying to get dead cap bounce on that one. ENVX, this stock, another one getting crushed. It's got great volume. Trying to come back a little bit. This is obviously, I don't know if it's, I don't know if this is a pharmaceutical stock. I gotta check it out. The best way to find out if they're pharmaceuticals, they say right here. You see when you get that word right there? When you get that, the T-H-E-R-P-H, that word right there, that long real, when you see something like this, I don't know. But this is basically what it is. RSLS, we know why that's coming down. That was the one from yesterday. Don't wonder why it's on the losers from the gainers. Let's go hop over here in the New York Stock Exchange to see what's going on over there. Red 9%, not really anything attractive. Yeah, remember, go-to's back on there. I forgot, we traded some of the Chinese stock yesterday. They did pretty well too. Baba's in there. You can see go-to's back on there. Hoya, these are all these Chinese stocks that we traded. They did pretty well too. I don't wanna go down the list. Here's the stock that's getting crushed too. It doesn't look pretty ugly on that chart, breaking all-time lows. Yeah, nothing really jumps out at us on the New York. All right, so the VVOS looks pretty cool. Looks like it's starting to trend up a little bit. We got some resistance levels back here in July. You could see here at $1.50. If you got level four, you should probably see some iceberg orders really nicely on the VVOS. I would love to show it, but for some reason, I lost connection. I gotta reconnect it again. So it just went down on me, but that's fine. It's like the first time in a while. Another stock that's moving pretty nicely too on a swing trade. Merck made a little bit of a pop here. So Merck has gapped up a little bit, breaking all-time highs. So I kinda like that stock's working out pretty well. And we'll see how Carvana does. All right, so that pretty much our watch list, everybody. Gern's moving. Is it? Let's check it out. Yeah. Grant, didn't move it up a little bit. Got a nice little pop there. Like I said, we got Gern on the watch list. That was pretty good. All right, anything else? I don't think we missed anything else. I think that's all of them. All right, Cyber Traders, let's get ready for the market opened up. We're up about 93 points. We'll see how the market pans out. Like I said, just because we didn't find anything in pre-market doesn't mean it's gonna get a nice little push from the market opens up. And you saw how that happened with Jasper yesterday. All right, good luck, everybody. Don't forget, today at 12 o'clock, open house here at Cyber Trade University. If you're watching YouTube Facebook Live, you wanna learn how to trade day trade swing trade. You wanna know how to follow the dominated market makers with high frequency trades, dark pools. Just click us and like us. Put the link over there. You can see it's up in the chat room. Cyber Traders, you're all also very welcome. And if you're new here to Cyber Trade University, a trial member, hopefully you're looking, listening and learning. Good luck, everybody. Happy trading, be safe. We'll see you then.


All right. Good morning, everyone. Or afternoon, I guess. No idea what time it is, apparently. All right, can you all hear me? All right, good. So, well, I'm excited to be here in Vancouver. This is actually my first open infra, so I get to kick it off with a talk. So, yeah. So today, my name's Kevin Hannon. I'm an open source maintainer on a project called Armada, which is a cloud-native computing foundation. It's under the cloud-native computing foundation as a sandbox project. And I'm going to talk about building a research platform using OpenStack and Kubernetes. Now, a little caveat. I will not talk too much about OpenStack. I won't be able to do it justice. Over there is Scott, who will be giving a talk on Thursday at 11 a.m. to talk about how we actually provision our bare metal nodes for Kubernetes using Ironic. I'm excited to attend that talk. And I hope you all attend the go. It's at room 8.15 at 11 a.m. on Thursday. And so without further ado, I wanted to give you a little background about who I am, what interests me, and why I'm here giving a talk. Well, I actually started my career as a computational scientist, mostly focused on utilizing hardware to accelerate modeling and simulations. So I, and in my masters, I was actually focused on running, on doing quantum chemistry, trying to explain chemical properties. And that's when I got interested in parallel computing and how you can use a lot of the really nice tools that are out there to speed up simulations. And I kind of got away from that for a few years. And then in the last couple of years I've been working. I had a job working at the National Institutes of Health, focusing on trying to run scientific workflows on high-performance computing clusters and or Kubernetes clusters. Machine learning is the elephant in the room. And if you've ever tried to provide any kind of platform, you know that there are machine learning people that want to use whatever cloud tools there are for deep learning. And it's actually kind of difficult to do that on a high-performance computing cluster. And it's also kind of difficult to run traditional model and simulation software on Kubernetes when most people have not even containerized that yet. So there's kind of this weird disconnect in the community. And so my first entry point into this field was just trying to say, I want to run a workflow. And I want to run it on either a Sarm cluster or Kubernetes cluster and have a similar API between both. And guess what? It doesn't exist, unfortunately. It was a tough task. Well, I realize that I think Kubernetes is a pretty powerful construct for having both. You can run Kubernetes on-prem and in the cloud. And I actually think there's a lot of work to be done to improve that. And so I'm actually now working at G Research under the open-source division where I am focused on better enabling batch workloads for researchers on Kubernetes as part of Armada. And I'm also a contributor to Kubernetes. So if you are interested in talking about that, please follow up with me. So who is G Research? Well, they're a trading company based out of London. We actually have a US location opened up in Dallas. And we are heavy users of OpenStack and Kubernetes. So if you are interested in looking at some job postings, we have quite a few available in the Dallas metro area around OpenStack and infrastructure. But generally, for the purposes of this talk, all we have, like, G Research, what we provide to quantitative analysts are, since it's a trading company, you can imagine we're trying to predict trends in the stock market. And we provide a really large research cluster for quantitative analysts to kind of just figure out trends. Now, what does that really mean? It can mean anything from data engineering to machine learning to trying to run ML simulations faster. And so we provide a really wide range of use cases. Every day when I talk to the researchers there, I find out about new things that they're actually doing. So I can't even do it justice all the types of workloads that we're actually providing for people to run. But so generally, G Research is focused on running batch jobs. Now, the definition for a batch job really is a finite lifetime job. Just means a job that will run the completion. It's actually kind of interesting the timelines depending on your domain. Like, in my masters, you could see calculations run for months. And that was considered a batch job. I've seen some researchers have jobs that actually run for years if they don't have enough hardware to speed it up. It's kind of amazing. But so generally, there's a lot of areas here. Computational science is one that I know and love, machine learning, elephant in the room, especially with large language models, and data engineering. And another area I like to talk a lot about is genomics. It's kind of a pretty scientific computing community. It's a little bit slower on adopting containers. But there is one, I think, one shining light in that community, which is the genomics community, where they've released a lot of their tools, free and open, via Docker. And they also couple it a lot with workflows. And one of their main focuses is trying to create reproducible science so that you can continuously run your workflows over time. And be able to reproduce those results. And so a lot of times, they have containers and workflows kind of coupled together.  So just an interesting one. So what are we trying to do? Really, there's kind of two areas of parallel computing that I like to think of. One is embarrassingly parallel. This just means if you can split your task into a series of bold-defined inputs and you require no communication, this is really easy to run on any kind of cluster. Kubernetes actually supported this out of the box. Day one, or maybe day one, I don't know if it was day one, but very close to that when they released it, they had a simple job API for representing batch jobs. But generally, what gets a lot harder is when you actually need to use all of the hardware on your entire node. And you know why? There's really two reasons why people want to use parallel computing. One, they want to speed up their simulations and they want to run faster. Two, they want to increase the science or the research that they are doing by leveraging more compute to answer harder problems. And so generally what that means is either you're using hardware to accelerate a problem or you are trying to split your problem up to different nodes so you can actually run your simulation. And so this gets really tricky because you actually have to kind of have a scheduler built in that can grab the nodes that you need and then have communication between those nodes. And that's usually done through a lower level layer called the message passing interface or MPI. And then there's also a lot of GPU-GPU communication that can happen in PyTorch frameworks and stuff like that. That is usually hidden under the hood. But generally, this is kind of a harder problem to solve. So what are our requirements? Well, generally, G-research, we run large amounts of batch jobs, but I want to keep saying it. Our order, we're usually on the order of around a couple million jobs per day. And this varies widely on the types of jobs. So you can see our target for this is around 100,000 nodes, a million cores. And we provide a lot of GPU accelerators for our quantitative analysts to try and predict, to try and speed up their simulations. And I think it helps to give a little bit of context about where G-research was and why we kind of built what we did. Generally, we were already running a lot of workflows that were batch jobs. And we were actually using HT Condor, which is a popular, it's a scheduler popular in the grid computing space. And then it was primarily targeted for Windows. And then we were like, hey, Linux is great. It's my great. And then, of course, a lot of pain ensued. But we're closer now. And so I would like to say that most of our workflows are container-first. And we actually run on Linux. And we are also using Kubernetes for our container orchestration now. But unfortunately, there are some problems with that. And this is three years ago. One, if you try to run millions of jobs on a single Kubernetes cluster, you're going to be in for a bad day. There are a lot of things that can actually happen. We've seen cases where your Kubernetes cluster goes down. We've seen cases where your API server gets so slow that you actually, it will lose jobs. And generally, it's not great for it. It's not really set up to run large amounts of jobs. So you kind of need some kind of queuing, scheduling system at the layer above Kubernetes if you're using an on-prem cluster like we are. Because you have quant, so researchers, that want to run more compute than you have nodes available. Maybe everyone has that problem. But generally, that's one thing we notice. We give them more compute. They give us more jobs. And so it's annoying. So generally, what I like to think we're providing for our quantitative analysts is a serverless platform for people to, most of our quants don't know anything about Kubernetes. They barely know anything about containers. They just say, here's my task. I have a build. It's now a container. Now I want to run it. And they don't want to know about the hardware. They don't want to know very much about any of that kind of stuff. And so that, of course, makes our day, because that means we have to implement that as something else for them to worry about. So generally, we provide a UI for our researchers to interact with so that they can know what kind of stuff is running and be able to tail the logs, be able to track the status of their jobs. But they don't have to know about. They don't have to use Kubernetes to actually do this. Well, why do we even need Kubernetes? Well, generally, there's two avenues of thought for running this kind of types of stuff. You can either go all in on Kubernetes, or you can look back and look at other options in one area that I'm familiar with as high-performance computing and have a little audience interaction.  OK, so about half the room. And so generally, users kind of, they have to SSH onto a login node. They interface with the scheduler. And then that scheduler has hooks in there to say, how many nodes I want, how many resources. And then if you don't have enough capacity, you queue it, whatever. That's great. It works pretty well until you start trying to get into stuff like cloud bursting, or you have a researcher that wants to submit a job from a slurm cluster on their laptop using a Jupyter notebook. And this gets much harder. This is where it starts to show that you might need to start thinking about an API for your cluster that users can interact with if they are saying, I don't want to SSH into this thing. I just want to be able to use this cluster. And I would say a lot of the ML data science community is kind of pushing this idea of here's a Jupyter notebook. I don't care where it is. I just wanted to run. Give me all the GPUs you need. I'll find it somewhere. And unfortunately, that makes the traditional HPC style workload a little bit trickier. So now you have Kubernetes. It's great, right? I'm sure.  OK, so half and half. So generally, one of the nice things about Kubernetes is users, once they have access to a config that defines their cluster using a Kubernetes config and they have a CLI, they're usually pretty much good to go as long as they know about Kubernetes. I know I said a very heavy statement there, but it is what it is. So generally, it's a pretty nice pattern for programmatic access. I remember when I first getting Kubernetes workflows running, once it was all set up, it's actually pretty good to get stuff started. You have a lot of tools. And if you want to do multi-cluster, there are some other options you can do, too. Unfortunately, it's not perfect. I would not be up here or have a job if I could say that Kubernetes is perfect and I don't need to work anymore. It'd be a great day. But generally, Kubernetes, there's kind of one way, like I said, 100,000 nodes is our target. Well, it's actually difficult to do 10,000 nodes with Kubernetes, so what are we doing? Well, and also, generally, I should say that it is actually quite, it's even more difficult to run a lot of nodes with Kubernetes. And there's some good posts from OpenAI talking about all the things that you need to consider if you're running at, they have one post, which is 2,500 nodes, and the next one is 7,500. And there is just a lot of engineering effort to make that reliable and scalable. And generally, if you're running your own infrastructure like we are, that's a lot of work on your engineers. So one of the things we actually, we cap our Kubernetes clusters around 1,000 nodes. And then, but now you're saying 10,000 nodes, 1,000 nodes per cluster, well, how are you doing that? What are you gonna do? Well, we actually use multi, we use multiple Kubernetes clusters.  And I thought it was kind of a neat thing to, oops, wrong slide. So what I'm switching here to just a terminal. So I'm using Kubernetes and Docker, which is just a simple way of provisioning very small Kubernetes clusters. And so, sorry, that is a little... So generally, I have two clusters. I have an Armada test and a demo one. And if I wanna switch between different clusters, I can just switch between two different contexts. And that's actually a pretty, I think a pretty powerful way to actually have multi-cluster. And I won't submit them. But generally, you can submit a work, you can submit a YAML file and then tell a certain context which cluster you actually wanna target. And you really, this is actually a pretty easy way to programmatically call and submit things. So that's kind of for multi-cluster Kubernetes. This is great. Now what is the, one of the problems is, is a lot of implementation leakage. If you go this route, your user needs to know which clusters are available or which clusters are there. What if you change the name of your cluster? What if they have a script that they wanna run every day and if it breaks, you're gonna get a call at three in the morning because you're doing, I don't know, maybe you won't be doing a maintenance effort at three in the morning, but maybe there's a cluster that goes down and it doesn't work. Well, what we wanted to try and do was, we wanna provide a way for our quants to interact with a stable API and then do a lot of our maintenance and our rebuilding of our nodes on our Kubernetes clusters without having to take down the entire farm without having to disrupt our research. So we created an open source project called Armada. At a high level, Armada has a, we have, we're like a hub and spoke architecture where we have our server is actually what users are submitting their pods to and then our UI is looking in the server to get the state of the world. And then we have our bread and butter for execution is in these Kubernetes worker clusters called executers. These are the things that actually talk to the Kubernetes API submitting pods, kind of doing the cube control, create kind of stuff. And then we also, our executers are pretty closely coupled to Kubernetes. So we use the Kubernetes scheduler and generally we keep pretty close to it because I like to say if you fight the platform, the platform's gonna win, unfortunately. And so if you try to get away from it, upgrades are gonna get harder and harder. Maybe after a couple of upgrades, you'll be fine, but maybe after a couple of years, you're gonna be like stuck in, a version from a couple of years ago and no one wants that, engineers like new and shiny things and so do researchers. So generally, we have an API and we provide a series of clients. Our API is defined in protocol buffers so it's easy to generate other clients. We have, we're a big.net shop so we have a.net client, a Python client and a Go client and we also have a CLI. I'm happy to say that we have our first Google Summer of Code start this summer who's actually trying to make the CLI part of the cube control as a plugin. It's his summer project and so we're excited to have him working on that. And you can see our API is pretty simple. You create queues. I didn't talk too much about queues but they're kind of the way that you have a resource, how much resources a user is allowed to use and you can easily create them, delete them and then you can submit a job. So I have this Python client. Well, what can I use for it? Oh, I'm sorry. I jumped out of myself. So generally, what is the unit of work? Well, we have what we call, we have a custom job spec and series of metadata which is what our queue is, our role-based access control for our resource limits for our job. We have this thing called a job set which is really just a, we do a lot of our models usually coupled with a workflow engine in our case and so typically our users like to title their, all their jobs are gonna submit as a job set so they can actually view the status of all the jobs without having to pull each individual job. And then we have a pod spec which is actually a Kubernetes pod spec. For those of you that are familiar with Kubernetes, it's kind of similar to like a pod template or a pod template and deployments or stateful sets. And what's powerful about using the Kubernetes API is we get, unfortunately, we get a lot of stuff that they like to use like init containers which can be challenges in its own way. We also have a list of containers but then what's powerful about this is if your users wanna target things like GPUs or they wanna start using fancier features of Kubernetes, we actually inherit this with our API. For those who are not familiar, taints and tolerations are ways to target a GPU node from your pod spec. So this is how users can say I wanna submit a pod and have it run on a GPU. And then generally that's a lot of that stuff. And so part of our job spec will be also enable. I won't talk about too much in this demo or in this talk but you have a pod spec but let's say you wanna have like something like a Jupyter notebook. Well, generally that it's not just a container, you would also need a service and ingress. So we actually provide that functionality when users submit. It will also create a service and ingress so our users can actually interact with a notebook via a permanent URL in the timeline of their job. So well, some use cases that we have with Armada, one of the top ones, we use a lot of workflow engines and g-research and some unfortunately are in-house. And I was happy to say we were able to convince them to look at some open source solutions. So we've been running actually Apache Airflow. For those of you who are not familiar, Airflow really pioneers this idea of workflows as code and a lot of people say the powerfulness in Kubernetes is an extensibility. I also think that's true for Airflow. You have hundreds of providers that you can use with Airflow. The main gotcha with Airflow, which I think a lot of people get trapped by is you really do not wanna be doing compute in Airflow. You wanna be running your compute in a compute environment such as like, in our case Armada or Kubernetes and that's where your job actually runs. And Airflow is there for orchestration. And so generally what we wanted to provide was we wanna start looking into ways, how can we use Armada in Airflow and have Airflow orchestrate workflows and have Armada run those individual pods. Well, so we created a Armada Airflow operator using our Python client. And this is a pretty simple script that just is running a simple hello world followed by Armada running a simple job. And this is a sequential task, running one job after another. And so I will switch to two demos. So generally what I will show first is, I will show our my Python client that I have. Okay, that's pretty good. Oops, not good. All right, so generally you can see here, this is a pretty simple pod spec. I'm running a container, it's just doing sleep because I don't trust myself to do anything more complicated than that during a demo. And then this is running a simple Ubuntu container that is sleeping and then my API is creating a job request and giving it a priority. And then I'm also creating queues and a job set. And then I have a GRPC client that's actually hooking this up and then this is kind of running this. So I have a demo available for anyone to check out, a demo server. And so this is free and open to use. What I'm doing now is I'm running this script and this is gonna submit 1,000 jobs and I'm gonna switch to my webpage. So up here, this is our website, armadaproject.io. We have a nice little demo page which is running an EKS cluster for Armada. You can click on this and then you can see this is our an overview page in our UI that kind of shows the queue is called openimfra and I have 129 jobs queued. And I'm gonna switch to the job panel to walk through this a little bit easier. So what is queued? Queued means that I have, I actually don't have enough capacity on my worker clusters to take my job so it is waiting on a message queue for a new, for capacity. So there's a lot of jobs queued because I don't have that large of a cluster running this. Pending is a state of it's in the Kubernetes, it's actually running in Kubernetes, it's pulling the container down, it's doing all that. And then running means it's actually running a job and then succeeded actually means it is running. So what's nice about this UI is you can click and drill down into individual jobs and you can actually start seeing some more information about your job. And these are just relatively simple stuff like this is by default, we add stuff like IDs, queues, all that. And you can see here this is when you get the information like which cluster it actually ran on and all that. And you can go down here and look at the job YAML and see a series of what the actual job that your user submitted. And so that is, that's that part of the demo.  So I'm just gonna show the Airflow integration. So generally, yeah, so Airflow has its own UI for representing workflows. This is, these are just a series of example DAGs and then you go scroll down the bottom here. You'll see a Hello Armada, which is the DAG that I created for this demo. And I can view the code here and this shows you that I'm actually using my Armada operator. And this is defining an Airflow workflow and submitting a simple sleep job and then running a DAG called Hello Armada. And yeah, it's very simple, the one that I already showed you. And then I can switch to, I can manually trigger the job, which just means to run it now. Usually Airflow runs on a schedule. That's usually what Airflow's bread and butter just to run on a recurrent basis. And then you're able to use Airflow to drill down into individual tasks. And then we kind of leverage our UI to get more information about what's actually going in the progress of the job, like what is the state in Kubernetes. And this way we can have, we can still leverage Armada and Airflow together. And that is, you can click on this task and then if you scroll down to the bottom here, you'll see that I have a series of, this is my Hello Armada DAG, this is actually what Airflow submitted and we're able to tag our job so we know that was submitted by Airflow. So not too much left. All right, oh, come on. So the last piece. So generally if you are interested in checking this project out, as an open source project, we also have swag. If you are interested, we have socks. They're nice Argyle socks. Please, I don't wanna carry them home. So I would love, if anyone wants them, please let me know. But generally on a left hand side, we have a little contribute page for how you can get involved. And this also links to a Slack channel that you can join as part of the CNCF Armada, CNCF slash hashtag Armada. This is a nice way you can engage with the community. We are actively growing. And we have our code, central QR code and then our website armadaproject.io where you can scan this QR code, which is, I love that Google has cute little dinosaurs. It's my favorite. But anyway, these are pretty much it and this is, if anyone has any questions, please feel free. You can go to, that's a nice place to talk. You can go to Scott's talk over here. He is giving a talk at 11, no, not 11, 11 a.m. on Thursday. And that would be a good way to understand how G-research is using OpenStack. A very quick question, sir. You said that HPC schedulers, likes normal PPS, are scalable but not user-friendly at home. So the other is Kubernetes is more user-friendly but not so scalable. But still you're building on Kubernetes. Are there any plans also to tackle the other big areas with the HPC schedulers to enable them in use? Sorry, let me repeat the question. You are asking whether or not is there work in the HPC community to make that more user-friendly? I know, is there a way for you, for Armada, to support also the HPC world? Oh, that's a good question. We actually have been talking lately a lot with some of the traditional HPC schedulers, especially in Slurm, to really think about ways to kind of maybe mix and match different schedulers. I believe pretty strongly that every tool has its use case and I think it's kind of, it's important to leverage all the tools you can. And so we do have interest in looking into that but it's really quite difficult. There's a lot of work being done in Kubernetes to try and bring HPC-like syntax to Kubernetes and then I don't see as much about trying to say, I want Kubernetes to run Slurm or vice versa but there are some people starting to look into that now. I think part of the challenge is just there's a lot of legacy around the HPC schedulers that they're building containers and adding that in also while they're also trying to add Kubernetes which is a layer above that. Hope that answers your question. Hi, I'm just trying to work at data machines. How do you deal with the overhead latency of scheduling as you scale up the number of jobs that you have and specifically the overhead latency just starting up these jobs? Does that become prohibitive when you're still using the Kubernetes scheduler as kind of the job submission mechanism? A big part that we run into is depending on what's the size of your containers. I mean, it's a very wide range of what, like we'll have users that might want to, they are, some of them using .NET might be using the machine learning frameworks in .NET so you can imagine those containers get really fat so we can have, you know, we've seen gigabytes worth of containers and it's actually kind of difficult, like no matter what we try, it's hard, like pulling the container off the internet can be really difficult, especially, it can take a long time and we do build in some caching layers to try and make that faster because usually our quants don't like to see more than a few minutes stuck in that pending state which is the Kubernetes state for saying I am pulling images, I'm starting, KubeLit is starting, all those kind of stuff. It's not quite yet running but it's trying to get the container up and running but generally those are those problems and there's obviously in Armada we have the layer of, you know, sending our pods to different clusters but usually that is much smaller scale than pulling our images from our internal Docker registry. Do you have like a pilot demon set that can pre-stage or do data scheduling for you to speed that up at all? I actually don't know that answer but yeah, it's a good question. We can follow you up if you really want to know. Yeah, it's a good question. The question was how are queues different from namespaces and generally they're very close and you'll see a lot of cases where we use, we have some internal tooling where we actually, we create a queue which actually creates a namespace with the right resources and generally we have some interesting usage patterns in G-research where each quantitative analyst has their own queue and they have their own namespace and usually they're one-to-one but we could imagine a world where people would want a team to have a namespace or a queue and they all have that kind of similar namespace so we could imagine a more, like many-to-one type of relationship but they're usually pretty close. And the second is would you advise to run jobs with Armada on a cluster that is already running other production typical Kubernetes deployments? Yeah, I would. I mean, so we, I think generally, I mean, a lot of it is, it depends, like, yeah, I think you could, yeah. I mean, I don't think there's too much overhead on running them depending on what your infrastructure is. We typically have our worker clusters being dedicated fully to Armada but I have seen some cases internally where we actually might share our hub cluster, our server with other resources but generally this is a multi-cluster Kubernetes thing so we have kind of, there might be some parts that might share our workloads with others. Thanks. Of course. All right, well, thank you all and I promise socks so if anyone wants socks, I told you I have a lot actually. So. So.