Hi everybody. Thanks for coming to the second iteration of Performing Knowledge. I'm Cori Tamler, and I'm Margaret Edwards. So Performing Knowledge, this is the second year that it's happening. It was conceived and put together last year for the first time by Amir Farjoon and Mara Valderrama, who are both students in the PhD program in theater and performance here at the Graduate Center, collaborating with Frank Henschker, who's the director of the Siegel Center. Margaret and I are also students in the same department, and so we've taken over it over the event this year and made it our own and built on the structure of the event. The idea is that it will be an annual thing that aims to build a community of scholars and artists and interdisciplinary folks who are working with ideas around knowledge and how we shift our relationship to knowledge, how we share it with each other, how we know what it feels like to know something, what it feels like to share knowledge, and all of those kinds of things. So it's a very open space, and part of our idea is that knowledge isn't passive. Knowing isn't passive, so we do really encourage you all, as audience, throughout the day. For however long or short you choose to stay, to talk to each other, to interact with each other and to engage with the kinds of questions that these lecture performances, knowledge performances, now and earlier in the day, in other spaces throughout the grad center have sort of brought up. And there's not time for, it's pretty rapid fire, there's not going to be time for formal audience talk back or discussion, but we'll be encouraging you to talk with each other during the 10 minute transitions from one piece to the next.  You can talk, you can run to the bathroom, you can engage with your neighbor. And there's also an audience response station outside by the table in front of the theater. And we would love if you jot down some questions there or respond to some of the prompts as a way of also kind of like activating that in lieu of a formal talk back. And finally, there will be a reception at the very end of the night, which is some hours from now. But if you're still here at 8.30, after the last performance goes off, please join us in the green room, which is the theater department lounge. It's upstairs on the third floor, room 3111. And there will be some snacks and drinks and another good place to talk. So we don't want to talk for too much longer, but the way that this process works, we've been working on these pieces in a kind of collective dramaturgical process since the beginning of this semester. And so some of the pieces have their own dramaturg, and, but we also have all been in conversation throughout and we've had several group meetings and a lot of exchange back and forth across the projects as well. And sort of we're thinking of dramaturgy. You might be most familiar with dramaturgy as a term as something that refers to the structure of a piece, the structure of a play, if it's a term you're familiar with at all. Many of our participants were not. But we are thinking in a kind of expanded form of dramaturgy and thinking about dramaturgy as something that really deals with the relationship between ideas and current events and ideas in the space that they're in, ideas and the body. And that's sort of where we're coming at how knowledge comes out and is expressed and is shared, sort of thinking about the dramaturgy of knowledge. So even if pieces don't have a specific dramaturg listed or attached to them in the program, they have been through some of this kind of dramaturgical process. And you'll see a lot of the strings showing throughout the day. We're not hiding any of the transition, any of the things we're doing. These are all works in process in some way. So just embrace that and feel free to take part and be loud. And just finally before we start, we wanna say a huge thank you to the Siegel Center for supporting this and being a good collaborator in this work. The theater department, the doctoral students council, the doctoral theater students association, the university student senate, who else is it, the Sydney Cone chair, and all of our incredible volunteers. We've had a ton of volunteers help us out throughout the day and who are continuing to help us out. And Mike, who's up in the booth, who's making all of the tech magic happen. Thank you so much, Mike. And May, who is also part of the Siegel Center and is doing amazing support and some dramaturgical work too for this whole thing. And did I forget anything? All right, all of these performances, one more thing. All of these performances are about 20 minutes long and you can come and go. You don't have to stay for the whole part. Other people will probably be coming in so that's also part of it. But we hope you'll stay for as much as you can. And with that, do you wanna introduce the first one? Yes, so I will introduce the first knowledge performance by Tomo Imamichi, dramaturg by Elio Akinchi. And it's called 20 Minutes on a Treadmill. Hello, welcome to 20 Minutes on a Treadmill. I am Tomo Imamichi and my field is environmental psychology which studies the relationship between people and their environments. One way to do this is exploring it via challenging tasks and challenging environments. Now, this is somewhat of a challenging task and this is somewhat of a challenging environment. Running is one thing, running on a treadmill is another. Also, facing all you wonderful people is also something quite different. I feel that I'm a little underdressed today. Usually, I would dress up a little bit more but this is also one major thing about the transforming relationships because over the course of the next 20 minutes I'll be getting much warmer even though you'll be still not so warm maybe. So one of my projects involved looking at marathon runners and how they basically deal with the task of training for and running in a marathon. Running in a marathon and then to look at the transformation that occur over the course of training and over the course of racing. An important part, of course, is to pace yourself. Something that I'm trying to do right now. Because as part of this performance, of course, I would like to talk about running but I would also like to demonstrate running and in order to do so, I set myself a goal. To basically go at an eight minute per mile pace which would allow me to cover about four kilometers or 2.5 miles, which is not that much of an accomplishment but considering that I'm also trying to run give a presentation about running, I think that is a pretty decent performance. Actually, if I were on the road right now, I probably could run up to Queens. So going back to pacing. In the beginning, the same pace will seem easy when I first set the pace. I was wondering, am I going to slow?  And of course, I'm only doing this for another 15 minutes. So I think I'll be able to manage. However, in the marathon, it's really key that you have to hold back at the beginning even though it seems slow. Then eventually you settle into a pace and then it almost happens by itself everything. You fall into this rhythm where everything sinks like the rhythm of the feet, the breathing, they're all going in sync and it becomes easy and it allows me to look around, taking the environment and this is actually also what most of my participants reported. But that is not the defining aspect of the marathon. Usually at around 20 miles, so the marathon, you would have another 6.2 miles to go. This is usually when it becomes challenging. This is also known as the wall. And then that pace that seemed easy suddenly becomes very challenging. And then you have to really push forward and there's this disconnect between objective time and distance and then the experienced time and distance. So that last 6.2 miles or the last 10K are usually the hardest part of the marathon and what makes the marathon a marathon. But of course, this isn't the only time and only aspect that I explored about running. I have also engaged in other project that could be situated in something that we might call moving methodologies or moving epistemologies because it is through moving through the environment. Well technically, I'm not really moving through the environment right now. I'm definitely moving though. But anyway, by some form of embodied activity it allows us to know more about ourselves and the world. And in a way, I'm using this performance not only to perform knowledge but actually also to generate knowledge. Because I usually do not run on treadmills and certainly not on a treadmill in front of people. However, in a way, I was looking forward to this experience just to get a sense of what it feels like and it feels pretty hot. I'm noticing that I'm beginning to sweat and I'm the only one in the room here I guess. Of course, the other part that I was looking forward to was in generating some knowledge in the Q&A session that we'll have for the last five minutes because oftentimes I can get inspired by some good questions or comments. And the thing is what better way to answer questions about running when you're running? Because you can take runners and have them talk about running but when they're not running how well can they talk about running? And I always notice that with myself too because some of my best ideas happen when I'm running and that might be in part explained by the activity of moving but it could also be that I'm inspired by the environment as well. What happened? I might have exhausted the treadmill and I still have ways to go but I'd rather have the treadmill break down than myself because that was also one of the concerns that I had is usually with a presentation. I also was somewhat concerned about not just the presentation part but also about the running part and then of course there are a couple of things that can go wrong. Sometimes it's the runner other times it seems to be the treadmill but I'm glad that it's you and not me. Yes, yes, so unfortunately I'm not on the treadmill anymore but I was going to do a little sprint too. There was definitely a moment after you talked about falling into a rhythm with the breath in the body that I saw you fall into a rhythm. There seemed to be a moment that your body, your rhythm, the words, everything seemed to kind of and I wondered if you, when you felt that moment if you felt that moment because I definitely saw a change in your body. Yes, there was a point as I was warming up that I was finding my rhythm and I think that also helped my presentation and it's quite interesting when sometimes people when they think about things how they like to pace up and down for example I think it has something to do that our thoughts sometimes come when we move. One thing that fascinates me about the transfer of embodied knowledge is that once it leaves the body is it still knowledge and then what is it exactly that gets passed or maybe a more pointed way to ask is what knowledge were you hoping to share or to perform specifically for us today? And I'm not sure whether I'll be answering your question but one thing is also that sometimes when I have certain thoughts and they occur while they're running they seem very specific to that activity and to that environment and then once I'm somewhere else and try to re-articulate them it seems very difficult that there's almost like that sometimes it seems that I leave my best thoughts on the road. So that was something that I think your question slash comment inspired that I find is very key. And also now that I'm off the treadmill it doesn't seem as genuine anymore to talk about the running. So I'm just sort of building off some of the other ideas that have come up I think it's really fascinating that idea of finding a rhythm and yes I think we've all had that moment of that kind of clarity that can come in physical movement and I'm wondering what you feel like how much of that is psychological and how much is physiological or whether it's perhaps impossible to... It's very interesting that people pick up on that because it's also, I should have mentioned that I also think of the concept of flow and I think this sort of like experience you could also call some form of synchronicity and I think where those boundaries between self, body, environment seem to disappear so I think it's contingent on that holistic experience which in a way a treadmill indoors is not quite the same as running outdoors I think Okay so I think we are out of time so thank you very much for running with me and yeah too bad for the treadmill Alright thank you Take the treadmill to its grave Quotidian ritual and festive, thank you Festive performances where food is a key feature and my focus is on ordinary people, the practices of ordinary people because I don't really care about rich people but for the purposes of today we're going to talk about rich people Also colonialism is a wonderful, magical thing for my work because it's responsible for the transnational diversification of food practices in this time period so that means I get to have some dissertation chapters and also that food in Europe during this time has something we call flavor but we're not going to talk about those things today either Today we're going to talk a little bit about the early modern aristocratic kitchen a little bit about banquets some things that we encounter when we examine our recipes and we're going to do a little bit of cooking so we can guess from someone's excited we can guess from printed books during this time period printed books about household management that mostly in ordinary households women did the cooking but these books were printed for women who had servants so literate women were expected to read these books and impart that knowledge onto their staff basically it's only in the 18th century that we have evidence of independent women actually making a living as chefs so surprise, surprise in noble early modern households the chefs were mostly men and he had loads of helpers he was paid very well he had dozens of meat as a bonus on holidays and if you've ever read any early modern recipes you know that these people really loved their meat and the head chef would have complete freedom when it came to hiring or firing staff for his kitchen lords would treat these chefs really, really well so that they would remain loyal so it's no wonder that no one ever got poisoned or entire households never got poisoned so banquets during this period were quite theatrical and could incorporate entertainments with allegorical themes and elaborate centerpieces also music this painting for example depicts the Lord Unton who was a diplomat during the Elizabethan period and here he is presiding over a banquet while a court mask is performed for his dinner guests the staff that worked with the banquets were also very specialized and very large for example you would have a special meat carver who would suspend an entire bird on his carving fork and slice pieces into a plate and it sort of looked like this there was also a specific person who bought preserved food specific person who bought the fruits you had a butler who would buy the dairy and the cheese and you had a wine steward and then beneath all of these special guys you would have an army of pages, servants and water bearers so at a typical banquet you definitely had more people serving than people eating food was also quite elaborate and spectacular at the banquets written by Vittorio Lanzalotti who was a food writer is from a banquet at an Italian court the first cold course consists of a veal pie in the form of a shield bearing the arms of the king there is a sugar glazed ham surrounded by pastry baskets with pistachio pastry stars touched up with gold and silver next is a roasted turkey larded with candied citron the ends of which are gold and silver as are the sugar pastry wings, neck and tail the second credenza course consists of an egg and sugar sculpture composition with a pastry crown, garlands marzipan rosettes and gold and silver touches there is also the curious invention of gelatin columns filled with little birds and fish suspended within as if alive and probably my favorite is the third credenza course where they had dueling marzipan germans as well as a little scene of dancing rustics molded out of butter so when it comes to presentation the early moderns were not playing around especially when it comes to birds particularly the peacock which was often roasted whole and then afterwards they would sew the feathers back onto the body for presentation the other thing they did with peacock was to use the head, wings and feathers to decorate meat pies that's an example of one that's another there are quite a few issues that come up when we're dealing with early modern recipes for one a lot of them look like this this is a page in the recipe section of a commonplace book belonging to a man named John Crophill who is a medical practitioner and a bailiff and it's dated between 1430 and 1485 somewhere on this page is a recipe for tarts of flesh so do we have some early modernists in the room? can any of you read that? alright do we have real early modernists in the room? I'm kidding I can't read that either because I haven't done the paleography training but even if you could read it you would have to wonder what some of these words even mean and we have resources like the OUD and lexatons of early modern English which help us figure out what the words are neither of which I used for today but I will tell you anyway what it says it's take the flesh of capons or of hens and pork and mix with figs, raisins and ground hard boiled eggs add raw eggs, saffron, ground ginger, cinnamon, gallangal sugar and whole cloves and maces then put them in a case made of thin pastry make another course from turkey or clover or other heavy game bird and another course from ground meat and eat them together well menj here either means mix them together or eat them together I'm hoping it means eat them together anyway today we're not making tarts of flesh because we're in New York we're gonna make the pigeon pie but New York pigeons are fucking vicious and really hard to catch I brought a chicken this recipe for pigeon pie comes from a lady named Hannah Woolley who is an English writer and she published books on household management the short title of her book is The Accomplished Ladies Delight and here she is looking very accomplished and here's the recipe we're gonna follow today trust your pigeons to bake and set them and lard the one half of them with bacon mince a few sweet herbs and parsley with a little beef suet the yolks of hard eggs and an onion or two season it with salt, beaten pepper cloves, mace and nutmeg work it up with a piece of butter and stuff the bellies of the pigeons season them with salt and pepper as before take also as many lamb stones seasoned as before with six collops of bacon the salt drawn out then make a round coffin and put in your pigeons and if you will put in lamb stones and some artichoke bottoms or other dry meat to soak up the juice because the pie will be very sweet and full of it then put a little white wine beaten up with a yolk of an egg when it comes out of the oven and so serve it we're gonna break all of that down in just a minute but just note that we're not given any measurements so to make a pie we need to have a crust first and for meat pies early modern recipes call specifically for a paste for thin baked meats this is a recipe for a paste for thin baked meats from Joseph Cooper chef to King Charles I so this is some royal crust alright? the paste for your thin baked meats must be made with boiling liquor as follow-up when your liquor, which is water boil it put to every peck of flour two pound of butter but let your butter boil in your liquor first in contemporary cooking we call this a hot water crust pastry and it's ideal for meat pies because it doesn't leak soggy fun fact pastry crust was never originally intended to be tasty it was meant to be used to carry food around in a safe little packet like a bento box kind of anyway we're not exactly told how much boiling water to use but I'm a much better cook than I am a scholar so I wasn't too worried about that and two pounds of butter is a little bit insane but what's more important is what is a peck of flour so according to manuscripts cookbook survey a peck of wheat flour is an understood weight of 12 or 14 pounds however in other recipes a peck of flour is a volume measurement of two gallons which would weigh only 8 to 10 pounds so a peck of flour is 12 or 14 or 8 to 10 pounds and I don't know if any of you bake but that's a big crust that would require quite a few chickens never mind pigeons so I've reduced it just a little the thing with hot water crust pastry is that it needs to be formed while it's still warm the top of the crust didn't make the subway ride so it needs to be formed while it's still warm so I made the coffin at home and I brought it with me now to make things go a little bit faster I'm going to need your help so I've broken down the rest of the recipe into these numbered steps so I'm going to start with the first one and I need you to read them aloud to me and I'll do it as I go okay alright so number one trust your pigeons to bake trust your pigeons to bake and set them and lard the one half of them with bacon so this is the chicken I've baked it already in the name of food safety because you know we're in the US and you guys can be fussy about that and it's been basted in its own fat so this is the chicken okay next please okay so for sweet herbs I mean I'm not sure what that means really but I use tarragon I use rice and I think some sage and I used the entire egg because I'm a graduate student and I'm not throwing away a good egg white and parsley so we're going to mix that all up and then where's the butter here we go a piece of butter how much would you say is a piece of butter all of it all of it there we go this is why I brought the big spoon because I knew someone was going to do that alright so we're going to mix that up that's good so now we're going to stuff the belly of this chicken that's good enough next step please with this alright next please okay so the juice that Hannah is worried about lamb stones and sweet breads can be a little bit divisive so I decided to soak up the juices we would use some potatoes now in the early modern period around the beginning of the 16th century potatoes were imported from Peru and north, north, north, west Bolivia but today our potatoes are imported from the exotic land of Idaho and that's what we're going to use to soak up the meats so we're just going to chop that I think that's great I think so last step please alright so since we don't have a top crust right now Margett is going to go make it in the back and she's also going to stick our pie in the Siegel Center's wood-fired oven thank you Margett don't forget to brush it with the wine and egg when it's done it doesn't tell us how long to bake it but my educated guess it's going to be like three to four hours so we have just enough time for some Q&A so would anybody like to ask a question yes testicles lamb testicles anybody else well actually time's up you're never ever going to know so thank you not done yet because the early moderns did not send a naked pie to a party and and I tried I really really tried to get peacock for you guys today but they didn't have them at Whole Foods so I brought you a crow trigger warning it's about to get graphic so if you're like the queasy type you should probably leave so here are the wings maybe one here does that look even to you great then where are my special tools there we go so if you're around at 8.30 today please go up to the green room for the reception where this pie will be served you guys were a great audience thank you so much so just as we set up for the next one again I encourage everybody to talk to each other in the transition but also just a warning about the next performance that if you have sensitive hearing this one has some very loud feedback and sort of high pitch sound so anyone with sensitive hearing might not want to stay for this one and we are going to start in about 5 ish minutes and so if you need to run to the loo and come back you have time thank you if you'd like we can have a few minutes for questions for our early modern chef if you would if you have any further questions for her that didn't get answered what was the one that okay welcome back we are ready to begin our next knowledge performance so I'm actually really encouraged to see all the conversation engendered I hope by the performances that you've seen so far our next knowledge performance is by vered engelhardt and carolin heiter and it's called a tuning detuning and retuning self our score and we're going to pass it around so that you can look at it yes it's plexiglass it's been carved a piling I guess we it's a pile of a series of scores that we did over the periods of rehearsals we one would start with a certain proposition and we would try it out see how we feel about it things we like things we don't like as far as control and improvisation and then we would come to the next rehearsal with a suggestion about how to frame the piece more and different things that we could score and other things that we would leave open to interpretation improvisation so the three systems of lines on the staffs are our vocal score that we were following in a cyclical repetition and then the duration or the amount of times that we would do each system is in the grid underneath the grid is a series of repetitions that layers up with the different sort of variables that we start adding on to other things in between technologies of the self technologies of the self it's a way of performing knowledge of the self oftentimes I think the performance and working on a piece of somebody else is a way of understanding yourself and when you are in collaboration together then it becomes about exploring and understanding the different ways in relation to one another that you process and internalize and then express whatever is open to expression and then the way that you adhere to the scored material as well the form of the duo also the duo format is such has such a history in therapy and other forms of self knowledge we are saying like what if composition and performance and tandem could help us achieve that sort of form in a mutual way but also against self knowledge so there is this constant coming together in a part I guess yeah it's a kind of collaborative creation of knowledge of the self and interpretation of knowledge of the self and performance of that and generation of that where it's all happening obviously in this sort of cyclical feedback loop where you kind of can get trapped in any other questions? what are the the context of our artists when we work together? what are the context of our artists when we work together? other artists that do this Paulina Rivera concepts that we read it over to well through co-assay technologies of the self it was very foundational in the creation of this piece yeah there was a Foucault there and then there's Pauline and then there's Pauline the different ways through which she scores her material yeah you had a question it wasn't exactly the question and some you know wasn't just yeah no I mean just a short response because we're out of time but when you're in conversation narratives always arise it's the way you tell yourself to the other so that was today's narrative it was just today thank you everybody should we go back in? let's just go in oh I have one welcome to How to Human Future Form would like to thank you for joining us today here in New York City for a very special one-time only event a training in Disruptor Mechanism Protocol brought to us by Elle shit I can't pronounce this Elle Something yes a very special training with our guest put your hands together folks these different places and when we're not able to live in these divided ways we're going to develop better communication devices that remove those blockages so there is some pure sound stuff that's gonna happen and that isn't our topic for today but we could have a whole other session about that if you go out in the lobby there's some really interesting posters from Future Form about their post-linguistic technologies super interesting so but that's not what we're here for today today we're talking a little bit about something unfortunately most of us have missed that's how to human how to human we forgot to learn you forgot to learn we figure it out a little later this could be you successful feeling confident in your humanoid body mind I'm here today with a little protocol set called the Disruptor Mechanism what's a Disruptor Mechanism anyway it might just be the answer you are looking for Systemic Collapse Planetary Collapse humans love the word Planetary Collapse well alright you worried a little bit worried maybe about the sixth extension pretty concerning but I also have bad news and so the good news is that a protocol like the non-linear alchemical Disruptor Mechanism may be able to forestall us moving in that direction but part of the good news is also that actually the word Planetary is kind of inaccurate because the planet will be fine and the planet and everything on it including some form of yourself will emerge again as the Earth figures itself out all of the organisms and all of the energies that are here are infinitely capable of moving itself through another system of homeostasis and restoring balance all humans may die in the process don't worry no one but you is really worried about that in the same way but that's a capacity that you as humans have developed it's an emotion it's a flag and the fact that you're having it now is part of the other good news the good news is that those flags are here to help you they are an evolutionary device that you have evolved to have so that when you get here you can start to say hello we may need to do something else unfortunately the sort of problem that in not really ever learning how to human like how to use your body which is something that in any other thing that you do you'll learn you're not a computer scientist if no one's like here's a computer here's how it works but unfortunately the form that you have no one really ever gave you many tools to use so we're here today to talk just a little tiny introduction in the nonlinear alchemical disruptive mechanisms are an intelligence augmentation embodied cognitive field technology from a set of books a long set of books from the fuken operating systems manual specifically volume 3 which is humanoid body mind systems lots more to talk about with that today but oh huh wrong way funny that nonlinear mind thing scurs me up every time alright so as humans we have incredible capacities but most of them have not been developed today we have three different types of protocol that I'm going to introduce to you that are going to help you develop some of those capacities and even though we only have a few minutes if you add these up it's like that adage that humans love so much it's kind of like water breaking away from a stone yet a little bit at the time and if systematized in your daily life these will add up to something that can create change and the ability for you to auto evolve the first of these is the diachronic biophilic recode and it's great to come into this talking about water on a stone because the diachronic biophilic recode asks us to remove from ourselves what that means for us to think about humaning verb I want you to think what it is to stone what is it to stone what is it to mycology what is it to animal what is it to rhizome these are the things that you want to start trying to get into your body because you have the capacity to be there so moving into spaces to ask the question how would a mushroom map this environment how might an animal see this room it isn't the way you see this room but it's hard for us to not think that our understanding of it is a priori accurate it's simply not the case we're stuck in that mapping yes the map precedes the territory someone in your century or so said that so number one diachronic biophilic recode and you know plants have been trying to help you with nonlinearity for a while a lot of your indigenous peoples also helped you out with that the language that we used to have did not always have the causality that we are now stuck in we're going to get there again forward which exists in that place you as you which isn't mushroom or mouse but in this humanoid body form goes into a room and you have an experience but you don't have an experience anymore because you have a word that's attached to that experience and you can't get rid of that word and so you're stuck here and so in order to move us beyond the stuckness that we're in we've got to work with that those words that stick us there yeah we're just stuck in them we walk around and we're just in this word soup and so amazingly we have a cognitive plasticity that is activated when we use different kinds of words so if you use words different from the words you normally use your brain actually becomes more plastic and you start to adapt and evolve so by attempting like an active use speculative language you will move yourself into a future that is liminal that doesn't yet exist thank you to the non binary people and everyone using the new pronouns they are living in a speculative space that does not yet exist but through the use of that language we facilitate a body that is prepared to be in that space and then the space follows the language use employing this in your life on a daily basis creating neologisms and living in a liminal space that is semi cognitive starts to move us out of where you're stuck great all right stuckness it also lives in the humanoid body in a way that if you don't work with agentively none of the other stuff is going to work the next problem is you've just got all this stuff in your nervous system it comes from trauma it comes from your experiences and you as a humanoid body are different from the other things in your biome the other plants and animals are phylogenetic which means that they will learn all the capacities of their body no matter what you are not like this which means that you will only learn if given the capacity to do so that kind of sucks but luckily if you know that this is the case and you remember that this is the case and you start to actually be like hey what are the capacities of my body how can I control my parasympathetic nervous system how can I know that in this moment I'm feeling fear because I learn to feel fear here and I can tell myself that that fear is not actually something that's happening fear structurally true but that immediately I am not being chased by a bear and so I can breathe in a certain way so that my mind can work better and my blood can work better and I can make a good decision and so when you learn to do these things you are more able to take on the other disruptor mechanisms and put them into place because you're not flailing around with your heart beating out of your ears hyperventilating which is much more convenient and easy to do but it isn't your fault that you don't know how to do this with your body in fact the whole Cartesian problem that's been happening for centuries where you were taught to separate your body from your mind and you learn that in school which is where you're supposed to be educated and it's all that's all right you've been told to silence you've been told to ignore it you've been taught to be intelligent parts of yourself everything that your body is saying to you everything it's saying to a kid saying move around that kid should move around everything it's learning will only be learned if it moves and yet we tell that kid to sit tight not a first relearn the semantics of our bodies if we're going to be able to put the other mechanisms into place that's kind of where you're going to have a choice because before the good news is planet is fine but maybe none of you will be there none of the emergence the plants, the animals, the machines and you but not cognitively because you haven't developed this nonlinear capacity yet are working towards resetting we're overpopulated a lot of people have to die it's not an emotional thing it feels like one because you have a ball to have that capacity you are sad no one else is sad not in the way that you understand sad but maybe that sadness can help you stick around a little more I wish I had time to stay here with you today we could talk about this for so long and you'll probably be thinking about it for the rest of your life how do human is really complicated but on a daily basis we can bring in these little mechanisms to perhaps stick around for the next chapter thank you so much hello so welcome back our next performance knowledge performance I should say is by Simone Johnson is by Simone Johnson dramaturg by Stephen Cedars and it is called the oceans are changing yes but to me it feels like strong drawn out changes I can't really articulate my reasons for being drawn to the abyss reasons hope things make sense this is their home these gelatinous creatures possess luminous organs to frighten off their enemies feel deeper 1000 meters below the surface in the ocean twilight we find as a form of force these are the largest kinds of plankton some measure up to 50 meters their organs are not spread out but grouped together on one side they have their stomachs on the other they use to float between the two they deploy a net to trap any food which falls from the surface descending into the abyss is like traveling back in a biological time machine there are maybe 2000 species of abyss dwelling fish a terrifying beast cheery fiber fish that can eat things larger than themselves helper eels that gobble everything which passes survival is tough at 1500 meters deep at around 3000 meters there is no more light it's the kingdom of the vampire squid a prehistoric cousin of the squid the vampire squid has blue blood laced with copper its enormous eyes detects light variations in contrast above it towards the surface its vision is entirely adapted to the shadows drawn out changes the abyss reasons what am I doing here poetic oceans poetic oceans merging imagining myself as the sea the sea imagining itself as me this is their home I wanted to say thank you to Steven the drama character I looked with and to Leah for making these wonderful algae lamps these are actually an algae culture in here and in here and I most of my oh I I wanted I wanted to say thank you again to the drama character I worked with Steven and Leah who has created these wonderful algae lamps there's actually algae culture here kind of algae that Leah knows more about and she actually created these with her hands so I was just like this is so cool and what I was working with was just I make a lot of work around water and I tend to want to make things that make sense and that are logical I was telling people earlier like shaping and strategizing because I want to get a meeting across and I wanted to be purposeful and I really I've been exploring surrealism and magical realism as loads of research and in surrealism they explore like the unconscious and intervening in things making sense so it was really hard for me because I kept trying to say does this make sense is this and I use like the the eggs came from a dream I had about a bat and I know in surrealism they use dreams and I researched bat symbolism and like it was like rebirth so I thought of eggs and so it's just really interesting to kind of I feel like I've been doing a lot of surrealist work but it's I so much want to get something across so that's what I was playing with and this is all in the context of trying to think about the relationship of oceans and climate change and I haven't really heard a lot of conversations about like I didn't realize ocean plays such a big role in absorbing carbon and I could go on and on but it's going to be a year long project and I'm just really grateful for all the people I worked with and for Margaret and Corey for organizing this amazing festival so thanks LG plays a role in taking carbon dioxide out of the air so that's one thing I started looking into with LG and then I started looking at ways to use it as a material for art making as lamps and as ink because there are a lot of sustainable uses future uses for LG people are looking at it well thank you and thank you Simone for your work thank you everyone we are going to take a break now our next this is the end of part 1 part 2 we'll begin at 6.30 please come back or send your representatives at 6.30 and that half of the evening will go until till about 8.30 and then the reception upstairs so thank you all very much I'd like to give a big thank you to all of the knowledge performers who presented this afternoon thank you very much


Hello, everybody, and welcome to the November 2023 episode of law.mit.edu's idea flow, a relatively informal set of discussion and demo sessions where we highlight what we think are some of the most interesting or important, relevant and timely happenings in the area of computational law generally, and nowadays specifically that means applications of generative AI to help solve legal use cases or to achieve, you know, goals that that we may have for law and legal processes. We've got two really interesting examples of just just those types of projects and initiatives and to get us started today. I've invited some friends and colleagues at consumer reports innovation lab to show us some really interesting and useful things that they're starting to do with privacy policies as part of a innovative application that puts more data control back in the hands of consumers. And so with that, I'd love to introduce Ginny, and also Dan, and if you could introduce yourselves and your roles, and then dive right into this extremely cool application of generative AI that you've, you've been devising and starting to deploy to get on top of privacy policies on behalf of consumers. Take it away. Thanks, Daza, and thanks for inviting us to be here. It's really awesome to be with all of you. I've attended this many times and seen a lot of great presentations here so it's an honor to get to present to this group. My name is Ginny Foss and I lead the product in R&D group within consumer reports innovation lab. So my team is responsible for a lot of research and fast prototyping about how we use technologies to power consumer protection use cases. And I'm here with my colleague Dan Leininger. Hi, yes, I am Dan Leininger. I'm the head of experimental engineering in the innovation lab at consumer reports. And very nice to meet you all. So today the two of us are going to let you all peek into the hood of what we have brewing here in our innovation lab. All of this work is very early stage experimental work. None of it has been deployed to production. However, the way that our lab works is we are building products that are serving consumers every day. And we use those products to help us figure out which problems we want to prioritize solving with new technologies. And so specifically our group has done a lot of work for a few years now in the area of privacy and consumer data rights. So this has been one of our kind of closest partners and co conspirators in this work, but consumer data rights are a new kind of fundamental right that we have as consumers in this country. These are rights that are enshrined under new laws to privacy in states like California and about a dozen others. And these are laws that give you a right to your data in various forms, the right to tele company to stop selling your data to tele company that they should send you a copy of your data, as well as to tele company to delete your data entirely. And these are rights that we've been really excited about a consumer reports because we think they give consumers a whole new kind of agency in a digital marketplace. We're doing a lot of research as well as experimentation and even product development around this idea of consumer data rights. I'm going to start by just showing you a product that is our kind of flagship data rights product that we just released, and that will give the context for some of the experimentation we've been doing with large language models and privacy policies. So I'm going to share my screen and give you all a glimpse of our new product, which is called permission slip confirming that this screen is visible to the group. Looks great. And so permission slip isn't a mobile app that helps you take back control of the data companies have about you. It's available on iOS and Android. And what we did with permission slip is we wanted to make it really easy and seamless for consumers to use their new rights to their data, their rights to privacy across the country. So what this is is it's a, it's an interface that allows you to find out what kinds of data a company collects. And we have a whole library of companies in the app, and we read the company's privacy policies to and interpret that for you so you understand what data a company like Home Depot has about you. So how do we give you as a consumer easy ways to manage that data by activating your data rights. And so we offer two rights right now in permissions left one is we let you send a request and tell a company to stop selling your data. And then the second is that we allow you to send a request that tells a company to delete your data entirely. So what's important to to understand from these screens is that a lot of what our team is doing under the hood every day is reading privacy policies from companies and interpreting them so that consumers can make an informed choice about how to manage data. So let's just segue into exactly what that process used to look like in a manual sense. So I'm hoping you all are seeing a slide right now about our teams process to help to to educate consumers about the kinds of data companies collect. So, what that process has been is that we collect copies of a company's official privacy policy. We have a consistent way of coding these privacy policies. We look for the kind of data companies collect and we try to articulate that data in one of 17 different buckets, which are the 17 types of data that different laws stipulate exist. And then we put that into this kind of nice and hopefully easy to interpret interface in the permission slip app. So for a company like McDonald's you can see what identifiers they collect about you what account information they collect about you etc etc. And this is something that our team has been doing manually for many months. We've hired people who are pursuing their PhDs and usable privacy to help us interpret these privacy policies on behalf of consumers. But what we've found, especially as you know chat GBT was announced there's all of this interest in large language models and how they might be applied. We realized that we had like a really clear cut use case for ways that a large language model could help us take in large numbers of privacy policies, interpret them consistently, and then show information to consumers in a way that helps them make an informed choice. So I'll show you all just kind of what this has looked like and then Dan will do like a live demo after we show you some of the kind of pre recorded behavior of this research app that primarily Dan has developed in partnership with with my team that runs the permission slip app. And so we had 24 steps or 24 things we would look at when we looked at a company's privacy policy, as we tried to interpret that privacy policy for consumers. And what's been cool is that we've been able to use large language models to automate all 24 steps of of kind of interpreting a privacy policy and then spitting out a distilled version of it that a consumer can use. So I'll just show a pre recorded demo and then hand it over to Dan to show you a little bit more in depth how the research app works. So let's say we want to add a new company's this is a company that we don't have in permission slip right now, but it's a company that we think consumers will want to manage their data with. So in this example we have Tinder the dating app has a lot of sensitive data about people. What you're seeing here is that we're able to get a description of Tinder, we're able to quickly fetch a logo. And then we're able to put in a URL for Tinder's privacy policy. And then we put that document to our database. And then here we go to the section of the privacy policy that is about the information that Tinder is collecting. And we snatch that section as and kind of use it as the context. And then Dan presses this create company button in the demo. It takes a few minutes. I'll actually move on to the next video to show you all what happens when this data is generated. So what you see here is what we've been able to glean from the privacy policy using large language models so we have been able to answer lots of different questions about privacy policy we even provide the source text for what parts of the privacy policy answer the questions. And all of this information is information that is then kind of fed into the reasoning behind how we advise the consumer on how to manage their data. And so what you see here is an experimental branch of the permission slip app. And this is an experimental Tinder card that we just generated in this demo. So you're able to see the 17 categories of data. You're able to see some advice on what you might consider if you're going to tell Tinder to stop selling your data or tell Tinder to delete it. So I'll stop sharing my screen there and hand it over to Dan to give you all more kind of in depth summary of kind of the process we went through and how this all works. But I guess before handing over to Dan just any questions on like the problem we're trying to solve or the general approach before we dive into some of the specifics. Firstly, let the record reflect that in chat one of the completely random unbidden comments was simply one word. Wow. And I concur. But but having said that it does raise so many questions and we've got a great group here. So does if anyone has questions, I'm feel free to pipe up at this point. Also, we can hold them until after after Dan's part. But Jenny gave a really great overview and please if anyone has any questions and wants to do a discussion of this. I'll go a little bit more deeply into, you know, some of the specific questions we were trying to answer and maybe some of the problems we ran into, and some of the work that we're thinking about going into the future. But yeah, please feel free to ask questions along the way. I think that I'll just get started in the back of my mind. I'm almost always asking the same question when I see this type of application, which is what actually was the prompt, and were you were you using. If it was opening I were you using a system prompt in conjunction with specific prompts. And then also how did you the other half of that that question about prompts is the input for the output. How did you control the output to basically regularize its shape and format so that it would fit into this very structured, you know, output that that that you need so that it doesn't start, you know, kind of like homespun, you know, things like hey thanks that's a great question. Here's my answers or other like extraneous verbiage that we don't want. Yeah, exactly. And here I'm going to try to give me a second I'll bring up some of the prompts that you so you guys can check them out and yes you're right. And all of this stuff and in my experience, working with LLMs generally and working with frameworks with LLMs. Like, always just sort of dive down until you can figure out what the prompt is and what people are talking about is is the best way to sort of step back and see what the system is trying to do. So I love that question does it and just give me give me one second here and I'll see if I can bring up a good example of one. And while you're doing that just for those who are here who may not be completely steeped in the technology or what we're even talking about. One of the things with this generative AI that was remark those are applications that you would have heard of would be chat GPT is the big one but there's there's quite a few is that it's natural language. So what that means is like the the prompt isn't like a sequel query or a Boolean or some something like that. It's you kind of it's it's conversational almost in the way that it works and that that provides all kinds of interesting opportunities to apply it for legal use cases where you know a lot it could hardly be more of a like a word and language based field.  Great. And thank you. It turns out that I cannot bring that up and I'm going to bring up just some code so I have a newer version of this app that goes into some detail, a little bit about what these prompts look like but instead I'm going to show you just on some code and we'll go through them here. Second. Let's see. So, you all see that now. It's good. Great. So, what this is a lot of this logic is the goal of this was to try to get these things to to export Jason. So, our goal like going into this there's a lot of ways and even in the releases that open I release yesterday that's like you can just sort of specify Jason coming out of this stuff but when I first started digging into this and this is still where it exists in our app there's a bunch of ways you can do this with a Jason schema and other ways but we were sort of. What I saw one example of someone using TypeScript schemas in order to identify the things that you wanted to get out of it, and then using comments in the code to identify more specifically what you want to get out of those definitions and so this app is basically based off of that and like I said you can do this with Jason schema and other ways to sort of define what the structure of the output is, but this is an example of a query that is we're trying to get out all of the data rights methods from a, from a privacy policy and those data rights methods are like the your how can a consumer and authorization exercise their right to, you know, opt out of sale exercise the right to and so this is the prompt that goes in and does it and you can see that it's a bunch of it's a regular sort of system prompt instruction. It sets up this schema and the schema says go through and find anywhere if we're used like a rag approach which basically we load these privacy policies into a vector database we use a context query to find these relevant sections of the document. We use these prompts to, to answer these questions based on those documents. And so this one. It's complicated it's what we're trying to figure out is like an array of submission methods. And so what we're turning is like basically a list, and those lists are like what is the method what is the type of method how can it be used so for example the action type would be phone, and here's the phone number. And here is, you know, let's see. Here is the privacy infrastructure provider if it's something like one trust or things like that. And all this information helps actually on our back end, when we're setting up companies to do these requests. It's a really good first pass, and I feel like it's, you know, none of this stuff is ever 100% correct it's always comes out somewhere around 90% correct. But it's a really good first pass at doing this research and so we go in and we pull out this information and we use this specific query to for our researchers to go and start to basically verify this information and set up how we're going to do these requests. So walking through one more. Just sort of simple one. Let's see. So, this one here is very similar but what we're asking the AI to do is we go as part of this we pull down a Wikipedia extract basically the first paragraph of a Wikipedia article. And then we ask it to answer a few different questions based off this schema from that Wikipedia article here. And one of the things I just want to highlight is something we found interesting for both validating and interesting. I'm not sure if it actually improves the results or not, but that we ask this thing to provide its reasoning along the way. And when you're going through and validating these things and we do all of these things that we put in the app we do there's human on the loop or validating the outputs. So having this reasoning here is you can sort of pick up really easily where an LLM is failing just by seeing how wrong its reasoning is for things. So, and that's an interesting thing to add to this. So let me share the app. Real quick. Again. Okay. So we'll just go to a little more detail about what these different things look like and what some of the output is. So these are the documents that we add in when we pull them back we count the tokens in the documents. And we try to parse out the date that the document was glass updated it's effective date. And then the overview. So we ask a bunch of questions on that. We try to determine whether or not the company is the data broker given downloads basically a bunch of CSV downloads from the California and Vermont data broker registries. This is an output of what those data categories are. And you can see, this is all the data that the company collects or what they said they collect in privacy policy. And they just score me and I'm the right like you go it seems to go from one to 10 or something. That's actually, because we're initially using models with small context windows those are the IDs of the source text, which is down here, where that is pulled out. And so that was just the way that we were going through and instead of it instead of it repeating the source text for each one of these things and filling up that context window. We mapped it to a set of IDs and then had it generate just one version of each source text. Got it and what you're saying so much in every sentence here but let me go back one more. Did I hear you say correctly that you pull the basically structured data every company that's listed in the California registry of data brokers. And then some point in here you're doing like a match to see whether the company that you're looking at the privacy policy of is also listed in the data broker registry and you're doing that through an LLM like through a prompt. Yeah, and it's like, whether or not to do that, that's easily something that could be done with like a fuzzy search. Also, you know, it's like super easy. And so the idea is that we sort of we have an LLM do it. And honestly, I think a fuzzy search is probably better at doing that. But I was curious to see how good an LLM would do it. So we added that in. What's the answer like how was the performance when you spot checked it. There are some things where it gets wrong. Like, you know, it will, it will say that a company is like zoom what we're on right now. There is a company called I think zoom info and the data brokers registry and when zoom.us is put into it. It goes, Oh, zoom info. This must be a data broker. So, so yeah, I would say, you know, like all stuff 90 to 90% great.  Very informative. Thank you. And sorry, I didn't mean to break your flow, but I just want to make sure we're, we're hitting some of the top of the waves as you, as you color. Yeah, so this is that first prompt that I showed you this is the output of that. Wow. So you can see the different data rights, we say, you know, do they explicitly say who can, who can use it, etc, etc. We'll try to see if there's some more interesting ones third party sharing and selling. You know, we're basically asking, do they say that they sell data to say that they share it, do they say that they sell it specifically as defined by the CCPA, who do they share and sell it to and with. Who do they buy data from who do they collect data from. This is relevant to see what they even have a user's data for us. This blurb we just sort of fine tuned the model that generates these really little burbs about a company given a Wikipedia extract. And down at the bottom, these are the sort of final output sentences that we use in the app. At the end so and again, we started to we this summer went through a process of figuring out a way to validate this information so created a framework to validate all of these questions and sub questions in here and created this is a pretty hacky interface but this is what we would use to validate things. So we have a set of validation instructors, instructions that were generated by this really brilliant fellow names one hey that was with us this summer. And we would go through and manually validate these things based on this rubric on the right and basically market as good wrong or could be better. So that we have in our database we have like the value which is the answer and then we have the edited value for each one of these. And so we would, if they're wrong, we fix them, and then with the fixed answers, hopefully somewhere down the road we could possibly use those as a way to, you know, find you in a better model to do these things or things like that in the future so the validation process is not just about validation but also creating a really good gold data set. So with that, and please let me know if anyone has questions. Standing. Dr. Lance Elliott, you're on active alert that I may call upon you just so you know, I see you're in the audience. One quick thing I just have to say is, this is great work thank you for taking the time to present it. Everybody can see how you're coming at this. There's it's really a time of true kind of innovation and creativity as people are seeing this these powerful tools, and trying to think how could this apply to something I'm actually trying to do. And you've really like taken it all the way. And it's it's well documented it's clear. One base question I have is, is, have you been able to get your head around, like the, for the average time it would take to process these policies manually at number one is, is it comparable time or I would imagine it would be significantly comparable time for similar information and number two, what about the quality so how much, you know, when, like I have been in the position of looking at terms and conditions and policies, hundreds and thousands at a time for various projects. And I know my quality, it starts to dip a little bit after the 10th and 50th and 1000 one of those but nonetheless like, how, how is the quality, compared to people through time and quality, l l m versus people for this task what's your assessment. The time this is way faster. I think that we calculated it was like 1500% faster or something like that so the first 80 companies we put into the app took months of researching into privacy policies to do it and in that process we had a brilliant researcher who created this like basically structure and protocol for doing this research and that really helped this process so it's like that protocol was used as ways that we, you know, talk to the l l m and get it to go and answer these things. Right.  So, regardless, it's way faster to do it quality wise, you know, like I said, there's like, it's still in that 90 to 95% range, but things are wrong and you have to go and find them and fix them but it's still way faster. And to your point as a, yeah, it's like our own researcher, you know, and even when we were going in creating like our own, we went through at the very end of this and the validation process and hand coded like 20 of these companies to use as like a gut check.  It's like, at least one thing in the world to do. So, yeah, hope that answers it. It does. Thank you so much. And I see we've got one more. Some simple, we've got a few questions in chat and through DMs here, but we can get to at least one more before we need to go to the next segment, which is from Nolan. And the question is, do you think generative AI based legal classification extraction will make the current generation of legal due diligence document discovery. AI modes obsolete. Yeah, I don't know that might be a question more for you like I, my sense of this project is this is a directed project and goal this project was was to produce the content we needed to produce in order to add infinite more companies into this app. So, I, my general sense of working with LMS is there a great at transforming information. So, I'll leave it at that and maybe if you have answers to that. I have one perspective, which is on your obsolete's a big word. And so, no, not, but I am what I'm already seeing actually what and I was just involved with a friend of mine who is general counsel of a company that was being acquired and there's a lot of due diligence here is that the current tool set still does some things that are non overlapping, especially when you're going through financials and, you know, due diligence and document discovery is kind of a big umbrella. There's a lot of different things you're doing based on the facet of the task, some of which don't overlap with LMS some some stuff LMS seem to be comparable or better at and really interestingly there's a whole class of things that these LMS can do that we could never do before at all. So I imagine we're going to see a period of sort of co evolution, will it be a, you know, like humans existed with, you know, like, with neanderthals for a long time simultaneously. So I think we're going to see like a kind of a period of dual evolution. And, and there's going to be some some things that other technologies continue to do that, at least long as we have the transformer model and the sort of probabilistic approach with LMS that that that that the that this technology just doesn't do so. But overall though, I just tend to see a lot of promise here. It just keeps over delivering on things that I think are interesting and good and very, very relevantly I just can't stress enough what Dan said, several times which is, it is not perfect. It has like some flaws, some limits, some biases and so don't worry y'all there's a continuous role for humans that have judgment and can look over the outputs and be the final say. Oh, it looks like Lance it so can I just want to do one quick depth deputization then we'll move to the next segment. We're actually joined by another colleague Lance Elliot who is a real expert on the advent of this technology in the legal field, and Lance I just want to give you a chance to come off mute to see if you had any answer as well since you're in the audience I want to like deputize you for a hot second to see if you have any perspectives on that question. Sure, can you hear me. You sound good. Oh great. So, first of all I apologize I was on another zoom call that finally ended. Secondly, it sounded like at the tail end there there was a bit of a discussion maybe and stop me right away if I'm off target. About the idea that large language models, machine learning deep learning is all focused on what in the I field they call sub symbolic approaches, which is you look at data. You find patterns and data based on those patterns, you try to make predictions. And that's really what generative AI is all about. Now, you might also know, there was an earlier era of the AI field about expert systems knowledge based systems where you explicitly wrote out rules that you would then have the computer system try to execute to perform those rules. It almost sound like they're at the tail end and I might be off target, that there was a bit of a discussion about where we headed with generative AI LLMs. What's that future look like. And if that is kind of the question, then my answer is, it's what many refer to as neuro symbolic AI. The idea is that you combine both the sub symbolic pattern matching with the kind of rules based approach. Together in combination. And even Sam Altman, who is the CEO of open AI, he has come out and said LLMs and machine learning the way that we conceive of it today can only go so far. Something else has to break through because today so far, we've gotten away with just getting larger and larger models and greater and greater computational processing to get where we are now. And the belief is yes that will continue to take us forward. But if we really want to make it like all the way to an AGI artificial general intelligence kind of capacity that we have to come up with something else and some believe that that something else is this neuro symbolic AI. So does I'll just pause there I don't know I may be off target from what the group was talking about but maybe that nonetheless is of interest to the group. That was of interest. Thank you and that and that was not overlapping with what I had said as well so I sort of said the assume we have the same kind or more powerful the same kind of LLMs. It won't be a complete opposite leading of the prior tech for due diligence and discovery. But that's a really good point which is one thing we can count on is change we're not going to have. I don't think this is the ceiling and there will not be any further evolution of the of the technology and in addition to the symbolic generative AI mash up I think people like in Microsoft and other places are looking at planning. And like just other modalities that they can start to mush together with all of which could could change the whole playing field again so thank you Lance for letting me tap you. We've got one more comment from Brian and then Kara and Rich you're on deck so you know kind of warm up your engines and we'll go to you as soon as Brian has made his contribution. I'll keep this short but to what does was saying, you know, obsolete is a huge, huge word and a huge concept and I don't think traditional due diligence will become obsolete. You know it's always been sort of in this pattern of continually evolving with the technology that is used to evaluate it with. So I think that trend will continue. You know, an example that I think demonstrates how this could happen is with some of these note taking apps that use generative AI. So, I know a lot of people here probably use them where the app makes a recording and a transcript and let you query the notes from what was said in the call. So one of the things that I've seen just in very personally that I've started doing in these calls is I'll start signaling like so the action items from this call or this. And so you start learning to use the tool in a way that allows you to leverage, you know, large language models generative AI to the fullest extent that it can be leveraged and I think that's sort of the area where due diligence will transform. I think people are going to find the sweet spots of using LLMs, generative AI, all of these cool new applications in ways that reduce the total amount of time that it takes that increases the effectiveness of the due diligence process that drives the operational expenses of that process down and it allows us to get through due diligence quicker with less of a headache, I think. So that was all I had. Awesome. Thank you. And incidentally, I'm saying all these names I should have mentioned hi I'm Daza Greenwood. I'm a I head up law dot MIT dot edu of which this idea flow is one little thread and Brian who just spoke Brian Wilson is editor in editor in chief of the MIT computational law report which is kind of the flagship publication and public facing facet of law dot MIT dot edu so there you go and thanks Brian. Now, let's move forward to the next course in this meal. We have with us co founders of a new initiative called describe dot AI, who are doing some, I think really impressive work, applying generative AI to case law. And so with that, Kara and Dan, I'm just going to ask if you could introduce yourselves and and tell us more about how you're applying generative AI to case law and you know for what does describe do and in particular what what can we do now that we could never do before. And then we'll go right to discussion. Excellent. Thank you Daza and thank you so much for having us here today we're very excited to be joining you from Newton, which is Newton Massachusetts which is right outside of Boston so certainly close to MIT. I'm the person she her and I am the co founder of describe dot AI as you mentioned and I am not the technologist. That would be rich my husband and I am a marketing and biz dev person so I think for this group rich will have the more interesting things to say so I will hand it off to him. Thank you. I am a, I come from a traditional computer science and software engineering background at in the corporate world so I was able to bring some of that experience and architecture and everything else, computer architecture over to this field so I think the best way to talk about this is, I could show you a quick demo, because it's the type of product where, once you see it. That's, that's really the magic of it. So let me do this. Okay, you got you got it. So, basically what we've done is you could search for any type of legal concept or term or case facts whatever you want in here. So I'll just do a quick one, you could see it. Search for I was just making things up landlord invasion. So I'll just do like child was injured at a playground.  You could search for specific legal terms that will are relevant to what you want to look for and if you're not a lawyer you could search for something simple like this and find case cases that might be relevant to you so that you could learn more about the law. So you look through it, you could narrow by state here. But the way it works is, we've read in all of the original legal opinions and, and kind of split them up and summarized them and made them searchable. So for example, here's a case. This opinion describes a case involving a child who was injured while playing on playground good. Well, not good for the kid, but a good for our search. So basically, if I click through, we will work a little backwards here. Here's the original opinion so you could always refer back to the original opinion. These opinions are super long as you all know some of them. There's an opinion I ran into that initially crashed the system that was like 977 pages. So, yeah, so there's all kinds of opinions out there. But if you. One is broken the opinions up, and then summarize them for, and it makes the search more precise. For example, in this you could see in this case Clyde right brought in negligence, then the second part of the opinion describes a child named Jordan was injured while hanging on the rollergate. This is another part. And you could see this fourth one down is actually the one that it found as a match. It describes a case involving a child who was injured while playing on a playground. So it, it, by breaking the opinions up. I mean, it was done out of necessity because the language models at the beginning of the year when we started in coding could only handle small context windows. And now that they're able to be bigger I still think this was the best approach for what we're doing, because the search engine searches and every single one of these, and then on the results. It shows you the one that match the best. So, that's one part of it, and we're working on other things to, for example, by click on this one. We're doing something like what they mentioned that consumer reports. I mean, on a different type of scale, but we're actually deep reading into the opinions and having it tell us like here's the parties. Here's the introduction background procedural history issues presented analysis, holding and conclusion, so that you could go right into here and if you're an attorney or someone else interested in the law. That's exactly what this opinion was, and then down below you still get the summaries and the original. But this is a good way to just quickly glance, and this was all I generated using some as as Dan could attest to it's a big iterative process using the figuring out these prompts to get this stuff out of there. We're going through and we're making these types of precise summaries for every single opinion, they're not generated on the fly it's like, as you when when you use a search engine you'll see everything's instant, because the cases don't change so there's no reason to. And you can't summarize them in advance, so we just summarize them keep them in the database match them up, and everything's super fast you don't have to wait like 20 seconds for it to come back and summarize. So, there you go that's that's the quick demo of it. Amazing. A couple of things. First of all, thank you for for taking the time to walk us through I agree is nothing better than a demo. Could you speak a little bit. So, part of the reason I wanted to bring you forward to show this so we could people could know about and we could discuss it was to demonstrate what's different about generative AI when it comes to searching for case law. One of the things it's that to me is kind of profoundly different and almost mind blowing and its potential is that we're not searching for words or phrases anymore, but we can in a sense in the kind of high dimensional vector space that this data has been atomized into and through the models. We can we can see, as it were, the concepts or the ideas behind the words. And that's what happens with the kind of cosine similarity and the semantic searches that are that are going on. And if I understand correctly that match score like in this case for real Linda unified school district versus superior court. We've got a zero dot eight nine to one which is pretty high. When I when I do these things. Um, could you speak on a little more detail about what is that little score. And what does it mean is isn't the same as like, we have three matches. Like it's not like yes or no. It's a score that relates to relevance of a of a new type, like what is the score what and what does it mean. Sure. So, on the tape, the people on the call may have heard of vector databases are seen it in a article or something that they've read on the, the way they work is is we don't have to get too much into it but what as I said they store these numbers to represent concepts and language and everything. And so, if I were to have store in that database something like that dog was sleeping in his doghouse. And then I typed in the dog was sleeping in his doghouse that would give 100% match. If I said, the cat was sleeping on the cat bed. That's like a similar it's it knows it's an animal. It knows it's sleeping.  In particular case, we're going to have a hard time getting over, like a 9089 is actually quite high, because we're searching these entire paragraphs and unless you get this exact paragraph almost exactly. You're not going to get like 100% match, but it finds this relevance because it talks about it. There's a child. There's a playgrounds there's an injury. There's, you know, they were that the more you put in or the less you put in the more it can match up to here. It, it, it will make that score a little higher. And now in this particular case when I click through each one of these, like this first one doesn't mention a playground at all. I don't think just glancing at it. So this probably would be a lower score. This, this looks like a slide onto concrete that might be a middle score than this one. This is the actual one of found. So it, the thing grades each one. And what I've done is just return the highest matching one without repeating the, you know, you don't want to repeat like you don't want to send back like five, from the same space because so it just pulls out the highest one that matches based on that vector matching score. So answer your question. Amazing. And just just to drill down a little bit more on this. Sure. Let's say you have the word child, we see the word child here so in some ways it's not a great example of what I was trying to get at but like it let's just say like you're a, you're just some person you don't really know the cases or maybe that the word child has like resonance and is noted in specific statutes and regs and case laws and so you're saying, maybe other words to talk about like young people or like my children, or my, or minors or, or maybe different facet of it like so people that have extra rights under the law who blah blah blah. It could start to sort of it sort of see these years of cases and can find the underlying meaning, and it's assessing relevance based on that. FYI everybody. This is huge. Like we could never see the shape of the law before we could only do very brittle, literal, like word and phrase searches and, you know, like rejects kind of stuff. This is pretty important because it might tell us for the first time, what is the law, anyway, like, just what are what are what is the what, what are the underlying rules, and how could they apply to my unique circumstances even if I'm using different words, or the facts and circumstances are playing out a little differently. It's down to relevance and relevance is, is, you know, is a deep deep rabbit hole, and the kind of semantic inference engines that are available through this technology, can it help us begin to plum the depths of these of these quintessential concepts for law. Dazza, do you want to on our, on our pre call you mentioned the thing about Wyoming do you want to mention that where you were using it when you found that and that's partly why why you're here today. So I was, I was helping the state of Wyoming that's a good anecdote, who was looking at some legislation to advance their policy agenda to give individuals and companies more rights over their digital information. In a sense that this is kind of, you know, compatible with with the consumer reports thematic, you know, initiative we just heard about giving people control over their digital information. But one of the things that's different is consumer reports was based on California Consumer Privacy Act. What they were trying to do in Wyoming was just look more generally at what is the relationship between people and our data. And they were following a state law framework of property they're saying among other things it's also property. It's a kind of personal property, but it's not physical, it's intangible or digital. You know, when we create it or when we perfect rights to it, it's still a type of property and we should be able to have at least basic property rights and property rights are a state law, very much an area of state law. So, this isn't a rather unusual, I think logically it applies perfectly. And they were looking to clarify some of that intellectual property is kind of confused that a little bit. So now we have this other statutory framework of, you know, that names things certain ways, but that's like a little off to the side from just basic property that we've had for millennia as a concept and law. And so I wanted to see cases that stood for that principle. And so I put in. So I asked some very basic questions about, you know, what are like legal, I'd have to go back and find the process to the worst the effect of like, what are legal principles that that stand for the, the rule that people have, you know, property rights over over their digital situation. And when I asked it that way it pulled up a bunch of cases that I'd never seen before, because I guess I hadn't been searching the words correctly but there was this whole rich thread in tax law about intangible property. There was a bunch of stuff about kind of interest that people have to like their, to their kind of like customer lists and to, you know, kind of database information that use very different words. There was a lot of information about like what people that are creating digital art, but it wasn't copyright. It was it was. So anyway, it was really interesting that it got underneath. And a she if you raise your hand if you want to say something I think I saw your finger. But anyway, so I was really impressed that I was able to ask a question one way and it identified a lot of case law that stood for the idea. And I've never been able to do that before. I kind of had to just, all the owners was on me to think of all the right words to use or I had to hit like pay dirt on a law review article of somebody that spent years, you know, tracking down every case. But even then we miss cases like how do we know that there's a case out there that would be relevant to what we're trying to do if we don't know what the words of that case are and if somebody hasn't gone through and tax on a fight in a way that relates to what we're searching for. All of this is very brittle it's very superficial, and it is it doesn't let us know what the law is this technology at least has the prospect to allow us to unleash and to reveal what the law is and to connect it to you know the questions that we have about how it would apply to given facts and circumstances so. So that's my sort of anecdote about how I started using describe and how I was like kind of blown away, basically, and it really was helpful to the legislature I sent them the stuff that we turned out and it helped inform their ideas of how to then frame things in in some, some statutory reforms that they've been considering so it's very useful it's very mind blowing is very practical. That's my anecdote. Cool. Yeah and and along those lines I know we were wrapping up in a minute. The way like people out there if you're a pro say person or just you're involved in the law for some reason you don't know all those words either so you could just type in what happened to you, and then suddenly you'll like if you type talk about your mother somebody coercer to change the law then suddenly you'll learn like okay that's called that's undue influence so why don't I, you know this thing will tell you it's undue influence and so you could know what to research. Um, so if people wanted to play with this so I want so we are out of time we had a little more time. I would actually suggest that everybody go to describe.ai. And try some searches and like try it for yourself. See how this plays out. And, and then I would ask you to share your results so we can talk about them so I can't do the last part but I can do the first part, which is go and check it out. To the extent that you're on LinkedIn go to the MIT computational law report post about today and make comments and you know let's continue the conversation in that way. Thank you rich thank you Cara everybody we have some really interesting things coming up. Rachel if you could go on mute. We've got the 2024 MIT computational law course is coming back at you it's free and open to all in January. And we're going to go much deeper into many of these topics and give you all an opportunity to use some tools, and to get even more deeply engaged. And, and thank you all for joining us for the sort of resurgence of idea flow. I know we've taken a long hiatus, but we're back baby. So thank you all, and we look forward to seeing you next time. Bye bye. Thank you.


we've just arrived now in a deep city this is a design center in the middle of the desert kind of somewhere between the design centers there's also some shops here and it's literally in the middle of the desert as you leave Jerusalem on the way to the Dead Sea in fact if I turn around this way you can just kind of see some of the sand there in the background only been open since about last august so it's still relatively new so let's see what's on the so these are all the stores here you can see there's a hotel on floor two for one this is all in Hebrew and then so yeah this place is relatively new but it feels already quite established really nice kind of an outdoor mall kind of a feel and elevators here going up to the second floor and they've got this overpass here as well so I'm recording this on a Friday just two hours before Shabbat but this is open for a few more areas here so I guess I'm going to be a new kind of luxury center shopping center for people in the area Malle Adumim has kind of emerged as a commuter town almost people looking to get away from the Jerusalem prices and on the second floor you can see it's still kind of a little bit under development you can probably see me there in the reflection the second is going is more tile stores and things of this nature this is an escalator leading up to the third floor but only going downwards at the moment recording this on a Friday as I mentioned so we've just actually had the first significant rainfall I would say of the winter here it's been fairly dry so far it's very nice it's a kind of outdoor theme malls they've got plenty of places to sit here and they've got live entertainment as you can hear behind me a little band on a stage obviously this place is really intended for design that's what the D stands for in D city but it's kind of got that American shopping center feel they've got a lot of music coming out of outdoor speakers here it smells a bit like a hotel and really kind of a high-class experience if you're into looking for a place to design your home in Israel 35 minutes drive approximately outside of Jerusalem I'm shooting these videos on the Femi Pam 2 which is a pocket gimbal picked it up yesterday and I must say I'm really really liking it so far definitely the settings take a little bit of getting used to over the DJ Osmo pocket to the battery life is longer and yeah so far it's really nice to not have to cry around a full-size gimbal and a camcorder and get some relatively decent quality footage very interesting mixture stores here this is the life height store cleaning supplies watching a guy do a demo here to a bunch of interested customers a little bit further up there is a coffee shop Casa barista and then their next-door neighbor one store up further is RGB real gaming business computer store with nice RGB and other stuff for sale there right to come into the centerpiece of the mall here the food course pizza dot and probably some stuff is closer given the time there is a the pop-up station looks like some kind of a garden store this is the indoor components are just putting on my mask so I'm going to say I'm a little bit muffled jewelry store at a high-end place fasher which is a well-known cheese shop in the shokin Jerusalem there's a sit-down restaurant here tv wall we're going to come now to a huge this is actually indoors you're not going to you're not going to believe this this is a artificial sky and to say this place is expansive is an understatement this is actually kind of massive this is all indoor so even if it's a rainy day like today you can sit here and enjoy me toast they've got a few brands so it's kind of incredible to be here does not feel like you're just a little bit outside Malia do meme in Jerusalem in the middle of the desert they've got another live performer here and you can just get a feel for the scale this place looks like the second floor isn't fully occupied yet but they're doing a very lively trade here if you can see there's literally dozens of people there sitting out and probably the weekend is their busiest day I'd imagine Milano optic is crazy it's just like going from night to day in a second here chocolate shop and Swarovski so really all the mixture of Israeli brands and big international stores represented here so it's actually modeled after Piazza San Marco in Venice and that's reflected in the architecture and these outdoor lights or these mimicked outdoor lights under the artificial sky I think the most interesting shop I've seen here so far is this place called Arak a box they've taken Arak which is a aniseed based spirit very popular in the Middle East and put together a few really interesting spirits with different flavors you can see here they're all fruit flavors and they have a very very wide selection one is a passion flower which is passion flower and they've got lemon they've got red grapefruit flavor and they've got almond Arak as well which is delicious so these guys are 16 alcohol by volume and they've got a few different branches in Israel so I've just picked up one here and they've got this little stall here in K city of course you have to be 18 or over to buy it but 70 shackles they're selling these bottles for a year so quite a quite a discovery I'm looking forward to trying this at home