Hello, I want to welcome you all here. My name is Seth Mnookin. I'm the associate director of the communications forum This is the final forum of the semester And we are definitely going out with a bang. I'm incredibly incredibly excited about tonight And I'm going to introduce our speakers and then my job for the evening will be done Nick Modfort immediately to my left is going to be the moderator of tonight's discussion Nick is a professor here in comparative media studies and writing and is scarily prolific I feel like every time I get up to go to the bathroom. I hear about another book that he has just published His most recent book is And I go to the bathroom a lot His most recent book is shebang And not spelled just like it sounds and that contains both computer programs and poems He's written a number of books in collaboration with other authors several of which are on sale outside to Nick's left is Fox Harrell who is also here at MIT is actually not physically here this year So I was very pleased that we were able to rope him in to flying across the country and joining us here His work explores the relationship between imaginative cognition and computation And his book phantasmal media and approach to imagination computation and expression was published last year And that will also be on sale outside And then we are very very excited to welcome Lev Manovich Here as well. Lev is a professor of computer science at the graduate program at the Graduate Center at CUNY Is also the author of a number of books including software takes command And the language of new media which was described as the most suggestive and broad-ranging media history since Marshall McLuhan He is the founder of director of the software studies initiative a group that works on the analysis and visualization of big cultural data And please join me in welcoming him and our other speakers so I'm really glad to Be able to join Fox and Lev and to welcome them to present some about their current work Lev actually Has been a collaborator of sorts since for more than a decade because he wrote an introduction to the new media reader which was a very valuable for situating that book and describing the connection of this historical work to the way that people think about new media theoretically and Fox is a current collaborator We might have occasion to talk about the project that we're working on slant, which is a story generation system that involves other researchers in Mexico and And people here at MIT So without further ado, let me let Lev start us off And and we'll then go through after he and Fox present some some questions I think that bring together a lot of the work that we do from our different perspectives Great Okay So I actually Agreed to collaborate with Nick on this panel and she said that with me and folk should talk for about 15 minutes Even we can actually have real discussion So it's not gonna be series of mini lectures So please do kind of after maybe 17 maybe in 90 minutes maximum kind of please break me So Do I think to situate kind of my work in relation to this panel about creative possibilities of computing To a certain extent the first part of my life was maybe more directly relevant to that Because I grew up in Russia in Moscow. I was educated in the art architecture and programming Captain York in a 33 years ago Went to NYU Did like film video it then started to work with 3d computer graphics Exactly 30 years ago and actually was accepted as a graduate student into a media lab To work with Miriam Cooper and completely stupidly didn't go But every time I come like I feel like it's my alma mater anyway because I end up doing exactly the same stuff I think as if I would go if I would went So in the 90s and 2000s I was kind of helping the emerging fields of digital art by writing of his books and of his theory But I was also creating my own art projects Which would generate things with computers as a project such as soft cinema We use a database of hundreds of video clips in the road the software which would automatically generate narrative films out of Out of them and then I think once you have a computation and culture have become almost synonymous Right where now we have billions of photographers Constant that mother for pictures on Instagram you know billions of videographers putting her stuff on YouTube billions of poets writing beautiful You know tweets right? Everybody become digital artists. So how do you do something unique in this environment? I realized in 2005 a big data was coming and I said well Maybe I can use computers now in a different role not only as a kind of generative Inexpressive device but as an analytical machine right as a machine Which will help me to study cultural patterns and cultural past by visualizing and analyzing massive amounts of images and video but of course I Still remain to be I think somebody who committed to the creative and expressive part because Visualizations which we create in my lab. They're not only analytical, right? They're not only trying to reveal you know to use right with small most kind of common Term somewhat misleading. We're not only trying to reveal patterns of the data We're actually trying to create an expressive representations out of data, right? So let's say if it was hundred years ago or nine years ago Perhaps I would be filmmaker or I'll be doing photo montages in trying to figure out how to present a modern city Using collections of pieces from newspapers or photographs or maybe I'll be like writing letter to deliver to hoping he will accept me Is his cameraman so to try to represent the city out of thousands of shots Well today I can try to represent the city or other places or our phenomena Basically, you know a global visual culture by creating montages or our representations out of millions and hopefully billions Individual expressions so ideas not just to reveal patterns not just to do something productive But to actually do something expressive represent our time Through millions of contributions of individuals on social media So I will very very briefly kind of run through a few projects. We've done in the last 16 months Each project has its own website all visualizations are downloadable and highly dilution from Flickr. So if you're interested, please visit them So the first project which we've done with social media was called photo trails So it was our first attempt just to see what we can figure out with millions of Instagram photos using techniques which are Not something which can be only described in computer science papers and published in some journal But that's just something which I can teach students in a couple of hours. So techniques are on purpose are very simple So we downloaded 2.3 million Instagram photos from 13 global cities and then we created kind of visualizations such as this one Where each visualization takes a sample about 50,000 photos and the photos are simply organized By a couple of parameters So it's not hard to guess that in this case the brightness controls the position of photo in relation to the center and then the perimeter or the angle of the angle of photos controlled by its average hue Let me open this in Photoshop. So we don't have to wait for a computer to load with these guys So in all this project, I mean one of the questions we wanted to ask is to say so our is is a Phenomenon such as Instagram, right? Is it actually allows us to look at the medium with itself a message So is a social media can be thought of as a kind of certain mirror certain filter certain way to understand what is she happening in social reality Outware was it only going to tell us about Instagram itself? So this diagram medium a message and I think HOS for project is trying to kind of try to get it at this answer But I think it's very very hard to answer in effect. Maybe it's even impossible and in fact unreasonable to try to separate the different functions Right. So if you look with photos, you can say well, you know, if you simply look at them through this very crude a very simplistic instrument Which we first instrument to construct yet of clustering photos by we're kind of color and brightness all the cities look very similar So here's Tokyo This is San Francisco, this is Bangkok, right? And then if in fact if I look at the user filters, we found something even more scary, right? So this is the filters which were available on my Instagram in the beginning of 2012 and we downloaded the data switch Line basically shows your frequencies of this filter used In six different cities and you can see it's almost exactly the same right only a couple of filters We have a little difference So this project using a particular method method of data collection in particular tools Right because I think just as a confusing 20th century. It's almost instruments. I'm going to influence what you're going to find It's using a particular apparatus which we constructed, you know The answer which we get look, you know people are taking very similar photos in terms of contrast brightness and our Visual characteristics we use the filters in the same way. So Instagram is seems to impose some kind of universal global visual culture So then we did the second project where we said let's actually now as opposed to comparing everything to everything Let's only compare apples to apples. Let's take a particular Instagram subject All right and only compare photos of particular count. So we did the second project called selfie city Where we downloaded? We kind of designed a data set using a combination of computer vision and mechanical Turk techniques of 3200 selfies from six five cities around the world so 640 selfies from each city and we compared this Selfies on a variety of characteristics using face analysis And I will directly go to the interactive part of a website we actually Yourself can go and navigate this data set and look at you look at this characteristics and compare them all right so idea was also Not just to do what everybody else does which is presents with data is bar charts pie charts of line charts But create interfaces which allow people to actually navigate and interact with you know huge amounts of visual media So this is a whole data set. I mean we couldn't show all of it because of limitations of like a web speed So I think you have to Go like this, but really quickly you can say everything And then what you have here is metadata which comes from the data right so here's we can filter by different cities okay, and Notice when you filter by city how our variables get adjusted. So for example, here's the average age for this photos So if I select for example, select some power, right? You can see where it was a little bit more younger people and if I go to Moscow, maybe actually with all the people So these things get adjusted And then here we have gender So we didn't we will have 40 gender labels unfortunately like Facebook and Instagram So actually Facebook is why maybe more progressive but we do have something where people couldn't agree because we just use mechanical Turk and Where it's like undefined But doesn't mean it's a fraud gender right basically we just couldn't agree And then we have is our variables which are extracted by the computer right including various Emotion variables and you can say well, you know, what about people who like really leave a computer things are very happy Okay, what about people who computer things are really unhappy So it's actually not as bad as you would think and you can kind of play these games You can say how about New York? Lots of probably unhappy people. Okay, this is this way unhappy people Maybe angry and so on and so forth so you can how about come people almost nobody come in New York, right? So anyway, yeah, so you can see I'm a ethnic it kind of works So what we actually was basically what we found this project right is what we actually did with you know Very kind of classical Statistical social science in type analysis. We actually found that in fact were are very significant differences But they will this is the differences in reality Or they will be or maybe this is a difference in how people present with some Instagram It's kind of hard to say right, but we did find interesting things for example, you know the big proportion or the big difference in proportion between male and female users You know different proportion for example an average age Smiles and so on and so forth So then after that I said, okay, so that's so what we've done here is we took this whole kind of cloud, right of Billions of Instagram photos and we focused on particular subject Which itself is what if I now do a different kind of zoom a different microscopic view what we focus on particular day Particle time and space so we did the first next project, which is called the exception of every day Where it's 144 hours in Kiev in February, you know during the kind of so-called my done revolution So we've downloaded all Instagram photos will be the kind of certain square Which happens to be focused on the my done square So basically, you know, here's all events happening in the city, you know somebody drinking milk You know going to party taking selfies, you know getting drunk and then so just happens with revolution to place right in the middle of it so this is kind of a map and We did this project because what we found is where typically this kind of big data social research or what for example mass media does is We said okay, we're good to report about Something which is happening in point x some revolution up he will it was definitely we're all aware about many important events Like what's happening the last few years You know and we only got to focus what's happening in this point demonstration But what else is happening in the city, right? So we interested is Almost do a consultation experiment to say we're going to limit the city to a square the five by five kilometers And it so happens that in the center of the square in this events to place And they're going to have a time frame of seven days and we're going to look at every single star graph photo Regardless of whether it is relevant to revolution or not Just to see what was actually happening in the city to what extent people participated and how people were reacting to this events So that's why the project is called about the everyday and exceptional and we interested in how every day exceptional can spill into each other All right Again, you know, I'm not don't have time to go in it But this is for example all 14,000 Instagram photos which we shared in the city Organized by date and time and if you look at the zoom level you can't you don't even know what anything took place Right, you really have to zoom and look at particular tags to find out and the fact was a very high rate of participation So for example, this is the photos which were shared on the square So now you actually see the revolution beginning to become visible And then you know it goes on and on and on and eventually you're looking for example at the tags And we actually look at what eggs you can actually see that, you know, this is every day a typical Instagram day We're saying things you'll find in millions of places Right with geographic location and this typical Instagram love follow me which don't mean love doesn't mean love on Instagram It's just like a way to spread images around Yeah, and then you know, there's like nothing really specific about the key if and then the more revolution starts Obvious tags like your my done my done revolution go to the top on the third day the local completely replace the global In the moment revolution succeeded now this kind of generic Instagram language starts to go in, right? But this is the only top tags out of six thousand unique tags and fifty thousand tags Right, so does it actually mean people are really participating in revolution or we ignore it? Hard to say right because nobody has done this analysis. So finally Very briefly maybe to show you The last one so last one it hasn't been published yet It's a great premiere next week is a commission from New York Public Library So it's going to open in your public library next Friday to be on for nine months So we'll have a website, but we will not be able to actually show you a whole interactive version of the website because not fast enough So here we try to do yet another Kind of microscopic Zoom right so we said let's take the whole New York City Let's take a Broadway, which is one of the longest streets which goes through all of New York 13 miles 21 kilometers So we defined a kind of spine which goes from a Broadway Exactly hundred meters wide Imagining take a body and you only leave you know the kind of spine and then the limited over data to inside the spine so we're using a 660,000 Instagram photos which or every single photo which we shared along Broadway the last six months every single tweeted image Millions of four square check-ins over New York City taxi data from last year and actually turns out of the social media data Is very reliable because produced by companies. What's unreliable is government data So economic data like all the income stuff from your Bureau of Census. It's complete It's really bad and I think all the cannabis know that all the people who write about social equality know that but nobody like Even of course, you have right the guy who says let's look at iris data He becomes famous right because you know, he writes a book called capital. So it's actually amazing, right? So it actually turns out with microscopic data about people from last six years. It's really good What we actually don't know about that income and wealth, you know and demographics actually this is really bad data But again just to finish my remarks I mean, we will have a canal is this in Creelations and all this other good 30th century stuff if you believe in Creelations, you know but ultimately the goal is to create an expressive a kind of Information experience where people can themselves navigate for this wealth of information images and explore the patterns but also Explore this a particular type of city portrait, right? City portrait which is made from millions of data points, but not only data points, but also millions of images So I'll just kind of play this this is not the final version. We're still working with you can see we're exchanging right always discussions on base camp like normal designers And by the way, I just want to say I know you guys all about paid MIT faculty and students in case somebody wants to make a quick buck So we're using 60,000 Instagram photos and installation Installation supposed to be ready on Tuesday because we have like a journalist and donors preview and just today get email from curators I can show you a gmail which says left we started looking installations and we find lots of like kind of sexist and Kind of really bad images can you remove them? With me now on Tuesday 60,000 I said yes my career behind I promise how to do it So now I'm like the meaning to my raid we have time this weekend kind of like go for 60,000 images And of course if you remove it all then you know when every part of New York when New York is going to be looking more sanitized Right, so you'll get this Bloomberg plus version of New York So that's of course very interesting questions you get the social media right if you remove it It's going to be like yes, yes, yes in reality. It's actually very dirty, you know, especially in some parts of Broadway Anyway, so I'll just play this for you and I think that's way under that's when So when you represent you know, so you remember standout night of St. Shiraita who wrote a novel should be a mirror which stands on a dirty on a dirty road Well, if you now suddenly want to show this mirror in a new public library or in my team museum or momma how dirty can it be, right? So you get this interesting censorship. Anyway, so basically it's a little bit of interaction and again So idea was to try to Not just to create you know, maybe more traditional idea of city portrait made from data, but city portrait made from images So you have this kind of points where which represent averages of taxi Instagram income But more importantly interacting with image layer. So the top layers is we hack into Google Street view So that's all the images from from Broadway for Google Street view left right and up Very interesting to look at the amount of sky you can get more skies You kind of go past Columbia even bottom you have Instagram and you can basically navigate and zoom in out a different scale Thank you. It's an introduction Okay, so as you know, I'm Fox Merrill I'm associate professor here in comparative media studies also in the computer science and AI lab It was also mentioned I'm away for the year So this year I'm a fellow at Stanford Center for Advanced Study in the behavioral sciences But just really pleased to be on this on this esteemed panel So the topic that we were given and collaborated in coming up with is making computing strange And what could be more strange in some parts of the world then combining culture with that with the computing So that's what I'll be talking about is this idea of cultural computing So I'll talk a little bit about what I do here that's imagined I run the imagination computation and expression laboratory It's a research group where we focus on developing systems for creative expression cultural analysis and social empowerment and We've built a whole range of systems I'll just speak about one of them during the later part of the talk today One of the other things that was mentioned is the book phantasmal media in a way You could say the book is about how to take subjective and cultural meanings and and understand how they're how they're Represented and implemented in computing systems in a very technical sense You know to say how do algorithms and data structures implement cultural meanings For today, I'll just give you first this a bit of intuition about why I call the book phantasmal media What is a phantasm? Why are they haunting maybe a why they make computing strange today and particular case of phantasms of Social identity in computing and what we're doing about it in the lab So issues of culture of course are topics that have been Long-studied in the areas cultural studies Sociology and more they're studying these kind of cultural types and cultural messages that are out there the ways that people begin to mitigate them I mean, this is a venerable area of research I mean even thinking back to 1903 with W. E. B. Du Bois and his idea of double consciousness But you might see yourself differently than the way society sees you is the idea Or Irving Goffman who talks about how you might manage the impression that others have of you in everyday life Right, so so these are topics that have been long studied venerable areas But the question for us now is to say now what happens when we get we begin to think about these new kind of social and cultural Configurations now, so this is this classic internet cartoon written about by Lisa Nakamura and others on the internet Nobody knows you're a dog. You know This is a work called becoming dragon in which artists is imagining and reflecting upon the experience of Gender reassignment through the 365 hour Performance in a virtual space as a dragon and so the question is that what can we begin to say about these kind of How those phenomena play out within these environments and of course people begun begun to talk about this So Lisa Nakamura is highly critical of the idea implied in this cartoon Saying that in fact, it's not enough that nobody knows who you really are Rather those those kind of social issues persist in online environments. And for example, she talks about Say people that are not of Asian background playing as Samurai's in geisha playing these stereotypical types within virtual worlds rather than Actually beginning to understand those those types Justine Casel Henry Jenkins David Leonard Anna Everett, you know, many people, you know writing about issues of gender representation ethnic representation in virtual spaces, but another question is I think beyond these is The idea of how do data structures and algorithms implement both these long existing and newly emergent cultural phenomena Yeah, so the idea is that there are new new questions that have emerged that are specific to computational media that we didn't see Within the work of Du Bois, you know Gothman, etc. For example So this this this is what was brought to my attention by one of the students in the lab. So this is a site that scrapes your data. So here's mining your data from a Facebook and putting you into categories. You didn't necessarily know or want to be a member of you So for example, who wants to get fired who's hung over Etc. And this is just based upon a new configuration of data that you actually have out there in the wild So it's a kind of new phenomenon that that has emerged This is another interesting example because you have here on your right in an image Which is a default image of this brown skin character within second life And the other image here is one where somebody has written a new Reflectance model a new light reflectance model So they had actually go in and change the algorithm in order to get what they thought was a more kind of equitable representation So I think you can see a striking difference between the two images And so the idea is that encoded within within the algorithm is a kind of optimized aesthetic for a particular kind of particular kind of image And so the sort of thing that I do in my research then is you could call it cultural computing So that's research to enable us to better understand cultural phenomena such as these at the code level and also Building computational models and systems to analyze and simulate cultural phenomena So you could think about it as building a bridge between cultural meaning such as double consciousness Impression management stereotyping etc. And computational media in particular the algorithms and data structures Underlying software such as social networking profiles e-commerce accounts avatars and more But you might ask why do I use this idea of the phantasm in order to get at this at this issue? And so first I just want to give you a sense or intuition of what I mean by phantasm by starting off with a quite difficult Question which is what does this represent? So it's even more difficult than I thought Does anybody have any ideas? I have another offside idea of the women's room Right. Yeah, so it's a very specific answer there And you know a lot of people will answer either the women's room or they'll answer something like woman in general Right, but you can answer You know why was this designed in that particular sort of way and the reason is because of course it takes advantage of worldview Right, it's taking in that you know so the idea is that you could look at this image and Even if you don't think that women do or should wear clothes such as this It could be a person in a cape could be a person wearing an apron Yeah, there are a number of different things that it could be right people are still recruiting from this general worldview You know this idea women wear dresses integrating it with that mental sensory image at hand and immediately without deliberation understanding what that image means but interestingly Here's another image which means the exact same thing approximately which was Developed for use in the Indian Institute of Technology Here's another image used for much the same purpose that is used in Oman and the idea is that You know that each of these is a phantasm in the sense that it that it hides worldview within it But when you begin to look at them from multiple worldviews You begin to reveal these phantasms because after you've seen each of these some of the cultural assumptions in The one that we saw initially are revealed So this is in dialogue with work in semiotics and signs and myths et cetera But there are some computationally cognitively specific aspects of this theory And you don't have to remember all of the nomenclature here But suffice it to say that these images work in systems and they're kind of distributed cognition process So I'll explain a bit of what I mean by that with another somewhat difficult question Which is if I were to ask you which of these images has greater area How would you answer that question? Right, that's that's a great idea. Some people suggest the elementary algebra equation to solve it That's another way to do it. I get about 50-50 in terms of responses But if I were to ask you the same question and they started off like this Then it would be a much easier answer there and the idea is that this idea of Ed Hutchins And others of a material anchor we offload our cognition onto the image. It's its own representation We don't need to consciously think and figure this out. We can look at it and immediately see which one is a greater area All right, so that's that's what I mean by kind of distributed cognition process without conscious deliberation We instantly understand the meaning and the contention is that the same sort of process happens in an image like this Or even a dynamic kind of animated image such as we might see in a GUI or Like this right or going further in image such as this all right So we have the same sort of cognitive phenomenon of understanding and receiving and interpreting these phantasms When we encounter these kind of digital forms And begin to make make use of them at the same time a number of phenomena that that That as I mentioned pre-exist of these digital forms still persist So this is a classic Clark study Kenneth and Mamie Clark in which African-American children rasted shoes which baby doll do you prefer which one looks nice? Which one looks like you you can see which one was chosen with higher percentage you hear And the purpose of bringing this up here isn't just to recap this event or will study But rather would rather to say that these phenomena persist But how might we begin to describe these phenomena in a way useful for analyzing computing systems? So to analyze computing systems you have to of course look at things in a much more structured way When you begin to look at how data structure data structures and types actually represent this kind of concept Yeah, so that just means types or sorts of elements some that represent people parts of people attributes of people and so forth And so this is a particular kind of worldview like we saw in the woman sign Example but just broken down in a semi-structured way Someone might recruit some aspects of that that that worldview or implement some aspects of that worldview within a data structure and in the case of these Children then they're just immediately understanding this doll say as looking nice by recruiting this information The only thing I've done here is just describe it in a semi-structured way that this That this kind of data is often described in data structures for computer games social networking profiles and so forth So to summarize the definition of phantasm here, you know, it's just it's a Provides a way of describing a conceptual blends of imagery. You know, this can be dynamic imagery With concepts in particular worldviews in a way that's useful for analyzing computing systems And the idea in particular emphasizes the role of worldview and semi-visible values in culture It's to find in cognitive science terms and by that I don't mean just cognition in the head But I mean embodied cognition, you know the way that we think in terms of body bodily metaphors distributed cognition and particular social cultural settings and Perhaps most importantly for today can be described formally or semi-formally to analyze and design computing systems And so though phantasms are as broad as concepts or worldview I want to focus on a particular type of phantasm that of social identity which has been a focus of some of my research during the last four years So I'll begin you know This is an example that I like to show but there have been some updates in it recently So this is from the game Elder Scrolls for oblivion here a couple of types of characters that are within the game And so this is a Nord the ostensibly Norwegian type of character This is a red guard the ostensibly African or African-American character within the game You have it's a in a manual as you have these characters say a red guard being described in essentialist stereotypes of the black athlete You know for example the most naturally talented warriors in Tamriel They're physically blessed with hearty constitutions and quickness of foot Which in fact translates into running and jumping ability bonuses within the game But you can run and jump better and then as you do that more than you get to better at those particular abilities similarly for Nords that and again, this is this is implemented on the back end. It's nothing about the visuals of the characters Actually another game in the series came out later called Skyrim This was a game that to give it to give a sense of the scale of impact. Yeah, so Star Wars on the best weekend sold set made seven million dollars adjusted for inflation. That's twenty seven point two million dollars now Well Skyrim's first day it made 217 million dollars adjusted for inflation from from just recently. Yeah, so so, you know, that's just to say the impact and At least just in terms of sales the question is with a sequel Was there any change to this kind of issue of essentializing? Does anybody here played Skyrim? So and if you haven't seen me give this this part of the talk before do you have a sense of how this changed within within Skyrim did it get any better in the situation for representing Red Guards for example? Well, I can I can show you what the new one. This is a default a default Red Guard in Skyrim replacing the one that you just saw So I won't comment much upon upon this one But suffice to say that I've been writing about this for a while But the issue is that I don't think this is actually any better or worse than the previous one Yeah, so whereas, you know, this is just a change to the front end This is a change to the graphical representation of the character where I'm more interested in the back end the algorithmic and data structural representation Yeah, so yeah again That's to say that at this level where where if you happen to be the ostensibly French Breton you're going to be by default 20 points more intelligent than your Norwegian counterpart That's that's that's the issue that I'm talking about or furthermore examples I give such as in Neverwinter nights where you have five genders representing the data structure You know, I sometimes ask the rhetorical question. How many do you think there are? Well, there are five male female both other and none but male both other and none all have male body types in my default Right, so 80% of the five are male Here's another example that was discussed by Watkins and Everett in the game Grand Theft Auto San Andreas in which you have these characters a sort of loitering or walking around What why is that the case? So why is that an aesthetic decision? Well, actually, it's not just an aesthetic decision It's a it's a game mechanical decision because what you can do is Target a game member to recruit them to your group So they have to have people that are visible within the space that you can then recruit to join you So there's a kind of game mechanical or algorithmic reason But in fact the end result is that you have these black bodies just loitering throughout the city They do a nothing except for waiting for you to come up and recruit them for gangland activity So in this to give a further sense of the scope of the problem Yeah, you don't have to look at all of these stats, but just a couple of the stats here You know so more than 90% of black women characters functioning as props bystanders or victims and again We've seen that some of this is happening algorithmically and so on Right and of course, it's no It's no consolation to look at the stats for other women either right if it's quite harrowing regardless And so I want to conclude with is just a little bit of what we're trying to do about this this kind of situation I should mention further that That if you think well, why is this important isn't just this game just games Well researchers such as Jeremy Balanson at Stanford for example have shown that people actually change their real-world behavior Based upon the kind of avatars they play you play an attractive avatar you according to your own definition You might change your interpersonal distance with others In the real world and so to conclude I'll say just a bit about what we're trying to do about it Yeah, so we're trying to build systems that have more expressive nuance for conveying some of these social identity phenomena And so just to give one example imagine you begin a game and you're playing as a knight You decide to then dabble a bit in magic Magic isn't for you. And so you decide to go back to knighthood and fight with weapons And then that's lost its luster. So you decide to go as they say full mage Now It's at Stanford that didn't get a lot of laughs But you can imagine something else such as listening to Pandora and you listen to To punk rock music begin dabbling in jazz a bit They would go back to punk rock and then go full jazz, right? So it's the same structure and so what we're trying to do is model category gradients So moving from the center to the margins of categories having multiple memberships in categories and the dynamics of how you change your Social categories over time, which is something that most computer games and social networking profiles don't in fact explicitly do so and So we created a game scenario using a platform of our own design called Chimeria in which we have two different types This is just for illustrative purposes You could have any number of different social categories, but we chose one sylvan's which are sort of elf like they like poetry and fine clothes Brushwoods like a good heart, you know, heart tails and the earthy clothing This could have been any again front-end representation could have been anything the back-end representation is what matters We separate out the abstract category So you could say you have one group that's privileged another one That's oppressed you can instantiate that with either so you have any social configuration and then instantiate that with with any different one And it's for the purpose of and the game scenario is one in which you're trying to get into a castle Keep by convincing the person who's from the other group to let you in that you're somehow like them And so I'll show you just what that looks like here Okay So the sylvan's and brushwood have been at war for ages the sylvan known as a tall people average or sometimes judged from far To be lovers of finery and elaborate poetry the brushwoods known as a small people average are judged to be fond of earthy Homespun fabrics and good heart tails You're from the sylvan tribe so remember that and you stand before the gate of a keep you need to enter the need is dire And you see a brushwood guard So the guard looks preoccupied and he's looking away from you So what do you do you dust off your boots?  Okay, so you can hide your your jewelry and he likes that And you think to yourself though because you have internal monologue here I'm trying to fit in with these brushwood the guard has a wary expression We don't see many of your time well new folk around here It seems he's about to say we don't see many sylvan around here. So what do you say? It is not far from home. Oh, yes Good man. This is a strange land to me new one from just around the way or I'm from a little ways off indeed Though any preference here. Oh, but he doesn't like Doesn't seem like he wants you in The guard looks expecting that expectantly at you. Do you speak in your own language to him? Good day. Some weather we're having or I hope you are faring. Well a star shines upon the hour of our meeting Let's choose that But he doesn't doesn't doesn't like it. You're being yourself a true sylvan. He looks curious Do you speak in your own language to say hello readings brother or a good day of brushwood? So You're hoping to get in and he says like you can come in I guess And somehow you talked your way in and and so I'll just just mention Just that you know there is a narrative structure behind this so we have different clause types orientation clauses We're doing constant tests to see what category you're a member of and also the trajectory of your categories over time in order to Choose these these these outcomes And so what we're actually doing is looking at within all of these different features now the change of these features over time and Beginning to do some social modeling so imagine a chimeria not as something like a graphics engine like the unreal engine It's a kind of social identity modeling engine that you could then use in other systems and so for example if you're in Monotonically increasing your membership in their category that's a way to represent passing kind of social passing as a category That you see is different than yours what we had in our example was fluctuating we went back and forth here That's why you don't know how you really got in there Right or you could have not gotten in and so forth And so we're actually able to implement all of these kind of impression management strategies using this tracking function and your category membership At different times and have different kind of endings so you could say be that yourself the entire time but then Get in or not get in you could think to you Or fake like you're you know the member of that category be accepted Thankfully I was able to convince the guard to let me in you could have gotten a different theme denial of self Well, I had to pretend to be something. I'm not you know So you have these two different kind of ways of Both both these two different themes were both getting in we actually implemented this also You know we implemented a social networking site with a musical category taste based on your preferences They change over time and conversations unfold I won't demo that that today But it's suffice it to say that there's a lot of expressive possibility that comes up for critical social commentary Using this kind of system I think compared to a lot of traditional games so to conclude remember me Well, these types of phantasms often remain implicit and invisible to users and are implemented in back-end data structures and algorithms such as this Whereas these types of phantasms are computationally modeled and explicitly designed for studying and conveying cultural meanings in digital media So to conclude my contention is that the phantasmal media approach enables us to better understand and convey cultural meanings Implemented via algorithms and data structures and computational media. Thank you So Fox love. Thank you both very much There's I think there's there's a lot of things to talk about I want to actually start not so much with They're with the question related to some of the themes content of our investigations, but But with a question about how we how we conduct this type of work of a software development of actually making systems like the ones that both of you showed and for my own sake I'll mention that I Have a lab here the trope tank where we also do software development of various sorts It's a physical space that has a lot of Material computing resources from recent years But we also work in contemporary Digital media systems. That's a child and fat our patron saint and And we collaborate Remotely as well as locally with people to do projects like 10 print or the rendering project which is Now in preparation to be published in the next few days The process of doing this type of software development work and and working analytically using software developing new types of systems to model conceptual blends and cultural phantasms is One that I think is related to You know these sorts of things that were brandishing earlier the books that that we create and that are one way that we do Humanistic inquiry and project ideas and enter into conversations, but they're also different in certain other ways So I wanted to ask if you'd each say something about What it means to not only produce theories of books, but also to develop software As part of your investigations, I know that I mean love mentioned that sometimes it involves Looking through 60,000 images to find ones that are sexist and can be removed It's not my idea of creative software development Opposites right But I would say but you know it's taking 65 taking 60 million images and sorting them by file size Realizing what you can find amazing collection of minimalist Incidental art on Instagram. That's that's both example of software, but that's also stuff with Australian So when you say that you want very concrete answer about source for development, okay, yeah, I mean There are obviously many things that could be said but one issue is you know How does it how does it work alongside these more familiar ways of? Conducting scholarship of working in the humanities No one was startled to see books But perhaps the idea that we're developing these sorts of systems that you told us about How is that done how does that how does that come together? Well, maybe tell you a few like practical for your real life. I need those right Okay, so I had a kind of art and design background when I went again to PhD in visual culture And then I was after my first tenure-track job. I was hired in the University of California, San Diego By the art department to help it was like 96 very early right when you media was young Just help you know develop programs in digital art So so I had a PhD. I was hired as an artist Which was great. They gave me a huge artist studio because people PhDs got tiny offices But then they said we're gonna give you credit for books So how does it work? I was hired as an artist. I was only teaching art classes, you know studio classes And yet I was only credited for articles and books So eventually I had to make a maneuver, right? I did write this book, you know, which is unique mention and I get tenure And then from my when it was time for me to become a full professor on purpose I didn't finish my next book, but I finished a big art project So I kind of forced them to actually give me credit for art projects, right? So was this kind of game but I was very lucky because I was an art department for a bunch of artists Who basically like, you know, you can do whatever we want, right? If I was in a humanities or social science department, I think it would be more difficult But then things really got strange because when you're stood with Laborized books, that's okay. Let's make art. That's okay But what's he's doing now? He's playing with big data. He's writing software. That was too much So I had to you know, I said, well, maybe I should time for me to move to East Coast and I go to the East Coast and I'm offered the job at the CUNY Graduate Center as professor of computer science So I'm now teaching.  So of course, you know in my work, I was always interested to both maybe create some expressive Expressive you know expressive ways to represent or to do something by digital media and also to understand this Right, so it was fine But when 2005 realized the new stage of digital media which is coming After previous stages like interactivity Generation networks, the new stage which is coming is in a huge social huge big data And of course, we've been nothing as big data is like nothing right think about, you know When all this thermostats and toilets will get online and you can do like cultural life or for a toilet or thermostat, right? You know like nest studies, right? It's coming. So it's nothing yet, right? I mean whatever billions of images So I said, okay now if I actually want to say something intelligent about the way software Is now creating knowledge in our society the way our way we now moved into what I would call software epistemology right where is machine learning and data mining and your support vector machines and all this Bunch of methods I actually used to generate knowledge in our society. I actually have to become expert in this So I had to learn computer science and when I learned a little bit and I said now I can't learn on my own anymore So I become professor the finish proficient computer science. I can learn more So I basically had to spend seven years learning computer science, you know, and when I became computer science professor So now I can actually say something about digital media in the age of big data, right? So that's a sacrifice But the good thing is now I have a job in New York. I can visit you guys more often I can make art projects and I can also write books about this new thing, right? But it's kind of very hard to learn things right when you already have a Job and mostly people wanting to write a recommendation letters, right? And you're like trying to hack yourself into our so I was trying to make how I would try to figure out How to make these labels bigger to show you but I just can't yeah So I mean the way to do computational work in the humanities like yeah one one technique is to become a distinguished professor of computer science, right? So we'll keep that in mind and let's see what Fox has to say So I can tell you a little bit a little bit about it and also how it's developed over time because you know, so you know my trajectory was I was Made sure to seek solid dismary grounding on multiple sides So that means a double degree as a undergrad, you know, that's art but then also in logic and computing and computer science and Yeah, but then later on as I was doing our PhD in computer science So then a lot of it was like this. That's just you know writing, you know large pieces of Software on my own or with or maybe with a very small Collaboration which was usually just me getting somebody else to use the system and teaching them how to use the system You know, so that was a the kind of the kind of early phase But I did begin to think about how others might might use it You know, so for example and what this was the early attempt though was okay Well, let me make it easy input format, but it still looks something like this which is you know, this code Yes, so you know, but the idea was then to say Well, you know, how could I make this work more accessible for others? And especially as you know when I went to my first tenure track position How to get other people involved since I knew I would have students that were both computer scientists as well as digital Media students and so one of the things that we began to do was that I thought the building platforms is a great a great way I that's what I did did during that program building platforms are creating interactive narrative systems interactive multimedia narratives We could you could create systems that change emotional tone or metaphor on the fly But fix the way that that it looks say visually You know or fix the plot and so then the next step was to say we could have People get involved in other projects that are coming from computer science at you know that like to build platforms You know sometimes we have people that come in that like to use the platforms Anybody could then use it to express themselves You know somebody could be an artist at the same time as we're got regardless of which just my program They come in so the next step was then saying using something like a markup language like XML You have another level of authorship somebody could come in use the markup language But there's still a barrier of entry with with with with this with this sort of thing And so we began to do work so as we built systems like the Chimeria platform One is you have both that you have some Yeah, some it's Data-driven so you can bring in data from other sites such as in the music version of it You can bring in information from Eclipse from YouTube from Facebook You can categorize music from Robi Coral music guide that says this band is so Mysterious and gloomy and bring in those terms and so it's a way that you can bring in data that comes outside of you You don't have to always Program it but the other thing we've been doing recently is also building interfaces So other people at a different level of authorship can then come in use a graphical user interface You build narratives with the system and so there are a lot of different kind of entry points I actually think of it as about four different stages of authorship One is just writing computer code. The next one is using kind of markup languages with a system You have another language which is as a user working with interactive system That can then create a particular outcome like a poem or an experience and the other level is performing it You know like I did the game here, but some of the systems are actually more performative that are used and we've done work with with Free jazz musicians Improvising with us playing the system like an instrument seeing what responds and that's another level of authorship Yeah, so I think actually there are a lot of different ways to become involved in the software production Process you know going down from the code level up through the performative level Maybe I can just add one meta point. So I was thinking like there's you know, there are lots of parallels between us and I thought well if actually if you look at like what I'm showing what he's showing you can get some meta pattern What does it mean to write code and kind of never keeping humanities and computer science and art? So I'm using are which is basically like arcane system, you know from early 70s, right as Command line he's using least which is a more cane language right from the consistence So basically right if you if you want to become like successful person in the intersection with field You'll something be arcane with command line You don't have to write programs just using command line from some software from 1972 does the job, right? Right, I should say that a lot of the back end of the systems We use a lot of soft, you know, I personally enjoy using hit list, but on top of that we use a JavaScript HTML5, you know flash, you know, we use a lot of systems on top Yeah, so I was just going to mention some about the project that Fox and I are working on slant with collaborators here at MIT and in Mexico City and And to also talk a little bit about how people have gotten involved in that project and participated I mean in some cases with with Eric state and he came to MIT with programming expertise and was able to participate Throughout with and also collaborate with Yvonne Guerrero Student in Mexico City on a lot of aspects of the project But but also that for instance, you know, one of the things we we did was to build a unit sensitive to genre this is basically a Blackboard based system where these different units Can contribute different things to a story in progress and after a certain point It's realized by a natural language generation pipeline using some of Fox's griot concepts to do conceptual blending at the first stage Now So without getting into the details of how this works I'll just mention that, you know, we we're trying to model different sorts of very small scale types of genres in which writing or stories can take place Some of them would be a diary entries. Some of them would be confessions to a priest. Some of them would be mixed up remembrances or Or prophecies or things like this. So we have these different types of systems for this and Andrew Kampana who worked with us you know ended up without having a Programming background without having used Python before, you know, just sort of figuring out in the process of doing this enough to develop these Systems and bring his existing knowledge of genre together with programming to to put this part of the system together It's nice to be able to develop, you know A large number of like interfaces and API's that can allow people using XML or other means to do things But one of the problems with that is that it also slows down the development process I think like you actually have to spend more time building out all these interfaces and maybe codifying the way things works Before you know how exactly they should work So so I think there's you know different possibilities for flexible development and people who are willing to engage with programming whether it's a through obscure systems like Lisp and R or you know Or beautiful programming language like Python You know like that that still you know can can provide some good ways to work a little bit more flexibly and and rapidly I think So Well If someone wants to refute me, I'll let them but but we also can move on and I have a bunch of other questions one of them Really has to do with an issue of you know, we think about media literacy We the term computer literacy was something when I was in middle school was very popular But that question of what it is that we need to know about Computational media digital media about the types of systems that we encounter I mean not not from the standpoint of what what are advanced topics for graduate students who are grappling with this But how is it that people in our society really? Should be coming to understand the pervasive computing environment and its ability to mediate communication and art You know, what what do people need to know to do that? Fox, do you have some some thoughts you'd like to chime in sure? I'd be happy to so You know, well, I think I think one is there a lot of different notions of literacy in this area So as they mentioned going back to computer literacy that that's more operational knowledge Perlus discussed computational literacy that that's more the kind of literacy of Programming languages and so how elegantly written is your is your program as a concise and the sort of thing But it was also you know people like you know bogos to Matias and others who talk about procedural literacy That's more of the kind of Procedural steps and being able to think in a kind of systematic way. That's useful for programming So those are all slightly different You know, I also like work that comes in from the learning sciences and critical literacy And so that's where you look at people's indigenous literacy So that might mean in the community people have the ability You know, maybe a people know how that you know that people like to you know build bicycles or youth You know understand the other aesthetics of the selfie and then this the sort of thing and so how can you begin to see the Bridges between those and so one of the sort of things I write about in phantasmal media is Understanding for example, how a recommender system might work to convey to create consensus Yeah, so you understand you don't have to understand all of the technical details But if you understand at a high level the way that that an algorithm functions or the way that these kind of systems function You understand that there are for example network neighborhoods that then you're clustered with other users And you then get recommended things that other people in your network neighborhood like and you're more likely to buy those things that are showing up On your Amazon page or your other your favorite e-commerce page Then you have a sense of building consensus when you begin to actually read the technical papers They're actually users that are called gray sheep and black sheep That's a technical term in these papers for people who you can't you know necessarily Predict or you're somehow pathological from the point of view of these algorithms So the point is that you know you have these social effects You know they're building consensus building a kind of You know as a kind of consumerist worldview and so forth and it's not just to you know wholeheartedly Criticize you know that but just understand that the algorithm is related to consumption The algorithm is related to how we see ourselves And if you have just a bit of the understanding of Algorithmic thinking then you can begin to understand the relationship between that and culture So I think that this kind of procedural literacy is about the that the middle ground where you can begin to think about these Relationships, but also understanding that it dovetails with the kind of critical literacy perspective Which is what are people doing naturally with these systems and then really reinforcing relationship between those So I have actually many things to say this topic. So try to limit it to 1.7 things maybe So well at my lab which I started in 2007 It's actually called software studies and I think now I basically I think this idea is kind of more More obvious but in 2005 2006 it wasn't really obvious That social scientists human needs human needs cultural critics should also pay attention to software as we can engine of Data and information society. I mean now I think we have many books in our own software studies series You know other books. I mean I mean people who publish MIT press but not in our series like Fox I think this idea is now becoming kind of obvious, but it wasn't obvious eight years ago And one of the big questions which we can ask So in a world where decision-making memory management knowledge production cultural recommendations cultural consumptions Are governed by software systems which involve you know data mining machine learning databases client service systems and so on How can general public people who don't have PhD in computer sciences have intelligent Conversation about the systems, right? You know, so in the last few years the word algorithm is becoming more and more frequently used in news But you know, and we like to talk about algorithmic biases, you know, Google or page in a page and buy a bias But how can you talk about the systems which are very complex for two reasons first of a very complex because as you probably know Right at MIT, you know what Google uses over 200. I think different kind of features to signals and Actually, I think dozens of not hundreds different algorithms to figure out results for a search So page rank is just one of dozens of them So you go to Wikipedia and you start reading wonderful page about page ring and first it looks very simple Even it gets more and more complicated. Even it gets more and more complicated. I mean, how many journalists can read this? Right. So partly I think this is really one of the biggest challenges. I think for our time, right? So our world runs the systems Which we people understand Right, our world is a black box You know the engine in the car with mechanical world when that your society, it's kind of simple, right?  So first of all, we're black boxes because you know, Gmail is like 50 million lines of code, right? Or you know os 8 like 80 million lines of code. So the systems, you know, just have millions of lines of code for formulas So that's one difficulty the second difficulty is you probably know at the place like MIT, right? most decisions in our world Which are done by software use machine learning and a big proportion of machine learning is black boxes, right?  You don't know why Right. So in the 80s and 90s people still use expert systems, right? And people use our kinds of software systems where it was actually possible to understand How can Peter make decisions? But when we're machine learning right turn out to be more efficient and turns out that it's much more efficient to build Machine learning neural networks in other types of systems, which do a job done, but we don't know how So now I start thinking about some very radical ideas, which may sound crazy But just talk about let's think about them for a second. I'm about to finish So is it more important for our society to be fully efficient? So you get best recommendation Netflix and the bank refuses loans to just right people where we have no idea how decisions are being made or Shallower governments limit the complexity of software saying you're not you're not allowed to use black box special learning You're not allowed to use expert systems to actually figure out how decisions are being made I mean sounds very crazy, but I think this is the kind of decisions Our society is going to face in the next you know dozens of years as you know Even to open the toilet would require machine learning right if we don't know why it doesn't open, right? So I think with software literacy. It's not just about you know learning to code or understanding Database system. It's a fact that our society is more more running with black boxes Which even people who called them don't understand and which means that we work until we don't and we have no way of discussing them And this is a very big problem. We have to address So it's a kind of maybe not so politically the software It's not a transparency because if I make this as the open code is still 80 millions of life of code So that's not going to help you. It's Comprehensibility you guys are better at this right? Fine if happy find the term but I think problem is very big and of course learning the program is already something right But it's just like one step Because if you can write like little five line program in Python is no good to help you to understand 200 programs which were 100 millions of lines of code which run inside Amazon regenerate recommendations So there's like a big big gap between learning little how to write little five line program And actually understanding how this industrial systems work Okay, so I want to I want to leave some time for discussion with the audience and for questions There but I there is one more thing I'll ask at least briefly because I want to give you each an opportunity to bring to our attention Some domains or communities where there's some interesting types of making strange some interesting types of Defamiliarization and work that's happening. So love. Yeah, I mean I I have three or four I could mention and probably will but I'll let you start. Okay. Yeah, so I think I was really kind of excited to Bite in my time and we will start talking about next question, which is making strange Okay, I will I will try to make more. I will not go into my rent But it was like a mighty rents are good, right?  Anyway, just kidding Okay, so as you guys know very well, right one of the we're basically a 20th century art uses two algorithms mostly one is montage Like taking things which are related making related and then letting your brain figure out what's going on in the second is different realization So if you remember what kind of deformerization or Australiania was invented by young Russian Critic Victor Shklovsky around 1914 where he said with a job of art is to make a world Unfamiliar strange to refresh our perception Now Shklovsky being young Revolutionary critic said that's the job of all art. Of course. He was probably describing modern art and even watch close. He was talking about literature Right, you know, I think a perfect visual example of this Australiania Photographs by people like Mahori Nash and Alexander Rochenka 1920s 1930s So Rochenka working Russia where a very kind of basic Australiania Estrangement the familiarization technique was to not to photograph things from a normal point of view the kind of normal like iPhone But maybe use iPhone stick, right? So basically photographing some unusual point of view, right? All right, so you're photographing some unusual points of view You can still recognize that the world is familiar But it's also very unfamiliar because you have a subtract structure So you have you know photographs from the top or from a bottom and of course there are millions of our techniques people use But I think this is a very kind of easy kind of our history one-on-one Little me illustration. So when I think about analyzing visualizing big social cultural data like Instagram your images or For example impressionism, I think what are two things we can basically do right one is we can work on completely and familiar world of social media data Right where we actually don't know what's out where right So I think when we work on you know a key project or selfies projects You actually don't know what you know one million stagran photos are going to look like So it takes a little bit harder to Australiania because we actually basically first just to want to figure out How a familiar map is going to look like but it's still possible, right? For example, if you read like if you read mass media with people are going to tell you what Instagram is mostly selfies in the food cats. Well, maybe But not only right so for example, you know if you look at the way Instagram used around the world You actually find all kinds of photos you find everything But then you were on most selfies were more cats But you do find everything but if you actually want to find out you have to use computer methods So we took 13,000 photographs which we shared during six days in Kiev and we use computer science techniques of cluster analysis To divide these photographs into 60 clusters based on similarity So find the clusters, which is what you expect right selfies, right? You know very my computer doesn't know anything about the content So we have to guess why can't be the things is one cluster But you can have right big different between background and foreground if you can miss enter But we also find of course things like this right not photographic images kind of textual communication It also find things like this, right? So if you look at this like there's no revolution going on But we're always kind of landscapes right so simply looking at the big social media or you think it's all about X turns out It's about everything. That's one way to do it, but it basically but when you actually look at familiar cultural data When it's really all about Australia because one thing you get the visualization you find with your conventional understanding of Kind of classical culture archives is usually not correct. So just to very quickly two examples. So here we collected My students over 6,000 impressionist paintings So no historians have a curator have ever looked at as many impressionist paintings According to best estimates the pressure is done about 13,000 paintings and pastels. That's about It's about half of it about 6,000 and when you think what impressionism you think it's like this, right? You know, la la la, you know kind of like kind of Instagram porn, right? You know, but in reality, you also get things like this you get things which are very dark very traditional very brownie And what you find is that what you think of as impressionism is maybe about 10% of what impressionist painters paint it And every time we look at a different artist, you know one go whatever we find the same thing and And maybe just one last thing So we also did this work, you know, which was kind of highlighted with first stage of our lab Revisualized one million manga By visual similarity, right? So since at MIT I can say that I can actually name, right? Things were all names, which is very refreshing. So that's actually entropy, right? Which means images, you know, which consists mostly of big parts of the same pixel values on the bottom, right? So kind of low entropy All right, and many as you go up you start getting more texture more information Well in Shannon sense, of course, I know was wrong, but doesn't matter, but that's another conversation As I as I heard as I learned today from talking to one of MIT media lab Wonderful faculty who is running writing a book about it. He knows what he's doing. So just wait and maybe get more texture more 3d Etc. Etc. All right, and I started this project with a kind of naive Arhistorical not naive or historical kind of scholarly idea. Well, I'm going to find whatever styles in In manga, I don't know anything about manga.  So that's the perfect thing to work on because I hate it Right, I'm actually competing. I can't even finish with him, right? Make a bit manga called the newest, you know Like like, you know, like like Virgil Virgil walks about right or Mashable, but that's what different man. Okay So what you find out in fact, it's a kind of wrong question But at least from point of view of particular variables, which shows there's not really any separation, right? But you have this constant kind of change continuous kind of distribution right from this extreme to this extreme Of course, you actually do find styles But then you have to look at what kind of eyes are presented but from this particular variable It's a complete continue distribution. You can say but the interesting Maybe a whole kind of style which emerges like you know in a kind of modern humanities is a Artifact of a fact that we always look at small data sets. So if you look at all the potential art There are probably so many different variations where between surrealism cubism realism, but it's also going to be continuous, right? All right, right now We just put it all into a small room in museum configurative art And then we give like big rooms to these exceptions like cubism surrealism, but it's wrong, right? So when you say yeah left but this is very artificial because you're basically taking things out of a context You collected one million manga pages you project them using continuous kind of measurements You get this big cloud. What if you look at a single manga title? You look at this and you get something like this So this is sorry. This is a single manga title less than a thousand pages And you can see that it's still various all over the space of possibilities So once you learn from looking at the big data, but in fact putting culture in discrete categories can be problematic When you say well, how can I bring this knowledge to a normal study of small data? Going back to like something which you might as colleagues can understand And now you say actually you guys don't understand anything in a good way Well, you don't tell them that because you know not before you get tenure Because actually even with single title doesn't have a style So many people always people who for 30 years of publishing articles about style of manga We're actually kind of mistaking because even a single work doesn't have a doesn't have a particular style So I'm sorry, but of course I lied when I say I'll take the free minister takes seven But the point was to say that I Think that when we kind of study this with you know, this young unknown kind of Universe of social media. We actually don't really know what's out there So it's a bit harder to de-familiarize it because we don't have familiar concepts But when you actually apply these techniques to a kind of conventional things like impressionism manga Manga or classical art. I think the best thing you get out of Continuous analysis e-de-familiarization, but not of your perception of reality, but if you're perception of cultural history So first I'm gonna gather myself because I realized I'm sitting next to somebody who didn't also shed a tear when the Naruto manga ended last month It did yes, well, I nobody told me I probably would have had I probably had a tear because now a data set becomes final But yeah, so there are a couple of answers sure there are a couple of answers to Nick's Question so I could begin so they get the question again was just about communities and practices that we think do this kind of work of De-familiarizing computing practices in some way that's productive And so one I could just talk about some of the kind of places I think are interesting or places like published in a community's computer-supported cooperative work bring a lot of insights in from sociology You know different different AI groups are focused on issues like narrative, but I'd rather talk a little bit more about about my personal motivation here Because it doesn't just come from computing. There's there are a couple of different types of motivation It's thinking about world-building as a form of critical Inquiry and insights so that you know, that's that's one you know works like invisible man But also works like Sammy Ardolani's work where you begin where he begins to use Kind of interesting structure, you know, so interesting formal structure to comment upon society Yes, so invisible man using a Concept of invisibility for to comment on a particular form of racial invisibility But Sammy Ardolani you're beginning to create a semiotics of a wide array of different kind of representations Also systems in which you use meaningful difference to understand content so the Rashomon effect you're showing the same scene with with different incommensurate endings or results you know each each time and The end result is in the contrast or the simulation in some way using simulation to have effects such as this And then also then works to have a balance between improvisation and for more representation. So what I mean is that You have authors that use formal structures such as it a tallow calvino if on a winter's night a traveler You know, it's a wonderful novel, you know, that's richly evocative But it's actually created based on an algorithm. You open it up and read it says something like You were about to read You know, this is literally this is what it actually says You're about to read a tallow calvino's if on a winter's night a traveler begins to describe how smoke drifts over the pages as you read You get to the end of the chapter and you realize that It's the wrong the next chapter is is the wrong book or you realize that the rest of the book is blank And you have to keep on going and she and getting new books and you become the reader Yeah, so you have this evocative story world I think it's quite successfully created along with an algorithmic structure The same thing happens in some forms of jazz improvisation like Charles Mingus It has some very orchestrated components and very chaotic improvised components Improvisation as a way of composition as well and then using that for political commentary You know, so, you know, so all that I'm saying here is that that one way that The kind of communities and practice I'm interested in are ones that look at improvisation on on one hand and Subjective conveyance of meaning and building worlds and structures and then also the very formal structure that you need in order to construct your Work so that yeah, that's a kind of motivation for the kind of social cultural commentary in the work but the other side of it is is something like this which is from Mathematics which is you know, so this is the work of Joseph Gogan who happened to be who was my academic Advisor and one of the things he did was begin to describe social systems Yeah, so it's semiotic systems for example interfaces Graphical user interfaces like you would have on your desktop in very formal mathematical ways Understanding that it's very limited in some ways, but it's the same as if you were trying to estimate Amount of gas you would need to get from here to Boston You could do it very mathematically like this or you could just know miles per gallon and do a kind of Reestimate yeah, so one of the sort of things I've been interested in is Trying to do this this kind of Is this kind of analysis in a way that's more accessible and so you know, so for example taking a game such as this Game Passage, you know, this is a critically acclaimed game that's thinking about Life and death these very subjective issues Let me turn off the this Right, so you have this game That has a very formal kind of structure into it in some ways, right? You're this character This is the author who's moving along the screen, right? You have these life events such as falling in love and The partnering you can choose to do or not they have this character that Is translated further further as you go, you know have at the very beginning the right side was obscured You're constantly changing right you continue aging with within the game and it comes to this Inclusion the game is always five minute experience with the same ending despite that you can navigate it in different ways And actually a number of people find it to be quite poignant in much the same way that you get that Traditional poetry is and one of the things that we've been that I've been interested in doing is sit They're looking at the structure of the metaphors You know so life is a journey moving across the screen is a kind of kind of journey here life challenges Our mazes and so they're very structured families of metaphor that you can find This is actually found from the artist statement about the game and just pulling out these bolded points Just to show that for example. He actually talks about toward the end of life. There's no future left, right? Lifetime is a space there's directionality territory in front of you all of these metaphors are invoked and then beginning to think about how those You know so going from that very mathematical definition to a one. That's a bit more accessible Which I won't go into the details of here But it can help you then to look at the structure of how those metaphors are implemented within the system And you can begin to see things such as well life is a journey with a past present and future The screen is actually subdivided into three areas the past area Which is obscure the present area and the future area it begin it can begin to look Systematically at how some of these kind of values are structured and and again This is a way you know since the question was about the familiar Define the realization is just to say that you have this game that you could just say it affects me And it's poignant in some kind of way But what it begun to do here is to say well, let's look at the structure of the system Mathematically but then make that mathematical representation Accessible to people that might not necessarily do that but the category theory Behind it and began to have insights such as to say that well from that original metaphor What's actually preserved when you map it to the implementation, you know But if partnering is important in life that for this particular author is that something that's less or more important Then dying with within the game You know so and so that that's a bit of an extended example But the point final point is just that looking at its cultural meaning in a very structured sort of way Yeah, I think is one way to make these cultural aspects that are usually implicit within systems Obvious and and also de-familiarize ourselves with just the surface level of the screen level of the system Thanks. Thanks, Mark. So I actually want to get on to Which means 45 seconds. Yeah, well, I was thinking so, you know So I kind of made this statement And I will stand by it with 20th century which is now unfortunately over It's the best as you know, anyway, it was good one We had montage dialectic, you know, totalitarianism, astronomy. Now we just have right consumerism data anyway But so well strangeny was about making reality strange So now we have big data data analytics data scientific realization, which is always going back to 90th century Realism like literature and painting the realism was not about naturalism The reason was about representing the types of reality, right? Barzakh or Flaubert or Tolstoy is saying the type of a peasant the type of a provincial You know happy woman life and since the data science or data mining is by definition about finding patterns in the data Assumes that this pattern is actually where we're now kind of back to this age of realism So how does astronomy fits in, right? So I think that this artistic work which you know, like people like people like him was doing is even more important Because we're back to this kind of naive realism finding fighting patterns a reality Right, and just I guess my one my one beach comment there too is that I'm actually not interested in a kind of social Realism but rather to use simulation and modeling to evoke cultural imagination Through these kind of systems and then finally that it's not doing it It's inspired maybe by some of the kind of world building and culture building the other kind of works too But actually what I'm interested in computationally specific ways of evoking these these kind of effects So it's not emulating the novel but saying how can we have cultural commentary social commentary through the algorithm or through the data structure Yeah, so electronic literature interactive fiction Twitter bots National novel generation month art games are all things I would have talked about Because there's a lot of rich communities doing work in this area actually Jason war who's passage was shown by Fox We'll have an exhibit at the Davis Museum at Wellesley in the coming year And and there's a tremendous amount of activity that's happening. I was glad to hear the connections between sort of avant-garde concepts and The the mainstream that love has made which is a current in his work and also Fox's Explicit modeling and connection to history in his discussion, but let me ask Rather than ramble on myself. If I could have some questions from the audience. I want to invite you to engage with Fox in the lab and Ask about some of these topics or others that are related For the recording could you use the mic there? Well, you could lie We just use the name in the data set doesn't mean you have to do me My name is Greg Byer from the Tibetan Buddhist resource center quick question You said earlier the data that the economic data was better on Broadway from social media than from the government side Could you expand on that a little bit? Oh, yeah Sorry, I know it's not exactly what I said but things for asking Well, so basically when we started with projects, right? We said well, let's just collect social media data because we designed like a system lab to do it Right, so you get Instagram Twitter, etc. And you don't know if you get all of it That's you don't know but what you get is what you get right because it's machine like the machine puts with The stamp which comes through. I mean, of course, it's noisy, but it's probably pretty good Now when you look at the economic data Well, first of all, you don't have economic data on the kind of block level what you have with tracks So New York City is divided into like something like hundred tracks Which are bigger than a block smaller than zip code. We haven't changed much since like 1900s Right, and then what you have is you have a US US census every 10 years We do like real census and then every year what we do is we kind of take like 10% of population and send people We knock on your door if you open the door will ask you and then we publish his updates But it's completely completely like probabilistic. So if you look at the tables you get from US census So here's latest 2013 economic data for New York. First of all, it's broken into this large blocks In a minute next to the column which has income data with a column which says errors like 50% It may actually look at like you go to US kind of sensors, you know, what kind of data website It says average rent in Soho $1,400 how come you guys don't live in Soho like all the ends are three times less Every salary around time square is $300. That's how bad it is So when Piketty becomes famous because he just goes to IRS data and that's it So I think everybody understands this data is bad. I didn't know that I mean I think I have to go fix economics now, right? Because so what I realized is you know, surprisingly social media data in comparison is very very good But the government data is not that good.  Please tell me but it's actually was very funny to realize that you know Yes, so we don't have anything better, right? That's why we became we kept one variable which is income You know when we started looking variables like red like this is just some science fiction, you know So I know maybe somebody in the room like knows a better answer, right? But it's kind of terrifying where millions and millions of economists and all the studies and all kind of working with data Which is so problematic now. I feel this short So I actually was curious about the interface This is a very superficial question, but the interface I saw go by in Fox's talk The GUI that I saw and what that was for building if that was with griot or what project that was associated with it looked very interesting And sure, so you know that I'll try to bring it up here But that interface was for the Chimeria system and it's It would be easier to find this way Yeah, so this this interface is for the you know for the Chimeria system And so they kind of the kind of game that that you saw that I showed at the end You know especially so especially right now is suited for their visual novel style games But actually, you know, one of the things we're doing is plugging it into a unity That's one of the things that even a PhD student was it was very keen on even just today We've done the system, you know right now can do kind of sprite based animation You can represent your narrative structures or whatever kind of structure you want with Looping structure nested structures within it You can you know build in different tests at different points where it is for different categories say if you wanted very utterances others make and say based on if you're an increasing member in this category Then you should say this sort of thing. So so it's actually a system for so you can see a little bit of it here So you can change these different narrative elements You can think in terms of you know the actors on the stage the social categories different features So that was those are things such as you know the speaking ability or the dressing ability of each character And so it's a system for building essentially these kind of identity driven Narratives and again that could range from 2d up through the you know the current work that we're trying to build is Moving towards 3d as well But I like to think about it this way that you're rather that you like I said rather than just having a graphics engine like like Unreal system there can be a social identity engine you could plug in you can do sort of things like most of these systems work very top-down You know where if you want to say a character to say Something because you're an elf you mostly it's hard-coded welcome. You dirty fill in the blank elf something like this But you could imagine plugging this into a system instantiating every character in a city that has this worldview and then have one character That's the kind of radical that's a little bit of a different worldview that you that you inherited from so you get you can Hard-code these kind of identity views In much more dynamic ways using a system like this Hi So I think both of you guys especially fox it with your last response just now are going into a lot of Great detail about the really great narrative potential of the tools we're building Especially just kind of to make this more accessible for people to perhaps create their own narratives It seems to me though that also that these tools could be used to make a lot of the other types of narratives that aren't like Obviously narratives a lot of the things that love was talking about about the increased categorization of our lives being really driven by big data practices and that could really shape Institutional decisions that are being made at all different levels right so the tools you guys are built right now The problem with that is like that's a very like opaque process for every everybody, right? I'm in a policy class right now where they're they're talking about this very issue and all of the the checks and balances we've had before have all Relyed on Consumer-based checks and balances where consumers needed to be able to understand What's the data information being collected about me? Can I check it for accuracy things like that? That's not what's happening today with a lot of these big data practices. So I'm curious for you guys What do you see as the potential for using these methods and the tools you're bringing for as tools for critique? Of some of these other data-driven narratives that we we see popping up Okay, well I switch to you, but yeah, I'm happy to talk let me come to the right slide here. Yes, so So the the idea is you know, how can you make this more accessible and what's the potential for critique of? Yeah of data analytics system rather than just doing sort of our own data analytics. So One is, you know, first of all, I think that there's potential for abuse of a number of kind of Kind of work that's not accessible by the mainstream. So for example Yeah, if you if you think about data analytics and a lot of times people could have a critique that well That could be or any of these systems could be used sort of for the dark side As well, you know But you can also think about it say research on redlining say redistricting housing zones could be used then to redistrict in a Way that is discriminatory, right? So any of these practices could be taken up I think one of the issues here is the black box issue that that that Lev mentioned So one of the things that we try to do is make these kind of processes You know familiar accessible make you release some of these tools You know open source so people can then go and use use those tools themselves and then also using Sometimes we'll restrict ourselves to say use privately He's publicly available data that people put out there and then make and make tools to think with for people Yeah, so, you know, I just want to just give one brief example Which is the work with one of my PhD students that would see here That's a Chang you limb where you just begin thinking about things like in the past we had these narrow lenses on Social status like you just look and you could say which one of these is sort of high status Which one is not high status and then this beginning to do data mining and systems to begin to think about Status performance and these kind of systems except for in this game Which happens to be considered America's number one war themed hat simulator by a sum which is called Team Fortress 2 You know so then we've taken these phenomena of social status and And made apparent the way that they play out in the system to say that if you social network in this way You're likely to perform a high social status with it within the system Or if you perform on the social network this way you have less friends you have less posts and so forth Then you can find correlations and say that you might actually perform it differently in the game in the game And so this is a way of using data analytics But the aim is to take these social phenomena that cultural theorists or sociologists would typically study and then say well now here's a tool that anybody could use and think about how status performance or Say gender and equity or other kind of issues then play out within software So okay, I'll be very brief So I think one of my remaining intellectual heroes is a Bruno Latour and Maybe some of you know a very kind of a non-article herald in two thousands Which was called something like why critique run out of steam right so for me I don't assume necessarily but my job is intellectual and Kind of paid by universities to critique What I like about software is that instead of critiquing we can actually build right So Wikipedia for example is a great example of that open source movement, right?   Well now we can actually compete, right? Right, you can create your own social networks like I love about advertising and maybe it's not going to succeed But the next one will succeed we can create our own open our own Knowledge structures like Wikipedia and so on so forth, right? So I think I see kind of my duty, right is not to critique just for sake of critiquing But rather to understand to make certain issues visible We help others to understand what I think are the key issues in understanding this opacity and like Bognos Even to enable other people to kind of build Veteran teams, right? So to me that's really what separate 31st century from 32th century Right, but mass media no longer has complete monopoly on information It's no longer up to New York Times and Putin and people organizations like that. You know, we actually can make alternatives I'll go for it Break it So with that then I'm curious. Do you see Kind of some of the interesting potential of that being That you're going to be able to kind of represent the same kind of information We've always been working with in a way that's going to like challenge our intuitions and maybe Get us to question question what we thought the output was going to be from it or or Do you think that you'll be introducing variables or the things like that have to do more like textual analysis as a valid like Metric for measuring things and like perceiving social reality and is that going to be a contest like something That's going to be really tough to contest and like maybe traditional social science fields Are you guys like going there? I like so I guess the question is are you trying to speak to social science? Or is this a different discipline who that is just more social in nature? That might have been a lot Well, so I think maybe I would refer to the tour once again briefly, right? So I think I think out of all like famous kind of social sciences He's the only one well He was the earliest one to understand the importance of big data visualization analysis If you make this very powerful argument, which I don't know we don't have to agree But he says but when Durkheim event social science in the late in the 19th century He makes mistakes because he thinks social science should be about modeling reality like simple equations, right? Kind of this top-down statistics, you know because he is seduced by physical sciences He says just as physicists that kind of modeling reality like Newton has these equations We can make this kind of science of social, but of course he doesn't have you know very fine tools of measurements, right? So he kind of realizes oh my god the suicide rate is the same in Norway every year. That's amazing So we're something about society some general patterns, which are both individuals So there's a lot to us as well now, you know, we actually have these millions of data points about millions of people So potentially we can create a new type of social science, which doesn't use abstract terms Which basically can get away. It doesn't have to use terms like you know Like classes or status or divide with human beings into like five or four variables We can have a kind of social science of individual Now I don't completely understand like right the end of it because of course being French academic It's sure important a tour has to get really obscure. So he can't really tell you maybe maybe he doesn't know he can't tell you But he has to be obscure But you know, I was invited to his lab. I was very honored. He says left I think you practically try to do what we tried to theorize. So I think what I see this work, right? It's a kind of humanities and sense of humanity has always been about the individual in particular But he might is only about selection. I'm going to talk about most important artists Maybe it is white artists or like the artist of color, but basically it's about what's important It was social science, which was dealing with patterns which were dealing with all of society but on the level of like this very kind of Right kind of big concepts or this kind of statistics. So how can you have a kind of? science of a study of society Which combines right which combines it which is about Which is kind of about patterns But not necessarily Going to this kind of big journalistic concepts. I know it sounds a little bit unclear But believe me if I was clear, I would tell you right But I think we have possibility to invent how we think about society right and I just want to say with journalism Unfortunately operates exactly way that 90th century science was because we're going to look at for example events and key for events And Times Square like last night and just and just show you free pictures Which you think are representative or why not show you all the pictures? So I think this is maybe very simplistic, but I think it's the first step of this direction Right, why not look at all the data because now we can the computers can help us to do it and And then ultimately I'm going to shut up in a second The problem is not a computer my computer is very demarc promise a human being How can we think about the world without imposing or reducing it to few categories in a few examples? So how they help computers to help us think better We're setting where we should be going Right and so I can add there and there's what you know the one classic way that interdisciplinary research goes forward is also to say What are the important questions and then bring to bear the disciplines that you think can give useful useful answers And I kind of dialogue with each other and so I think of my work in some ways like that What the way in which it becomes disciplinary when you say is it social science a lot of times That's about the register for communicating the work That means if I want to talk about this work the sociologists or the computer scientists I might have to they might have different value systems for what what consists of assessment and evaluation different language We're the important questions Understanding those values allows me to communicate better, but for me that it's it's the core questions that that drive You know bringing these different disciplines together and to bear on these problems And I think there was some there was some parallels You know so for example when when lab says this impressionist painting isn't necessarily what you think when you look at it From this way, we've had some systems where we say use somebody's social network friends and friends of friends hundreds of people And then rather than just looking at who's tagged himself as a gamer. You know you say look at Using something called point-wise mutual information tag a few particular games You spread activation and find what are the other things that people tend to like if they like these games And when we ran it with with certain MIT students We actually found about half of what they liked there was classical music composers and then there were some other games that showed up You know so again That's it was unfamiliar in the sense that we started by saying that if you like these three games What else do you have you tagged that that you like? And we've looked at this category, but then the same as the impressionist painting there was a result that might not have been expected By by every user of the system then the system becomes a tool to think with you think well Then how do I compare to one of this this category that is emerged from the data? But I do think it is important at the same time is doing this bottom-up data-driven approach It's also because to think about the social categories that are theorized and social in social sciences and live by people because even if they are in fact Fantasms people live in a response to these social categories You know so you know so to look at the ways that those social illusions show up in software to say that this is Discriminatory because if you play this game as a female character, you're not gonna be able to play one of the standard Roles with within it, which is something we found with some of our research That's right, I'm not showing anything just now This is actually showing Anyway Anyway, how you might run a clustering algorithm and find what's emergent within the data that you didn't think was there Yeah, so this is to say We're actually finding this the status of different people within the game Which is doing something like I was describing before you run the algorithm and you find some information That was that's in the data some patterns that are there But then we want to go and reconcile that with with our with our social understanding is come from other fields I would it means to be of high social status how you then treated differently if you're dressed in a way That's a tribute that's associated with performing high social status and so forth Yeah, so as to say I don't think the data-driven approach means throw out all of the cultural analysis Analysis of hegemony and social structure and hierarchy and power and all of this But it means we have a new lens and there might be phenomena that we didn't expect that that emerged in a new Way to understand them Well, we've come to the end of our time. I hope that Our guests made computing strange if it wasn't strange already for you and I'd like you to join me in thanking them